When asked to name one of the great things about our network, I have to say it’s our fiber footprint. With over 57,000 miles of intercity and 27,000 miles of metro fiber, our footprint is a grand design of both buried and aerial paths. It makes us unique, but with that also comes one of the worst things about our network, and that is the different types of damage we see to our fiber plant. From errant excavators to crazed squirrels, there are so many different ways to wreak havoc on our network, it boggles the imagination!
While I was watching Letterman’s “Top Ten” reasons the government cannot agree on resolving the debt ceiling crisis, I started thinking. Over the years I have seen fiber cuts that range from frustrating to downright bizarre. And just when I think I’ve seen it all, I am rudely reminded that anything can happen, and typically does. So allow me to present my top ten most annoying and bizarre causes of fiber cuts (with real photos from my team):
- The biggest pain and the most common cause of fiber cuts come from construction companies and excavators that don’t call before they dig! One time we had an outage in California where the excavation company had dug a trench and found a steel pipe about 4 feet underground. Now, you would think that they would have called someone and tried to make sure they weren’t cutting into something dangerous like gas or oil, but no. They jumped down in the trench with a saw and cut through the pipe and into our fiber! What a group of Einsteins!
- While we can try to reason with humans and publicize our underground cable, there is nothing we can do about our next biggest pain in the rear, and that’s squirrels! Of all the animals in the whole world, almost all of our animal damage comes from this furry little nut eater. Squirrel chews account for a whopping 17% of our damages so far this year! But let me add that it is down from 28% just last year and it continues to decrease since we added cable guards to our plant. Honestly, I don’t understand what the big attraction is or why they feel compelled to gnaw through cables. Our guys in the field have given this some thought and jokingly suspect the cable manufacturers of using peanut oil in the sheathing. If you have any new ideas on how we can combat these wayward rodents, I’d love to hear from you. We are always looking for ways to improve.
- Our next biggest offender would have to be Mother Nature and her extreme weather conditions. I could fill 10 pages with stories from the hurricanes, mud slides and ice storms we’ve had to deal with. One incident that sticks out in my mind was during mud slides in Utah a few years back. We were trying to repair a cable across a ravine that was literally over a quarter of a mile wide and filled with raging water. Every piece of equipment and truck we had was stuck in mud up to the axels. We launched a couple of Sea-doos and a boat to try and pull the cable across the ravine to make the repairs, but no luck. We finally had to shoot the cable across with a line gun. We got the job done and luckily no one was hurt. But it was pretty scary.
- Speaking of big trucks, vehicle damage is another one we have to contend with. From people running into telephone poles to truckers underestimating the height of their rigs – it’s all part of the problem. One time in Pennsylvania a trucker got lost and accidently turned down a residential street. His rig got tangled up in a mess of overhead phone cables. But that didn’t stop him! He kept pushing forward until his rig was tied up like a Christmas present. He was dragging a 20 foot section of broken telephone pole down the street before he stopped to see what was impeding his progress. Future DOT Instructor??
- Putting phone cables and electrical cables on the same pole can be convenient, but it can sometimes cause an outage by default. Such was the case in Boise, Idaho when strong winds during a dust storm blew down one telephone pole. The stress on the cable pulled down a second pole, then a third, until 19 poles were lying on the ground. But that’s not what caused the outage. Our cable was down, but still intact, until one of the poles sent an arc of electricity that started a grass fire which melted the fiber. Our guys repaired the cable while airplanes flew overhead dumping fire retardant to battle the blazes. Incoming!
- One of the dumbest reasons I can think of for an outage is downright vandalism. It doesn’t happen all that often, but it certainly happens more often than it should. People sometimes cut the fiber thinking it has value and can be sold in pieces. Or they vandalize it in other ways. We can attribute about 7% of our annual outages to people using our fiber cable for gun practice. More often than not, it happens in the rough parts of town. Making the repairs usually requires that we dispatch a couple of body guards with our techs, just to keep everyone out of harm’s way. By the way – it’s not copper!
- Everything I’ve mentioned so far can be considered all in a day’s work. But every now and then I hear about something so out of the norm that I just want to say “What? Are you kidding me??” Such was the case with a recent outage in California. A small airplane was attempting to land at the Burbank International Airport and overshot the runway and crashed in a residential area. It clipped the poles that our aerial fiber was attached to, causing everything to come down. No one was killed. But I wouldn’t want to fly with that guy!
- And earlier this year, there was an ice storm in Chalfont, Pennsylvania that brought down some tree limbs. The limbs fell onto the PECO Electric utility primary power which crossed into the communications space. The cable caught on fire in multiple places while suspended in the air and surrounded by ice covered limbs. Fire and ice!
- You never know what to expect during natural disasters. I would go up against a squirrel any day rather than encounter some of the other creatures that are unearthed during a hurricane. Alligators, snakes and sharks are all creatures you don’t ever want to encounter on a fiber cut. But sometimes it happens. For example, during the cleanup efforts after hurricane Katrina, one of our field managers was about 2 miles inland when he spotted a three foot long shark in one of the trenches beside our fiber. That is probably the craziest animal encounter we’ve had in the field to date!
- Lastly, never under estimate a Southern gentleman with a backhoe and a shotgun. Remember how I feel about vandalism? Well I’ve saved the best for last. There was a landowner whose property stretched across the border between Georgia and Florida. He was mad at Florida DOT because he didn’t get enough money when they purchased the right-of-way to widen the highway that cut through his property. Level 3 had fiber in the right-of-way, so he was mad at us too. One day he decided on revenge, so he jumped onto his backhoe and drove across the state line from Georgia to Florida, right up to the edge of the ROW and dug a 2 foot wide by 10 foot long trench. He then got down in the hole and cut the fiber and the ducts. Then he moved 15 feet south and dug a second trench until he found more fiber and ducts and cut them in a second location.When our field techs got on the scene, Mr. Landowner was waiting on them with his 12 gauge shotgun! He refused to let anyone repair the fiber on threat of death! When law enforcement arrived, Mr. Landowner had moved back over to the Georgia side and claimed he had no idea how the damage had been done. He was out of their jurisdiction. There were no witnesses, and all the law enforcement could do was talk to him and try to get him to confess. At least we were able to repair the damage. But during the conversation with the law, Mr. Landowner spewed anger and said he was going to come back tomorrow and cut the fiber again. Well, that was admission of intent to commit a crime and the rules of jurisdiction didn’t apply anymore. Ha! He was arrested and we were able to see frontline justice after all.
During all of this, natural disasters, impossible odds, wild animals and angry landowners, I am thankful that I have a team of experienced professionals who are willing to get down into the trenches, make the necessary repairs and restore service fast. No matter what they encounter, this team can handle it.
Subscribe to Level 3′s Beyond Bandwidth (e-mail):
About Fred Lawler
My favorite is when the gas company uses the dig safe lines as their pattern line for drilling holes to test for leaks.
Very funny — There was a fiber company that added cayenne or something like that during the manufacture of the cable and it worked to keep pests away.
Tom, I was going to recommend cayenne pepper, too! It works to keep the chipmunks, squirrels and deer away from my tulip bulbs! Sure does take a lot of it, though
Doesn’t work on rabbits. We tried cayenne to keep them out of our garden and they just thanked us for seasoning their salad!
Nice… gotta remember that when I lay some fiber/cable/etc…
What was the name of the company? We may be interested in that here in south Louisiana.
Fred – an interesting read indeed.
Try to work with the manufacturers to get a pepper or chilli spray bonded in the fibre coating. Organic, cheap and will keep squirrels off. Temporarily in regular trouble spots you can also sprinkle red chilli powder and they will keep you clean for a few months and send the pests elsewhere.
Take care of your crew when handling it though.
Let me know the results if you use this, hope it works for you.
We market a non-toxic, clear gel called Rat-Out. It has garlic oil and white pepper as its active ingredients. It’s also exempt from EPA registration because it’s all food-grade materials, so anyone can apply it. Rodents hate the smell, and if they touch it they get an uncomfortable warming sensation. It works on rats, mice, squirrels and many other mammals. It will keep animals off wires.
I was working for a major telco in western New York when we had a fiber cut caused by a homeless man. He was sheltering in an abandoned aqueduct that ran across a river. The telco had run the fiber, in conduit, across the ceiling of the aqueduct space. When the homeless man lit a roaring fire in a 55-gallon drum right under the conduit to keep warm… melted fiber.
The bad part was that someone had decided to cut costs and run both legs of the SONET ring through said conduit, presumably because of the limited number of river crossings and the cost associated with them…
Was this in Rochester in the summer of 1999? If so, I absolutely remember this…
This was in Rochester NY if I remember correctly. This story is famous!! All very true!
We had a similar occurence when some homeless people lite a fire in an abandoned warehouse. It was an old wooden structure and it went up fast. We saw the fire from our shop and thought it may endanger our equipment hut. When we arrived we noticed, much to our relief that our hut was safe. Then someone asked, “wait, what is our route our of here….” You guessed it the fiber rand along the same road as the burning warehouse and sure enough it was on fire. Just as we noticed this the alarms started rolling in. Hardest part was waiting until the fire was out before we could finish our work.
This catalog reminded me of an earlier attempt to collate the various ways fiber could be damaged.
Among those, my all-time favorite has to be this one:
Seems from the photo that the good folks of San Antonio, Texas, (a) have too much time on their hands and (b) are well-prepared for the next Alamo defense.
As the former President of Frontier Telephone of Rochester (NY) I can tell you why the squirrels eat cable. during the fall they are stocking up on vitamins and minerals and they would eat all of the casing off of aerial fiber and copper. A good read.
Ford
We have found that squirrels will also drop acorns and pecans down riser pipes and then attempt to retrieve them by chewing through the first thing that gets in their way, the fiber. We started capping the risers with expandable foam that you can get at any hardware store. It is easy to spray in and easy to remove when you need to access the riser. So far the squirrels have not tried to dig it out once it is in place.
This is a good one. I do have a customer that a woodpecker took out their fiber twice. Bottom line if true redundancy they need multiple routes and possibly different sources. As a microwave vendor I obviously like wireless as a redundant back up to fiber. They work well together.
I agree I truly can’t understand why the vandalism.
I love the photo of the toy superheros showing off their squirrel “trophy.” Squirrels are indeed a nemesis. Here in Virginia I’ve caught them chewing up chimney flashings, plastic gasoline cans and even supposedly foolproof motorcycles locks, so I’m not surprised to learn they wreck fiber cables, too. What works for me: a 12-gauge shotgun and a box of #4 shot. Since taking an aggressive stance toward these furry-tailed rats the score thus far is Jim, 11; Squirrels, 0.
Nobody wants to hear about how you kill animals.
Not unless you include some tasty-sounding recipes.
I do! Squirrels get into everything, and I mean everything so the saying the only good squirrel is a dead squirrel applies here.
I do, Way to go Jim!!
Our back yard in Vienna, VA has too many squirrels. Maybe because of the bird feeders, though we have switched to safflower seed, and they don’t seem to care for it unless they are terribly hard up. But, occasionally a lovely hawk visits our yard, and I just wish that I could persuade it to nest nearby. That’s be the best deterrent.
cable sheath can be formulated with repellant in it.
we’re talking fiber here but:
http://www.okidensen.co.jp/en/prod/cable/lan/mouse_lan.html
is an example.
It would appear that vulture scat is very acidic and will corrode aerial fiber casings over time. So we often see issues with the optical signal in a few key sections of our fiber plant due to the large number of Vultures in those areas. We also lose fiber to gunshots ( I can tell the first day of Dove season) as people try and take the easy shots.
Ok gentleman, that kind of things happen just because you did not think about putting fibre inside sewer pipes. All those stories seem pre-historical to me, never happen inside sewer pipes. Just ask me how. Best to all.
Then comes the dudes diahrea that has an acidity that eats away the fibre.
My favorite: a certain university was laying a triangle of fiber to connect three buildings. 1st line goes in fine. 2nd is too long, so the installers cut it to fit. Mysteriously, the 3rd line was too short (by the same amount that the 2nd was too long…)
Back in the day, we used “R” rated cable, which is supposed to be “Rodent” proof. Rodents love to knaw on cable because it sharpens their teeth. They think it’s this great big durable root. Construction guys know that everything dangerous (power, gas) is encased in a concrete vault – their philosophy is that’s it’s easier (and cheaper for them) to ask for forgiveness than permission.
it’s easier (and cheaper for them) to ask for forgiveness than permission. makes a lot of sense
I’m ex-TCI/AT&T Broadband. Engineered thousands of miles of fiber over the years.
Favorite story was garbage truck driving down alley with forks up. Caught lots of fiber and affected about 290000 people. Other one was vandalism just before Superbowl. That one was fixed JUST IN TIME for kickoff. Our techs were superstars and would work into the wee hours and in sub-zero temps. Say what you will about the cable industry – I never ran into a better bunch of dedicated fiber pros as the ones at TCI and Media-One
Thanks Fred. I’ve seen many kinds of damages too, but recently we’ve been hit several times by other companies placing cable by directional boring. This technique is great, but unless locates are done accurately, it’s also an accident waiting to happen! Attention to good as-builts is very important, and cooperation with your One-Call system, like 811 or Miss Utility is critical. The thousands of hours our cable locators spend is money well spent, every time we avert a two-legged creature from digging up our facilities.
Well if you ever are hiring traveling armed guards for your technicians, shoot me an e-mail and we’ll talk.
Have Gun, Will Travel as they say…
Does this AR-15 make me look fat?
Be thankful you don’t work in Australia. We have wombats chewing through armored cables.
Be thankful you don’t work on an asteroid. Those mynochs are pesky bastards.
Wow. Like Lake Woebegone only real! Loved the truck “pulling fiber” – and the pole. Wow.
Those squirrels do a lot of damage. My favorite squirrel story:
I was working as a mainframe systems programmer at a shop at the southern end of the Texas Medical Center. The area was supported by two substations, though our shop was only connected to one, located behind the building.
Our group was scheduled to go on an outing to see the Houston Astros play, a few miles away at the Astrodome. As we were gathering to head out, the power died, and we heard a boom out behind the building. We were standing around cussing when the power blipped on for a moment, accompanied by another boom. Then it did it again. I realized that the intervals seemed regular, and looked at my watch…sure enough, 60 second later, it did it again. Someone looking at the substation noticed a flash among the switchgear.
*blip* *boom* *flash*
*blip* *boom* *flash*
*blip* *boom* *flash* …every 60 seconds. This went on for 16 cycles before it quit.
We found out later that a huge contactor with contacts the size of a Volkswagen was trying to reset, and failing because it was feeding into a dead fault. The cause, according to Houston Lighting & Power? A squirrel.
We started accusing HL&P of having throw-down squirrels tucked away.
Here a story – During the “Big Dig” in Boston, contractors were awarded early and on-time bonuses for meeting deadlines. Utility contractor was going to get a $100K bonus for finishing that day. Backhoe comes across a cable bundle with 4 fiber cables inside. Driver calls the Project Manager. Stopping the work would put them into the next day and they lose their bonus. Project Manager looks up the regulations and reads that cut cables were $2500 a piece. Project Manager calls the driver back and tells him to cut the cables and finish the job.
Yes, the below is a relevant humor link regarding fiber optic cables and its attendant mishaps, not a spam message about viagra…
http://groups.google.com/group/rec.aviation.military/msg/39d55e5f4163ad00?hl=en
Another one… when I was working for a big multinational at one point, we had a problem with a link that ran across northern Russia. We demanded that our provider tell us what was taking so long to fix it. It turned out that winter up there is so cold that vehicles aren’t reliable – when that cable gets cut they dispatch a tech driving a *dog sled team* to repair it… I couldn’t even make this stuff up.
My personal favorite in 11 years in the CATV industry was vandalism by a farmer of fiber crossing through his fields. During the week between Christmas & New Years he drove a piece of farm equipment into the middle of his field and damaged the aerial plant. We were called out at dark and could not find the cut in the dark so with approval went back out early the next morning. At that point we just followed the tracks in the fresh snowfall to the point of the damage. The land owner had no idea why those tracks led to and from his outbuilding or how the fiber was damaged…
The squirrels are probably eating the fiber jackets for the same reason that mice chewed through a wiring harness on my wife’s Honda: electrical insulation made of soybeans. Our mechanic says that this is becoming a big problem.
Google on “electrical insulation soy rodents”.
Here’s one discussion:
http://thisbluemarble.com/showthread.php?t=25163
Rabbits are also doing it:
http://science.slashdot.org/story/10/10/14/1425219/Denver-Airport-Overrun-by-Car-Eating-Rabbits
http://www.aversiontech.com/denatonium-saccharide.html. Spread this yearly on exposed cables. Supposed to keep even the most rabid squirrel away.
This article rocks. I’m the guy that says “well, according to the traceroute, it’s the ISP’s sh*t.” Thoroughly enjoyed it.
Dude. I fricking love you. Level3 is my world. Everything I use, I know where
the pingtrace runs. Without level3, I have no internet. If I could get level3 fiber
to my home, it would be my 21st century equivalent of a 1980′s T1.
Level3 is the sh*t.
T1 s – invented in 1967. One year after Charlie K. Koa invented low-loss fiber we’re talking about. ( See 2009 Nobel Prize winner C. K. Koa)
Yeah…some of us are that old!
On the topic of cable casings that prohibit chewing – it seems a bitterant would be more effective than a pepper. wormwood or cactus. not much gnaws on those things.
Excellent list of cable-cut causes! Reminds when I worked in the field and had to strugle with cable-cuts too caused by people who had to remove a tree (whole tv-network Amsterdam down), crushed handholes caused by heave cranes and the top was a constructor who had to drill into the ground and cut a complete trench of ducts filled with fibercable. Approximately 12 cables (all different operators) were cut-off and happily half had redundancy. Keep up the good work!
My best shot against rodent was an experiment with a cable manufacturer.
Using classical repellants (peppers, some poisonous chemicals, whatever) with a parafine base to coat a textile layer between two PEHD layers in the external coating does the job very nicely.
It makes those cable huge and expensive because we added fibers in the exterior coating to detect rodent attacks using a reflectometer and proactivaly replace partially damaged cables, bu still, this design can hold 72 G.655 fibers in a 17mm thick aerial cable.
Interesting and whiny at the same time.
Two clamps up.
When I was working in telco/cable I saw some interesting cuts. The best two are the drunk dump truck driver who decided he always wanted to drive though Big Lick Virginia’s “down town” with the dump bed raised. He managed to snag and pull every aerial that crossed the road for about two miles. The plant engineer must have owned stock in Alcoa because he built in enough service loop that the system almost survived. The break happened when the police came to impound the truck and their heavy tow hooked it, pulled forward a couple of feet, and then SNAP!
The second of my favorite cuts was caused by a farmer digging a pond for his cattle. Standard backhoe induced fiber fault (BIFF), but when we came to fix the cable his bull got out and charged us, the truck, and anything that moved in “his” pasture. We ended up having to call out a helicopter to scare the bull away after gun shots, banging on the side of the truck, and bullhorns all failed.
We track all of the wild stuff on a daily basis with our Eaton Blackout Tracker: http://blackouttracker.powerware.com/index.htm
Squirrels, vandals, storms, ospreys, mylar balloons, car crashes, a bald eagle dropping a deer head on powerlines–all pretty amazing stuff. We’ve got an annual report PDF that you can download, too.
Damage by Animals.
Really excavators are a menace of fiber cut. In animals porcupine also damages the fiber.
I work for a Telco too, and we also have fiber cables running around the country. And we’ve had our share of fiber cuts. The vandalism issue we’ve had to contend with as well. The really weird ones? Once we had an outage during a storm. What happened was a fiber running along a bridge crossing a river got severed by, get this, a log that got knocked into the river some ways upstream and passed under the bridge on its way downstream. And it happened while its redundant fiber some 30 miles away got hit by vandals a few days earlier.
a 3-foot long shark is harmless — it probably eats guppies and herring fry (if not plankton).
I didn’t see the infamous fiber cut in Boston due to the homeless group living under the Tobin bridge, lit a mattress on fire that got out of control, melted the fiber that was running on the underside of the bridge.
Awesome squirrel picture – very funny. Wish I’d thought of that. Can I use it for my Christmas card? (With proper attribution.)
Great article too.
Nothing recent, it has been a while since I worked in the industry:
Repeater station becomes unavailable completely. That’s unusual. Hands gets sent.
“The station isn’t there any more” – “yes we know, why has it gone offline?” – “There’s a rather large truck where it used to be”.
A customer has redundant connections. One day, there are street works. Classical backhoe induced fiber cut.
So the backhoe driver stops their work on the one end of the building and turn their attention to the other end, where they promptly dig up the backup fiber.
KPNQwest Germany has their fiber in gas ducts. It was claimed in a previous comment these were in concrete ducts, they’re not in .de, but they have a bright red sand layer “sufficiently” high above it. This stops most backhoe drivers, but when it doesn’t, you get to wait at least 4 hours while the area gets evacuated and the injured gas pipe left to bleed empty until you are allowed near the cut to mend it.
All right, I have to share these two:
circa 2008: Highway widening off a major coast-to-coast interstate, nobody called first. Boring through the ground (in a very small town you’ve probably never heard of, it’s called Houston, TX), cut an 1892(?) count copper & fiber cable IN FIVE PLACES. Company in question realised what they’d done, and *removed the identifying placards off their construction equipment before fleeing the scene*.
El Niño’s done a few, too. Maybe 1997/1998-ish, had a report that the engineers “watched the cable float away on the floodwaters” and were waiting for the floodwaters to recede to splice. They were circling the area in a helicopter with spare fiber on board, and had to leave twice to refuel.
Is Level 3 using FO cable with steel armor tape wrapped around the cable core and under the PE jacket? Double jacketed armored cables are also available. This steel armor tape was tested for 20 years by DOW chemical and no rodent on planet earth chewed its way through, not even the notorious “Denver Gopher”. They are more expensive but a lot less cost in the long run that a network outage or digging up an replacing a cable. Squirrels think this is nuts!!
I forgot to mention the armor tape is ballistic proof as well……
There are some new classes of deterrents on the market that could easily solve this problem.
we have this same problem with squirrels and rats chewing on electric wires in businesses and homes, this is dangerous and can cause fires.
if your are interested go to our website or email me at the above site.
Fred Lawler
Take a look at our web site http://www.detourforrodents.com
Yours is a common problem and easily sol ved with the new gel technologies with capcisan
cliff
southwestern regional manager
Pignx,LLC
This article and comments are priceless! Fred – email me and let’s get you published in OSP Magazine! Anyone else with great anecdotal stories is invited as well. Pictures are even better.
Email your editor: svollman@pracom.com.
Let’s get it started.
I appreciate all of your spice tips – wait – is this a cooking blog? Haha. The best resolution for rodent based fiber damage is truly to eliminate fiber from your outdoor network. The best means for that is wireless point to point or wireless point to multipoint – invisible fiber.
I know the USA standardized on fiber years ago as the nations communication backbone but it’s only the backbone where humans can trench, dig or run continuous above ground cable. Fiber gets us pretty much from coast to coast but then where the wires can’t run, the Nation’s carriers rely on licensed microwave technology to carry the load (over the rockies for example) and then once on safe, flat ground they pick up again with fiber.
With options for speed and bandwidth North of 1Gb and SLA’s to support 99.999% up time, it’s a compelling and worthwhile option especially if you are leasing fiber, that costly monthly expense can be eliminated entirely with a wireless link. Something to consider…
Great stories. I learned a lot I didn’t know about fiber optic line!
Well to see story number 3 really takes me back. I was the one who made the initial OTDR shot and found the cable in pitch black of morning. I try and tell the whole story to people and find myself wondering if it was true. We also tried a bow & arrow..taping a cooler so it would be air tight and floating it down in hopes of the others side could grab it somehow.. The most important and perhaps the hardest part of this story to believe is that the guys that were hired to use their wave-runners shot jet line across the raging river with a potato gun attached to a full Coors light can.. Unbelievably true..
I just discovered this blog and it was really informative and humorous. I loved it!
Synthetic pheromones may be the answer….
Merit/Michnet had a fiber cut between one of our switch rooms and the local Intermediate School District (county wide services across districts). Shot the fiber from our end and it was the right number of km. They went to the ISD and shot it, about 300 ft. Walked out to the pole where it went from ground to arial and found a garage sale sign nailed thru the guard with the next door neighbors phone number painted on it… Expensive advertising.