Observations of an Internet Middleman

We received a lot of positive feedback, as well as a lot of questions when Mike posted his recent story, Chicken. Many of the questions asked for more specific data about the scale and size of the problem. This post is an attempt to provide some of that data.

Our Internet Services product sits on top of this global infrastructure:

network_mapThe orange lines are cable systems that Level 3 built and fully owns (the yellow lines are owned by multiple carriers or leased).  That means thousands of miles of fiber in trenches across land and thousands of miles of fiber in cables on the seabed. In all, our network contains approximately 180,000 miles of fiber – enough to circle the equator seven times.

That fiber is then turned into usable bandwidth by installing equipment in data centers in each of those red and yellow dots, and also roughly every one hundred miles along each terrestrial cable. That bandwidth is then turned into an Internet Service by installing routers and switches in key locations. The Level 3 Internet Service consists of more than 10,000 Ethernet connections – getting bigger every month.  The original invested capital in the Level 3 network was approximately $40 billion.

Level 3 uses that network to sell Internet Services to tens of thousands of customers all around the world. But, despite the huge amount of infrastructure that we built and contribute, we are only one part of the global Internet. When we sell Internet Services, we have to make available every single route on the Internet to our customers – not just the routes we ourselves own. That means we have to provide access to all of the networks owned and operated by others, which right now means about 46,000 other networks – some of which also make use of Level 3’s fiber and bandwidth services.

Level 3 builds a route map of the Internet by connecting its tens of thousands of customers together and allowing them to communicate. So a Level 3 customer in Hong Kong can communicate with a Level 3 customer in Sao Paulo. But to complete the map we also need to fill in interconnection to everyone who isn’t a direct Level 3 customer, so that our customers can also communicate with those who are not our customers. We do that through connections to other networks and their customers. This latter sort of connectivity is often called peering. Peering connections allow for exchanges of traffic between the respective customers of each peer.

While Level 3 has tens of thousands of customers, it only has 51 peers[1]. That total set of interconnections enables our customers to “see” the whole Internet. And what is important here is the “distance” our customers see between themselves and any other part of the Internet. That is often referred to as the number of “hops”; or number of other networks a packet has to traverse to reach its destination. We strive to make that number as low as possible to offer our customers the best performance; more hops can introduce more delay and more potential for quality degradations when the other networks don’t invest enough in performance, redundancy and capacity.

So how does all this compare to other networks? Renesys does a good job describing the interconnectedness of the Internet, but their reports are often misunderstood. They do not show how much traffic each network carries on any sort of relative basis. They merely show the interconnectedness of the networks on a relative basis. As you can see, Level 3’s network is the most interconnected. This list of companies along with our combined investment, innovation and competition has enabled the Internet to grow dramatically by carrying enormous flows of traffic around the globe. Removing these middlemen would leave a massive hole in the Internet.

Much has been made of peering agreements. Many peering agreements were made between engineers in the early days of the Internet and consisted of not much more than a single page of text – if there was anything written down at all. They weren’t really contracts in the way you might consider a formal legal agreement. But over the last decade or so, they have become legal contracts that have a defined term and a set of expectations that each party agrees to adhere to. The vast majority of those contracts are settlement free. For example, 48 of the 51 Level 3 peering agreements are settlement free. In one case, a peer pays us for access to a number of routes in a region where their network doesn’t go; a choice they made rather than buying Internet Services from another party. As we have explained a number of times, our policy is to refuse to pay arbitrary charges to add interconnection capacity (more detail to come in our forthcoming solutions blog post).

But there are also typically shared costs for networks to interconnect. Each party pays to augment its own network to allow for more traffic exchange (the expense to augment capacity is not significant for either party). And since we often choose to interconnect in a third party data center, the networks usually agree to share the cost of the cross connects by paying for them on an alternating basis.

The table below shows the connection locations Level 3 has with its peers, and the total interconnection capacity exceeds 13,600Gbps.

peering_routesLevel 3 has 51 peers that are interconnected in 45 cities through over 1,360 10 Gigabit Ethernet ports (plus a few smaller ports). The distribution of that capacity with individual peers ranges from a single 10 Gigabit Ethernet port to 148 ports. The average number of interconnection cities per peer is five, but ranges from one to 20.

The average utilization across all those interconnected ports is 36 percent. So you might be asking – what is all the fuss about with peering? And why did we write the Chicken post? Well, our peers fall into two broad categories; global or regional Internet Services providers like Level 3 (those “middlemen” listed in the Renesys report), and Broadband consumer networks like AT&T. If I use that distinction as a filter to look at congested ports, the story looks very different.

A port that is on average utilised at 90 percent will be saturated, dropping packets, for several hours a day. We have congested ports saturated to those levels with 12 of our 51 peers. Six of those 12 have a single congested port, and we are both (Level 3 and our peer) in the process of making upgrades – this is business as usual and happens occasionally as traffic swings around the Internet as customers change providers.

That leaves the remaining six peers with congestion on almost all of the interconnect ports between us. Congestion that is permanent, has been in place for well over a year and where our peer refuses to augment capacity. They are deliberately harming the service they deliver to their paying customers. They are not allowing us to fulfil the requests their customers make for content.

Five of those congested peers are in the United States and one is in Europe. There are none in any other part of the world. All six are large Broadband consumer networks with a dominant or exclusive market share in their local market. In countries or markets where consumers have multiple Broadband choices (like the UK) there are no congested peers.

As an example, this is what one of those congested interconnections looks like. It is a 100Gbps interconnect in Dallas for the week ending April 3. The graph on the left shows flat tops for most of each day – the port is congested and cannot accept all of the traffic that is trying to get through. Not only are packets being dropped (the number dropped are on the right), but all those not being dropped are also subject to delay.  The effect of dropped and delayed packets is discussed in our prior post.

route_info_1For comparison, below is an uncongested interconnection. This is also 100Gbps but in Washington, D.C. with another peer. This shows no congestion, although there isn’t much headroom, so a capacity augment is underway. The graph on the right shows absolutely no dropped packets.

route_info_2One final point; the companies with the congested peering interconnects also happen to rank dead last in customer satisfaction across all industries in the U.S.[2] Not only dead last, but by a massive statistical margin of almost three standard deviations.

csatShouldn’t a broadband consumer network with near monopoly control over their customers be expected, if not obligated, to deliver a better experience than this?

Tweet Chat



[1] Our peers are generally other networks that complement our own and that agree with us to exchange traffic on fair and reasonable terms – see our peering policy.

[2] According to the 2013 American Customer Satisfaction Index ( www.theacsi.org )

The following two tabs change content below.

Mark Taylor

I work as VP of Content and Media here at Level 3. English expat and passionate new tech energy evangelist.

Latest posts by Mark Taylor (see all)

221 thoughts on “Observations of an Internet Middleman

  1. So why does Level 3 not terminate the peering agreement with these companies. They are clearly in violation of your peering policy. If they want access to your customers they would be better off buying access from a transit provider.


    • Because if we terminated the connectivity we would damage the customer experience (for our customers and for the broadband provider’s customers) even more. We are not willing to do that.

      • Perhaps it’s what needs to happen to show how fragile Comcast’s (and other networks) peering really is. Shove a few extra x00 Gbps through their already congested Tata ports.

        No pain, no gain?

        • You cannot. The closest thing you can do is to to try pick an ISP that has good transit connectivity, but as a consumer, it’s really hard to figure this out, and you hardly ever have a choice of ISPs anyway.

          You could of course try to complain to their helpdesk, but the odds are that it will never be forwarded anywhere.

          • This is where naming the companies would really help consumers. We can’t make this choice because we don’t have the data. But a monthly report detailing the average load and time over 90% would help us make that educated decision (if it’s even possible).

        • You most likely can’t since you’ll need to meet networks like Level3 at a colocation facility or be lucky enough to have access to transport from a last-mile network to reach a colocation facility. Costs a residential consumer will likely not want to bear.

      • Mark, I feel that you’re damaging your own customers’ experience by maintaining these saturated interconnects. The customers (both Level3’s and the cable companies’) see the packet loss and there’s a 50% chance that they’re blaming Level3. Consequently, by allowing this situation to continue, Level3 is diminishing the value of it’s own service.

        Level3 and the rest of the short list of global fiber providers are the only ones with the power to do anything about this situation. Force these guys to fulfill their end of the agreement or give them the boot. Your customers will thank you.

        -A Comcast Business Class customer

        • “Giving them the boot”, as you say, would actually make the situation worse for our customers and for the broadband network’s customers. That is not something we are willing to do.

          • You need to think about the larger picture and/or long term situation here. What is a provider like Comcast going to do when faced with the decision of “upgrade infrastructure or loose peering?” Well, they’re going to upgrade, as they should. It’s a simple equation IMHO.

        • “Level3 and the rest of the short list of global fiber providers are the only ones with the power to do anything about this situation.”

          This is not true. We have he power as well. Sure they could drop service from Comcast, AT&T, Verizon, Time Warner and other ISP’s that are not living up to their part of the agreement and that might help but it would hurt customers. The ISP’s listen to loss of profit and thats what you’re talking about but how is another way to do this?

          If enough customers “probably in the millions” dropped their ISP’s…or even threatened to this could work as well. How can we ask Level 3 to do something that will harm other customers if we’re not willing to do something as well? This won’t happen because people need their internet and TV. So we can’t ask them to do something like this either.

      • My first question from the form of business over politics would be if L3 possibly looked at the prospect of leasing the third parties the hardware / resources required to stay within the needed performance indicators.

      • “…we would damage the customer experience…We are not willing to do that.”

        I REALLY wish that the greedy companies you service through peering felt as you do. They clearly are no longer run by techies, but have been taken over by accountants who only serve one purpose – deliver more profit for less service.

        At least now we end users know something about how it all works, and now have sources to show to those overpaid and underworked hacks who occupy seats in the Congress. We just might be able to help the situation from our side of the connection.

      • Hi Mark, I am a hi-tech conslutant and worked for Nortel for 29 years.

        I am wondering if I can ask you some questions?’
        I have been studing networks for years and am try to sort out how big is the Internet is the US?
        If Netflix is 35% of upstream and 9% of down stream, however Netflix peering in US looked to be less than 3Tb/s. Could this be true, or have I missed a few Tb/s?

        Hope we can communicate in the future if your interested….

        • Don, a lot of people have tried to estimate the size of the Internet.

          I’m not sure anyone has been successful. The nature of the Internet makes it very hard. For example there are more than 46,000 individual networks that form the Internet. They are interconnected in all sorts of ways. And a packet of data can flow over one or many of those networks when it moves from one ‘side’ of the Internet to the other.

          I have no idea if the 3Tb/s number is correct but you mention that in the context of their peering. Netflix has other relationships with providers that are not usually classified as peering. So the amount of connectivity they have to the Internet would be higher.

  2. How much would fixing the interconnection issue at a single location like your Dallas example cost for an ISP? I’m thinking it’s not in the millions of dollars.

    • Each 10Gbps port would be $10-20k. Additional costs might be triggered if there were no more card slots in the router there.

      • Marks figure is about inline with what my cost would be on a Juniper MX.

        Which really isn’t that much money to these huge networks.

        Assuming a billion dollars in net profit a year. Let’s say the cost of the port hardware is $20k / per port, that’s 10-11 minutes of operating time to pay for that port. Surely these networks have much larger economies of scale so you can come to your own conclusions on the cost for ports to these networks.

        • Yeah… he left off the “per month” part. It’s not $10k one time.

          I’m a big fan of Level(3), but the tone of this blog post is disappointing. Sure there are crappy MSP’s out there, but there’s a bigger picture than the one this guy is painting.

          • Actually it is a one time cost. The question was about the cost of the interface cards on the interconnecting router.

      • 10 Gbit for 20k? wow
        how come some datacenters providers sells 10gbit connections lines guaranteed for 500-1k usd for colo or server rent? Overselling?

        • That’s the cost of the network interface card. Which I think answers the question I was asked. There is a network behind that which costs a lot more to build and maintain. Actually that was the point of the first part of the post. To give some idea of the scale and cost of building a global network.

          • I think he has answered the question asked. It is a ONE time cost for the hardware involved. I maintain a large network and routing infrastructure for a video game development company and we host our own dedicated servers.

            I also built and designed the network for Austin College. They have direct service from Level3 as I installed and set up the router, and made the connection. I assume level 3 does thi sort of things for college etc. I didn’t sign the contract I just configured the equipment.

            However. The router needs the Port, and the module to connect 10gb fiber. This cost can get quite high per port (I just ordered 4 of them here) You need Nexus level switching gear as well.

            You buy all of this yes..it is a one time cost. We have spent a LOT on hardware. The monthly cost of the bandwidth coming in however does exist but it isnt as high as one might expect.

            The cost for hosting and or terminating these types of circuits is quite high initially. The one thing you had right is that most ISP’s could easily afford the equipment and see a return on investment pretty quickly. Especially if their service level picks up. That means more customers etc. Then again more customers means more bandwidth needed means more hardware..the cycle continues. Such is the life of business.

            Its hard for a provider to not oversell their service as I mean..if they are the only provider in an area..what are they gonna tell you? Sorry…we are sold out?

            So yes..they need to build their stuff and maintain the equipment to raise their quality of service. However..that means spending money and most businesses like ISP’s just want to see the ROI. Problem is guys like me, and other engineers dont write the checks. Guys in suits do.

            Those guys are hired..and payed. to spend as little as possible. Really. Their job is to spend a little and show a higher ROI. Period. You can manipulate this number by making 100k. and spending no money. Or you can spend the money, show an ROI of say 50k for the year..but make that up in a 4 or 5 year horizon…but these guys get bonuses..etc.

            So if guys like me wrote the checks..sure the network would be right. When you got guys in suits who have no idea what 10gb even means making the call..this is what happens.

  3. Mark:

    I’m very glad to see this degree of detail, backing up an excellent post. I think it’s invaluable to shine a light on Internet peering global norms, as well as the harm to consumers that’s done by those who deviate from them. Thank you for a very well-written, well-supported, and timely article. We couldn’t agree more with the sentiments expressed, and our own data show a very similar group of outliers, who are abusing their own customers to extract excess rent from their effective monopolies: https://www.pch.net/resources/papers//peering-survey/PCH-Peering-Survey-2011.pdf See figures 6 and 7.


    -Bill Woodcock

  4. “Shouldn’t a broadband consumer network with near monopoly control over their customers be expected, if not obligated, to deliver a better experience than this?”

    I don’t think you understand how monopolies work 😛

    But on a serious note, thanks for writing this and the “Chicken” article.

    • Earned monopolies must keep up a minimum customer experience or a start-up can swoop in and take big chunks of market share. Government enforced monopolies don’t have this problem. Most Telecoms are government enforced monopolies because municipal governments won’t grant easement to lay new infrastructure, or state governments won’t issue licenses, etc.

    • At this time, we have decided not to specifically identify the peers with significant congestion; however we can say they are large incumbent broadband providers in the U.S. and Europe.

        • I’d just look at the top 5 ISP’s on DSL Reports by number of complaints. That’s all I’ll say… Part of not saying specific names here is that they’re in a business relationship and it might come off as unprofessional.

      • I’m sure there are other considerations at play here, but on the surface the decision not to name the offenders seems to contradict your claim to put customer satisfaction first. Nothing will change as long as taxpayers remain unaware of the problem providers. If nothing changes they will have no incentive to improve their service.

        • Just want to add that the bad peers probably have a lot of political power (which is how they became a monopoly in the first place, and how they manage to keep their monopoly position despite atrocious customer satisfaction), so your caution is understandable.

      • One one level, I agree with, and respect the decision not to name and “Call out” the 6 broadband providers that are refusing to upgrade their connections (Though I have my suspicions as to which ones they are)
        On the other hand, and this may be reading between the lines, This post does seem to indicate that there is a feeling that they need to be called out.

        Has there been any discussion about when/if they will be named at some point in the future?

        I love the MRTG Graphs BTW.

      • I think you need to name names. As far as the average consumer and the press are concerned they are hiding in the shadows. The public needs to know who is to blame so they can bring pressure on their elected governments, at all levels, to enact change. Right now the FCC is blindly accepting all the major Telcos tell them because there is no major public push against them. Yet these same Telcos have no problems naming names for those services they see as congesting their networks. What’s sauce for the goose is sauce for the gander.

  5. Mark, You explained this very well. Thank You….
    As far as customer satisfaction surveys go, you will never find ISPs in the higher ranking industries. This is due to the the fact that people use IP traffic more hours per day than they travel on an airplane. People today, recall the problem they had for 10 minutes and they will remember it for months. It overshadows all the years of great performance experience. Clearly, industry satisfaction comparison is not relative. In this industry, ISPs are only as good as the length of time since your last outage. ISPs even get blamed for Google’s outages. A single Google resource un-available while someone at Google changes that routing is the ISPs fault. The end customer feels the entire ISP network was down because they couldn’t reach that one server.

    • You are being too easy on ISPs with that excuse.

      I have been with my provider for twelve years now. About four years ago, my speed slowed down, yet when I contacted my provider about it, they said that they can’t control what AT&T (the owner of the lines they lease) does to them. Since then, service slowdowns and interruptions are more frequent. I really would like to have another option to cut AT&T out of my service, but thanks to monopolism I cannot.

  6. Pingback: An internet middleman's take on net neutrality - iDoss

  7. if Comcast was bold enough to get Tom Wheeler his job as FCC chairman what else do you think they’ll do to protect their revenue streams? you guys need to stop being so naive. Comcast owns the FCC and most of Congress. This blog post will do nothing. If you want anything to change then you guys need to take action. Call Milo Medin at Google and make him an offer he cannot refuse. You guys need to help bring in new blood. Can you tell us how it is working with Google Fiber compared to Comcast? Which company would you say you would prefer? Everyone I know would switch to Google Fiber in an instant if they had the chance to. Comcast, Time Warner, Verizon, Cox, AT&T… these companies you’re afraid to even mention by name… they will never change their ways. It’s on YOU to help us, otherwise they will just keep growing and becoming more powerful.

    • Morgan, Mark is, by his own admission, an Internet “middle man.” He does not walk in the same circles as the people you mention, nor does he have control over these decisions.

      • If you look at Google Fiber in KS, they are connected directly to Level 3. So what I’m saying is Level 3 should better incentivise new ISPs, like Google Fiber, to compete against the 5 ISPs that Level 3 is too afraid to even mention by name (Comcast, Time Warner, Verizon, Cox, and AT&T) I’m asking Level 3 to call up Google Fiber right now and say “each new city you expand to, you get first year for free” or something… something to make it more affordable initially to expand and compete.

        To quote another source:
        “Comcast’s apparatchik, Wheeler, is the head of the FCC and is doing everything possible to use the government’s power to help Comcast achieve a total monopoly. Wheeler is a threat to the Internet, freedom of speech, and the Internet-based economy. He needs to be fired immediately.”

        Trying to work through Congress or the FCC will accomplish NOTHING. The only option is to work around them. but I fear companies like Level 3, and most citizens are too naive and are going to let Comcast get away with it…

  8. Has Level 3 ever considered getting into the last mile game? I know it would be expensive, but you guys literally own the backbone already. Why not drum up some investors or partners and start expanding fiber to people’s homes? We’re (consumers) out here ready to buy. Hook up with some electric utilities or others. We’re begging you! Get us out of the clutches of AT&erizon!

    Help us Level 3 Kenobi, you’re our only hope!

    • For the same reason that nobody else can compete in the last mile. A duopoly is a duopoly because Level3 (and everyone else) isn’t allowed to expand “competition” beyond two “competitors.”

  9. Mark, I understand their may be legal reasons that you won’t disclose the names of the companies, but if consumers are to fight back they must be informed. If it would truly cost the ISPs less than a dollar per subscriber to add another interconnect between your networks…well, that’s indefensible to their customers. but the customers have to know.

  10. What needs to happen is for ISPs to move to metered pricing (pay for what you use). Since no ISP really wants to do that for fear of raising prices for users, they end up doing this to try to get the money from the content providers.
    If the ISPs would charge a reasonable rate for metered usage, they could recover the costs of the increased capacity from the users who are causing the need to increase it.

      • Very good articles, they make me angry and sad though.

        The second article you wrote

        “maybe the appropriate additional charge for a network operator to levy on its customer is use during the busy hour. If a higher percentage of their customer’s use occurs during the busy hour(s) then they should expect to pay more than people who only use the network when there is lots of unused bandwidth. Or if you want high speed during the busy hour(s) you pay a premium. If you don’t need high speed during the busy hour(s) you pay less.”

        The problem with this is we’re leading to a lot of things needing internet access. FiOS and Uverse stream every channel at once to your box, so that uses bandwidth unlike other TV technology that when you go to a channel you see your program. Playstation is going to start streaming games to your box (like OnLive does), so if you want to play a PS3, 2 or 1 game you can without having to download it. Microsoft might do this later on with the Xbox One, but right now they’re doing this with AI and other textures to your console from their servers. People are listening to streaming music and we’re going to have internet enabled appliances and services that Microsoft, Google, Blackberry and apple are going to bring us. Not using the internet during peak hours is going to get more difficult. Not to mention when 4k becomes the norm, thats going to take even more bandwidth.

        We’re not only going to need more download speeds but upload. I have a friend works in IT, he owns his own small company but he logs into people computers remotely. He can’t afford to not work during peak hours but he’s also not rich enough to pay the high price of paying the extra money. This is true with a lot of people who work from home. This could kill small businesses…not to mention family’s who have their kids home for Summer and other holidays.

        What if you pay for one of these plans that says you don’t need high speed internet at peak hours? That would mean that your internet would be slowed or you wouldn’t get access to the internet till when 8pm? 10pm? later?

        Cell phone data is metered and it sucks. Some carriers are making big deals that you can have your whole family up to 4 on one account and share 5 gigs of date. Thats 1.25 gigs per person. So we went from unlimited data to 4 gigs per month to now 1.25 gigs per month…whats going to happen next .25 gigs per month? So I agree with you there metered would be really bad.

        I think things should go as they were. ISP’s are making billion upon billions of dollars and yes they want to make billions more in profits but those profits aren’t doing us the consumers any good because we’re not seeing the spoils of what we paid for. ISP’s want to double dip, make billions off of us and make billions off of content originators. I think they should either do one or the other. If they’re going to charge us then they shouldn’t charge the content originators and if they want to charge the content originators then they shouldn’t charge us (or at least charge us much less). They won’t do this because they have a monopoly and the government isn’t going to do anything about it (because the government sanctioned the monopoly in the first place). So we’re behind a rock and a hard place.

        Like I said in a comment earlier we the consumer can do something but we won’t. We want everyone to help us because we’re not going to help ourselves. You have people who are cutting the cord so they can just have internet and maybe go with Netflix (Hulu, Amazon Prime…etc) instead of TV, and that is only a hand full of people. Even fewer are willing to even call up their ISP and say “give us a fair price for internet with no slow and fast lane, give us net neutrality and a decent price or we’re going to get rid of the internet all together”. Maybe the ISP’s would call our bluff and if they did we’d have to in droves…millions of people would have to cancel their service to scare the crap out of these ISP’s. We’re not willing to take action like that. So things will stay the same and most likely get worse. We’ll end up paying more and more for our internet and TV and complaining about it, writing on blogs like this and doing nothing to help ourselves.

        I think it would take to much to make them notice though. I know 1% wouldn’t do it…would 10%? 30%, 50% of the US cancelling their internet work? So we do have the power, but we won’t use it.

  11. If people know their internet is slow, they will get involved.

    Could you make a website that will tell people if they are accessing it via an overloaded connection?

    Just imagine how people react when cable TV goes offline for ten minutes. They will react the same when netflix or google is degraded. BUT only if they know where the problem is located. If I know my service is being degraded, I can ask my ISP to fix the issue. It doesn’t matter why it’s degraded. Whether it’s net neutrality, or just bad infrastructure, either way I can make an informed request.

  12. Looking at those graphs is interesting. The traffic appears to be very unidirectional. Does Level 3 peer with any companies that have the same traffic profile, but in the reverse? Does Level 3 offer peering to Netflix, Akamai and other CDNs.

  13. Thanks for the clear presentation of the internet service. Now I have some idea why our internet speed is so slow. We used to be Time Warner customer, then switched to AT&T when TW kept raising price and lowing quality. Before I made the switched, I called and made sure the performance was ok for what I normally do everyday, like browsing, internet shopping, watching youtube. After a month or so, the speed became intolerably slow and I wanted to switch back.

    I don’t understand why. Is there anything that customers can do about this?
    I just came back from Beijing. There is a firewall against some sites, but people can always use some software to “jump over the wall.” The speed is super fast and the cost is half of what I pay here in the states. Plus, there are many free internet access after you register [free]. Honestly, the U.S. is supposed to be the richest country in the world and yet still enjoy second-class service.

  14. I’d be so, so happy if a future post listed the providers with technical details, or at least if L3 announced plans to do so if nothing changes.

  15. How much of a dent can companies with a web presence put into this problem if they were to compress, minify and optimize their content to reduce overall internet bandwidth consumption?

    I think bandwidth costs for many large companies are considered negligible when compared to the rest of their expenses. Large corporations mainly care about making money. Reducing bandwidth consumption would be the very last thing on their mind. What can be done to make bandwidth conservation a priority for all websites?

    • Actually all content owners who care about performance already do this to the greatest extent possible. That’s because reducing the payload improves the delivery speed. And faster speeds are directly correlated to higher customer experience scores and greater revenue per customer.

  16. If consumers are to mount a “mad as hell” campaign, what exactly should they be demanding? NN doesn’t seem to exactly cut it – it’s noy a question of equality but capacity, and the “cheesing” of both customers and content, holding them to ransom for a premium.. I have a button machine, maybe I should make some “NO CHICKEN” badges?

  17. Pingback: Internet congestionata (ISPs vs Level 3)

  18. Feel like naming and shaming? I am especially curious who the one European peer is that shows this behaviour.

  19. Hey Mark, great post!

    East coast US resident here. If you cannot name the guilty parties, could you name 1-2 companies who are NOT intentionally congesting traffic? If no to that, will this type of data be public in any form for those who want to go digging around?

    Thanks for posting this. I have a small hope now that Comcast won’t rule the world.

    • We are not disclosing any names at this time. The data isn’t readily available unfortunately. As I said in another reply the Sam Knows web site and TE Netflix blog have some data that shed some light on it.

  20. Hi great article. I always assumed it was down to the ISP not paying the tier 1 isp sufficient amounts and thus over subscribing their available bandwidth. You would not be able to get away with such fraud in any other industry but thanks to contention ratio non sense the ISP can get away with over subscribing bandwidth because they rely on the customers to not use all their available bandwidth all the time.

    May be it is time for Level 3 to start selling gigabit internet to residential users (in the us) and instead of cutting the middle man out, cut the ISP out that are using your bandwidth to rip off the end user.

  21. Where can we get the data on usage between peers? It would fairly easy to make a website showing congestion on a local ISP if we can get the data from the bandwidth providers. This tool would really help smaller ISPs get more customers and hopefully start to get away from the monopoly we have today.

    I am currently working on my CCNP. Would I need a CCIE to work with L3?

    • The Sam Knows website and the Netflix streaming data on their blog get close to this. But it’s actually hard to do because there are so many network connections. Figuring out all the underlying topology is very, very difficult. A problem for this task but actually a strength for the resilience of the Internet.

  22. Monopolies have no incentives to make their customer’s experiences better.

    The nay-sayers would argue we have no broadband monopolies in the U.S., but THEY LIE. There are generally two players in each region. The cable operator and the phone company. Only Verizon has attempted to bring a competitive Internet service to their regions to compete against the incumbent cable operator. So in most regions this leaves the cable operator as the single, high speed carrier.

    And Comcast, Time Warner knows this is the case.

    And they want to squeeze every penny they can from any point of revenue – from the consumers to the peering network providers to even the content creators (Netflix).

    The problem with Internet traffic is that the long haul segments have to grow capacity in lock step with the last mile providers. If either one of them gets greedy, the whole system becomes out of balanced.

    Right now, the last mile providers have the advantage.

    We, the consumers, wait patiently for a better last mile connection like we waited for the cable operators to save us from the old plain old telephone line providers and their dial-up and DSL.

    Isn’t it ironic that our saviors 14 years ago have now become our masters!

    • Which is why no one should ever be viewed as a savior, just as a potential master. Checks and balances, boys and girls, in politics, in business, everywhere! 😉

  23. I would gladly switch broadband providers except, as you point out, there is no real alternative. At some point Google Fiber or Sonic’s Fusion/Fiber might become available to me but the ROI is not there, yet. 🙁

    Thanks for an enlightening post!

  24. Has Level 3 reported the 5 US based ISPs to the FCC or the FTC regarding consumer protection and quality of service? If this was going on for over a year, then it would have been a case for the FCC to investigate. Consumer protection is their job. Why are we just now hearing about this? What political agenda/lobbying firm is Level 3 supporting regarding this?

  25. Pingback: Internet providers continue to intentionally slow down speeds for additional money | Android Authority

  26. Pingback: Internet providers continue to intentionally slow down speeds for additional money - AndroidBoard | Latest Android News and Reviews | AndroidBoard | Latest Android News and Reviews

  27. Isn’t the whole premise of settlement-free peering based around the idea that there will be roughly the same amount of traffic flowing in each direction between the two peers? In the instances where the ports are being saturated (and the peer won’t upgrade them), how does the traffic level flowing into Level 3 compare to the traffic level flowing out of Level 3?

    • You are highlighting the problem with the “traditional” concept of ratios. It simply doesn’t work as a way of measuring equal business value when one network is a global backbone selling synchronous services, and one network is a consumer broadband network selling asynchronous services. Traffic will not be in balance at the interconnect because no where inside the last mile is traffic in balance – it is architected to be out of balance. As a broadband consumer you get more downstream bandwidth than upstream.

  28. Pingback: droidultimate » Internet providers continue to intentionally slow down speeds for additional money

  29. Even as a fellow expat I’d contribute to a US-based crowd funding campaign that was targeting two things… 1: a structured form of corporate representation via a consumer advocate board member working towards circumventing bad actors; and 2: transparency into both the monopolies and government actions that sustains them.

    Obviously there’s the LVLT stock, but purchasing it doesn’t equate into the access and advocacy that is so needed for both privacy and neutrality at the consumer level.

    Thanks for writing Mark. LVLT might be sitting on a massive customer base that would pay not just for goal of future service, but advocacy from an entity large enough to grab government ears..

  30. I have firsthand experience with this problem recently. We have one office in Chicago with a comcast internet connection and another office in Tennessee connected with ATT.

    A traceroute reveals that Comcast, rather than using their own nationwide fiber ibone network uses level3 as transport from Chicago to Georgia. In Georgia we experience high packet loss periodically. This must be a congested peering point between Level3 and ATT.

    It is adversely affecting our business on a daily basis. We are not just a casual Netflix user – this is impacting our day to day business. These types of changes don’t just affect consumers!!

    • In the example you give you also have to wonder why there isn’t a better interconnect between Comcast and ATT direct. If you are buying a dedicated business internet service (one with the same bandwidth up stream and downstream then I might suggest a company called Level 3 as an alternative in both offices.

  31. Your article says the ISPs are deliberately harming their customers. While I don’t doubt your assessment, I don’t think we’ll see any changes until the customers notice. As long as they can still watch Netflix and Hulu and access their favorite websites, they’re never going to notice. I need high-speed access at home and I pay for 16Gbps. It typically fluctuates between 15 and 18 Gbps. As long as I get what I pay for, I’m not complaining. Again, I don’t intend to offer this as an excuse for the ISPs, but I don’t see them changing their ways unless they suffer consequences from either their customers (me) or providers (you).

  32. Your own graphs seem to suggest very unidirectional traffic originating from L3, why should the peers increase intake? Let’s cut to the chase on what this is really about, L3 provides hosting to netflix and other content originators who are the pay tv/ pay internet service’s competitors. The pay tv services don’t want to allow the additional traffic because the better the experience is the more it devalues their services. While I can’t stand the pay tv model and think the cable / phone companies are fat monopolies who are holding back US internet experience the arguments being put forth in this blog post are disingenuous.

    • If you think some large ISPs are using their monopolies to benefit their own service offerings, then I think we are actually in agreement–see our prior Chicken post. We intend to make the point in more detail in blogs to come.

  33. This finger pointing needs to stop.
    Everyone is trying to make money, that is granted, no one is altruistic here. We need to look why isn’t the market self-regulating, what is broken.

    Consumers didn’t use to use Internet in practical terms at all, it didn’t much factor in to the costs how much consumer uses the Internet, other cost contributors masked capacity consumption completely.
    In such cast, product should be flat-rate.

    But things have changed, Netflix and Youtube can easily be 2/3 of all traffic eyeball SP sees, traffic that didn’t use to exist. And things are going progressively ‘worse’. This means capacity consumption matters to your cost structure, but this cost is transferred to consumer to in very opaque, ineffective disruptive manner via indirect billing through content networks.

    Today there is strong financial incentive for eyeball not to deliver bits, because every bit delivered reduces your margins, this is absolutely ridiculous position to be in.

    We should remove flat-rate INET, have some base charge and then per GB charge. This would make eyeballs cry for the traffic, there would be financial incentive to upgrade links ASAP, since the more traffic you can attract, the more money you are making.
    There would be incentive to quickly fix broken connection, non-working connection is not generating revenue.

    Charge just has to be fair, forget mobile billing, forget roaming, it might be as low as 0.1c or as high as 10c per GB, depending on location.
    You could choose pre-paid billing, with 256kbps rate-limiter when credits run out, or you could choose maximum daily, weekly, monthly, yearly billing, with 256kbps rate-limiter when limit runs out.
    Consumer could be notified via SMS or email about credit exceeded.

    • “We need to look why isn’t the market self-regulating, what is broken.” Because it’s illegal. Reminds me of the old “If voting worked, it would be illegal.” line. You literally can not legally compete with Comcast and the local phone company in most of Comcast’s markets.

  34. Pingback: Level 3 Accuses Six ISPs of Not Updating Networks to Ease Congestion »

  35. Mark,
    I heard it’s actually worse than this, at least as far as streaming performance.
    Or there have been allegations of this being intentional, and of “bandwidth management appliances” being installed to slow down the services that need QOS.

    So they don’t work properly. It’s intentional.

    It seems if what you say is going on they don’t even need to intentionally screw things up, they can do it by refusing a peering upgrade that every legitimate ISP would consider normal.

    Glad Level3 is legit.

  36. Pingback: Six Major ISPs Accused of Deliberate Traffic Throttling | Digital Trends

  37. Pingback: NewsSprocket | Major internet firm blames ISPs for purposely slowing down web traffic

  38. Thanks for at least blow a bit on the whistle. But it would be helpful to let us know in which states in the US and what country in Europe the culprits are.

  39. The traffic graphs in the article are not even close to symmetric and exceeds 1:2 ratio by a long shot. Based on that, It is reasonable and fair that the party overloading the other party with traffic should pay for the excessive traffic. Peering is only settlement free when the agreement is of mutual benefit to both parties, which is not the case when for example the ratio is off the chart like in this article.
    If Level 3 are exceeding their ratio, Level 3 should pay the other party for its excessive usage. This is nothing new and is common knowledge between most/all internet service providers. Some providers however tend to forget that.

    • Jørgen let’s say I carry 100Mbps for 100 miles and then hand it to you to carry 1 mile. And you carry 1Mbps for 1 mile and then hand it to me to carry 100 miles. According to a traffic ratio we are out of balance by 100:1 as I’m sending so much more traffic to you than I receive. But my cost could be 100 times greater than yours. A ratio was always a proxy for equal business value. That can work within the backbone of the internet. But cannot work when a last mile network sells an asynchronous service to its customers and then connects to a network who doesn’t. In that case ratio is no longer a workable proxy for shared business value. That’s why we have suggested a change to bit miles. Measure how much you carry and how far. That’s a better approximation of shared business value for interconnecting networks.

      • Mark, you are a backbone provider, naturally you are going to carry farther.

        I recognize what you are saying but I can’t see anyone wanting to pay metered by the mile same as a short haul. If you can’t carry over long distances for less than short haul, no one would want you at all. They would have to figure out another way.

        • I’m not suggesting anyone pays for all traffic per mile. The bit mile construct is meant to be the guiding mechanism for peers to interconnect their networks settlement free. It is a better system that ratios. A ratio of traffic sent and received between networks simply cannot act as a proxy for equal business value from an interconnect when one network is a global backbone selling synchronous services and one network is a consumer broadband network selling asynchronous services. Traffic will not be in balance at the interconnect because no where inside the last mile is traffic in balance – it is architected to be out of balance. As a broadband consumer you get more downstream bandwidth than upstream. So, bit miles are meant to provide a measure of equal business value for networks that are genuinely interested in settlement free interconnect.

      • If you decide to forward a packet for 1 million miles then thats your decision by design. There are lots of ISPs who have decided not to forwarded packets travelling that far and keep it within a certain region. However, if you do choose to forward it 1 million miles most peers would take that into account when defining “mutual benefit”. If they don’t, then simply don’t forward the packet.

        • I would describe it as a more subtle sort of discrimination. Because as you say the visibility to the broadband consumer of any interconnect degradation is more immediately apparent for certain applications.

        • Jørgen I don’t get to decide where a packet originates or where it goes to. Now it is true that I control how I physically build my network. I don’t build things in straight lines between cities. And I may build long circuitous routes for all sorts of reasons. And that is why our bit mile construct defines miles based on the straight line between where a packet enters and leaves each network. That way all network operator decisions about how they architect their network don’t need to complicate things. And we don’t get into the arguments that may ensue when one operator doesn’t agree with his the other built their network. The Internet routing protocols take over and decide how a packet gets from a to b.

  40. Pingback: Level 3 charges 6 unnamed ISPs of not adhering to their peering arrangements | Solus Network Solutions

  41. Pingback: Level 3 Accuses Six ISPs of Not Updating Networks to Ease Congestion | Telecom Judge

  42. Pingback: The people who run your ISP are putzes | Beeblehead

  43. “Shouldn’t a broadband consumer network with near monopoly control over their customers be expected, if not obligated, to deliver a better experience than this?”

    The question contains it’s own answer. When you have a captive market, you can treat them like dirt. For example in Jersey City, for years I subscribed to Comcast who had taken over the defunct Jersey City Cablevision franchise, which had frequent issues and service fails. When Verizon moved into compete by bringing in it’s fiber service, they HAD to be better than Comcast to get people to switch over… and at least in this locality, they are. The experience even within the same company will vary heavily depending on whether or not they are the defacto monopoly in a given location.

  44. Instead of kicking them off of Level 3, why don’t you use the power of supply and demand to encourage change? How? Revise your pricing policy to charge severe penalties for connections that run above 90% capacity. For example: base price x ([average utilization]-0.9). A peering that ran below 90% average would pay [base price], but a connection that ran 95% would pay [base price] x 5.

    • Someone who is willing to let interconnects congest will not be willing to pay a penalty when they do.

      • Last time i checked contractual business rules would take over here, non-payment equates to loss of service,problem solved by simply charging premiums for above 90% utilization. Let them not pay, then shut them off. They will come around fast.

        At some point you have to stop using circular reasoning. “these companies do this for profit, over utilize the network, refuse to upgrade and cause problems, but we cant give you their names because we might lose profit”..pfft. cop outs is what i hear to protect your investors stock price.

        I am sorry but its hard for me to believe that the 40% packet loss Ive had to lvl3 in washington DC for the last week is all some anon, unscrupulous ISP i use (Suddenlink).

        • Clifton, we are trying to get to a place where the contracts that govern peering do contain some service levels with teeth. In other words that contain specific obligations as well as remedies should those obligations not be met. And in the vast majority of cases we have done that. Both Level 3 and our peer agree that it is in both of our interests, and of course those of our respective customers, that contracts are written in this way. But it takes two to agree to a contract. And if one party refuses to agree to those reciprocal obligations we cannot take the actions that you suggest as we have no legal basis on which to act. In addition, cutting someone off completely makes the situation far worse for everyone using the Internet and so isn’t a realistic remedy either. Hence the importance of governance in relation to these interconnections and why we have been writing about it.

          In terms of your specific example; Suddenlink manage their connectivity to the broader Internet very well. They buy connectivity off multiple providers and ensure that there isn’t congestion at those interconnection points. However, faults do still occur and we did see some packet loss in Washington last Sunday.

          • Two or three thoughts then based on the above, well after the fact.

            1) A REALLY good question if this is such an issue, is where was the ball dropped? This is a pretty humongous issue to have just arisen out of thin air, and I’d suggest it hasn’t. Why were contracts with decent clauses never considered previous that would have amended this issue? I ask, only because looking at current status, it’s pretty late in the game to push something that should have been there from day 1.

            2) It absoutely IS a solution. The reasoning you use I feel is somewhat akin to the bank bailout reasoning that was used. At least the very basic version explained to the public anyway. You can’t legally drop them because it ‘makes the situation worse’ and hurts your customers. But you can’t just let the situation stand because it ALSO hurts your customers every bit as badly. 40% packet loss (which is still a thing, see point 3), is just about as unusable as 100%. Except for certain EXTREMELY limited traffic, you might as well drop that router and find another way to transfer your data via 56k, because that’s about the speed you’re gonna get.

            3) As I said before, it absolutely IS a thing. Example: I was kind enough to send an email to the handler for ae-2-70.edge3.washington4.level3.net after denoting the other evening it was dropping about 30-50% of the packets sent through it to a fairly decently large gaming community (wargaming.net). It’ll likely be ignored and not fixed, or is incapable of being fixed at current, as a quick forum search or two uncovered that issues through that particular router have been a problem for, quite literally, over a year. It’s continuing as we speak, providing some pretty bad publicity to a group who are, when all’s said and done, the type of people aware of backbone providers and who likely work with you daily from ISP support and other fields.

            Overall, agreed, this isn’t an EASILY solvable problem, but the general thought I’m having here is, it’s not like there’s a whole lot of easy options for ‘them’ either. They either cave and agree to meet reasonable terms for interconnect, or are forced to go through a lot of work and expenditure to find ways to replace the connectivity you provide at best circumstance, or fail to find equitable relations elsewhere and fail or come crawling back.

  45. Pingback: Level 3 Communications Calls Out US ISPs for Intentionally Slowing Down Networks | Bright Side of News*

  46. Pingback: The future of the Internet, courtesy of Teddy and Franklin Roosevelt | PandoDaily

  47. So how much delay are your interconnects tolerating before they start dropping packets?

    (If it’s greater than 5ms, why aren’t you using RED or some other AQM or packet scheduling system?)

  48. Pingback: Wow great article on the connectivity of level 3 One of our many great partners | michaelhudak

  49. Pingback: Major internet firm blames ISPs for purposely slowing down web traffic | Mushing News

  50. As more and more content providers establish paid peering relationships with the last mile providers, the congested interconnections shown above will likely become less and less saturated, because the content providers will increasingly bypass the “middlemen” networks to deliver their content (ie.video) directly to the last mile providers. This will improve network performance and will result in improved customer experience. This has been happening for several years now and is rapidly gaining momentum now that Netflix is getting on board. Over the past 15+ years the price of raw IP transit has compressed over 100 fold, while last mile access prices have compressed far less if at all. There is no reason why this will change. Paid peering is how content will reach the end customers and the last mile providers know it.

  51. Mark, without mentioning names surely you can provide a table showing what the average and min/max cost/BB-sub a 10gig card/port represents for the 6 offenders. The avg ISP charges something like $50 for a BB connection. If the above illustrated ISP had 8m subs, what is the one-time cost per sub for expansion capital? $0.0025? A quarter of a cent?!? And therefore the total cost of connectivity can’t be more than a NICKEL per sub; one-time.
    Marginal costs (even estimated to 90% accuracy) at every layer and boundary point need to be widely broadcast to regulators (who are clueless) and the general public so they appreciate how much the end-user is being overcharged. It’s absurd to think I pay $60 in NYC for 30/5 and my Netflix continuously buffers with TW. I estimate they don’t even deliver on 10% of what they promise me (stock or flow).
    The 4K, 2-way HD collaborative, seamless mobile BB, and IoT trends aren’t just approaching; they are here and this FCC and industry are ill-prepared to deliver it to a hungry market.
    Good article,

  52. The answer is easy, you fine them for the congested peering interconnects. Any shared hosting provider on a server knocks your site down for more than 10% CU usage. They promise unlimited bandwidth, but it’s the CPU usage they boot you for. Your contracts need to stipulate HOW MUCH THEY CONGEST any given connection. Why your contracts don’t say this is ridicules on Level3’s part in my eyes, you allowed them to do it.

    • Two parties have to agree to any binding contract. I¹d love to have that sort of obligation in an interconnect agreement. And in fact the vast majority of the interconnect agreements we have work well and DO contain language that obligates both parties to augment the interconnects as they start to be utilized above a certain level for a certain period of time.

  53. Pingback: Los proveedores de internet, ¿se quedan sin ancho de banda? | Pablo Alejandro Fain

  54. So why does the link in your picture show you’re pushing out 80% traffic into the blurred out network and only taking in 20%? I’ve seen graphs like this where I work. This is what a transit connection FROM my company looks like and I pay a butt load for it. This doesn’t look like what I’d expect a peer to look like at all.

    • That is the nature of traffic flows across a last mile network (or a Content Consumption Network as a previous commenter more accurately refers to them). Broadband is an asynchronous service by design. Consumer consumption patterns further magnify that effect and so you end up with that sort of traffic flow through ports that the last mile network uses to connect to the rest of the Internet. This is the service they sell. And its the service their customers use in the way they do.

      • Kind of funny how for years, the ISPs provides such horrible UL/DL ratios because they could exploit the peering relationship and not have to pay for outgoing traffic, and now they’re complaining because their own customers are… downloading a lot! If UL was more available, then the whole issue with Netflix wouldn’t exist because you could make it work like Bit Torrent…

  55. Quick question regarding peering arrangements as they stand today. Could Level 3 sell bandwidth to a massive content provider, (say, Netflix or YouTube) for dirt cheap and use a peering agreement as the delivery mechanism?

    Let’s take your Dallas example. Imagine it was saturated because Dallas is where you peer with broadband ISP “A” on router “R”. You sell bandwidth to a massive content provider (let’s call them “CONTENT X”) for dirt cheap and tell them that they must connect to router “R” in Dallas and only send data to ISP “A”‘s address space over this connection. Level 3 basically then routes the traffic over router “R” and the traffic from “CONTENT X” doesn’t really hit Level 3’s network. Level 3 would essentially be selling bandwidth on ISP “A”‘s network via a peering agreement.

    If I were ISP “A” in this scenario, I’d be reluctant to add more ports too. I’d want the content providers to pay. Is this scenario actually happening? Perhaps not with Level 3, but are you familiar with this sort of scenario?

    • The majority of peering agreements do not allow selective announcements of the other parties’ routes and do not allow a “peer only” offer to be sold in the market. In addition, our recommended approach of a bit mile construct for peering would make this sort of behavior unlikely. The sender in your example would have zero bit miles and the receiver would probably have a lot of bit miles. So if that was the only traffic exchanged between the two parties then the one with zero bit miles would have to start carrying traffic further on its own network or otherwise pay the other party.

  56. Pingback: Level 3 Peering Dispute: Ports to 5 U.S. Internet Providers are Permanently Congested - Telecompetitor

  57. Pingback: Level 3 accuses Comcast, other ISPs of 'deliberately harming' broadband service - LamboArchie Blog

  58. Pingback: ISPs allow 'permanent congestion' and want cash to clear it up, says report - LamboArchie Blog

  59. Pingback: “Chicken” | A Game Played as a Child and by some ISPs with the Internet | netwatch

  60. Pingback: Five U.S. ISPs accused of 'deliberately harming' broadband speeds | eSoftware

  61. Broadband Internet Access to Homes and Small Business needs to be tariffed and regulated by the FCC, with technical standards for quality of service, and bandwidth contention ratios.

    The notion that the Internet is self-regulating is a farce because in America there are a small number of very large corporations providing Internet access.
    The consumer has no choice because in the 10Mbps+ access speed, it’s primarily the cable TV company that can deliver that.

    Longer term, fiber to the home should be built as a public utility. Locally accountable to the people it serves.
    It works for electricity, water, sewer and gas, why not for Internet access.

  62. Pingback: Internet providers continue to intentionally slow down speeds for additional money

  63. Pingback: Major Internet Firm Blames ISPs For Purposely Slowing Down Web Traffic – 7 May 2014 | Lucas 2012 Infos

  64. The last-mile monopoly in the United States reminds me of the Bell System monopoly. The Content Delivery Networks seem to be healthy with plenty of choice, competition, and are reasonably priced. Unfortunately, the last-mile Content Consumption Networks (residential home broadband) have monopolies over entire areas and don’t compete directly with other last-mile home broadband providers.

    These last-mile monopolies are what’s causing all the peering problems. This allows last-mile residential monopolies to extort high transit fees from Content Delivery Networks, because they’re holding residential customers hostage from the Content Delivery Networks being able to reach those customers.

    I believe it’s time to either breakup these last-mile residential monopolies or regulate them as a public utility. Just like Bell System.

  65. Is there any way that Level 3 can publish this information to allow the public and consumers to make better informed choices about ISPs? Information = power and it seems that Level3 has some valuable information that could benefit consumers.
    Peter D

  66. Pingback: Major internet firm blames ISPs for purposely slowing down web traffic

  67. Thank you for reinforcing with Data what most already knew. However, what is it that you are hoping that will happen from this information? The telecom Oligopoly have a strong hold on the consumers and our elected officials.Hence why the most hated companies by is customer base keep getting record profits.They have no choice.

    Besides Google Fiber expanding I don’t see anything changing for the consumers. The consumer is at their mercy. What can we do? Calls and letters to elected officials have not been effective when the Telecom Titans are making it rain for them.

    Do you endorse or are aware of any new open mesh type networks that the people could get behind to battle these titans? How do we go about getting access to your backbone and make community wireless last mile connections?

    Again thanks, but I have little faith on the internet getting better for the consumers.

  68. Can you provide some bandwidth comparisons between countries/providers? I would imagine traffic being higher in the US but I am not sure by how much.

  69. Pingback: An ‘internet middleman’ calls out six ISPs for letting connections slow down | My great WordPress blog

  70. Europe’s problem is most certainly “Deutsche Telekom AG”. There is no slower network available, it’s also the most expensive for its customers and the competition is limited. To drive this point home: try to watch some live stream through DTAG’s network, say Twitch or YouTube, preferably during the “prime times”, when most people are at home. You don’t have to believe me on this, just search the customer support forums of DTAG. Sometimes tracerouting a lagging connection helps speeding it up, I’m almost tempted to say “magically”. Though that’s not hard evidence, that they try to see whether you’re checking where something is lagging and then give that data stream a higher priority. To be clear: that’s no evidence at all, even if five traceroutes showed some lagging on an edge router of DTAG and then it suddenly becomes responsive, that could also just mean the traffic dropped. But it is a little fishy to see the lagging go away more often than not the second you have run those traceroutes and not come back for several hours.

  71. Pingback: NewsSprocket | Yet Another Reason to Hate Your Broadband Provider

  72. Pingback: Level3 is without peer, now what to do? | Tech Zone

  73. Pingback: Level3 is without peer, now what to do? | Bydio

  74. Glad to come across this blog. Very informative. It’s about greed. The big providers are more interested in next quarters profit than the nations economy and competitiveness.

  75. Are there any limitations based on where the traffic comes from or how much traffic comes from there (instead of just hardware that needs to be upgraded)? If so, maybe you guys can develop software similar to Tor, but which instead anonymizes where traffic comes from (without any extra routing, like NAT). With the cooperation of other Tier 1 providers, this could allow Tier 1 providers to enforce Net Neutrality themselves.

    • No one is treating packets differently dependent on where they come from, what they carry or any other discriminatory method. The sort of congestion I mentioned affects all packets in a completely in-discriminatory way. Its like you stepped on a hose pipe – all the water molecules are affected. Making them anonymous doesn¹t protect them from having to pass through the constraint.

      • Gotcha, so it’s not something being held against the video streaming at all… Though that’s where download performance is most likely to be visible for end users causing those companies to need to purchase that extra link to ISP’s.

        A lot of people are talking about this scandal as if it was a discriminatory problem instead of a general short-changing problem.

        • I would describe it as a more subtle sort of discrimination. Because as you say the visibility to the broadband consumer of any interconnect degradation is more immediately apparent for certain applications.

  76. Pingback: Your Broadband Company May Be Holding Your Internet Access Hostage - ACROSS THE FADER - BIZ

  77. Pingback: Level 3 alleges Deliberate Broadband Throttling - SLUniverse Forums

  78. Interesting post!

    How many *uncongested* peers of Level 3 are consumer/non-regional – do you have many good peers in Europe?

    • Additionally, does anyone know how the total capital expenditures for the entire history of building out their entire network compare between Tier 1 networks like Level 3, and last-mile consumer broadband providers, both in America and elsewhere?

  79. Pingback: Amazon, Facebook, and other tech giants fight the FCC’s proposed net neutrality rules | PandoDaily

  80. Pingback: The best way to protest net neutrality’s end is with an Internet-wide slowdown – The Guardian – internet – Google News | Internet Technology Business

  81. Pingback: Level 3 Accuses Six ISPs in Europe and US of Deliberately Slowing Traffic | Hosting Reviews

  82. Pingback: How the FCC Created Its Own Net Neutrality Mess | News In Marketing

  83. Pingback: graph of Netflix speeds shows the importance of net neutrality - Page 2 - MyLesPaul.com

  84. Pingback: Anonymous

  85. Well, nice post. Now we know that not Level 3 is to blame regarding the messed up connection in Hamburg (Germany) which leads to endless paket loss and other strange things….

  86. This is exactly what I has been taken note of from my home. Working with ip all my life, I noticed when the ip-provider wanted to sell moore bandwith, they put a filter on the youtube videos.

  87. Pingback: Internet Provider: ISPs Allowing Network Congestion - Computing Now | Newsfeed - IEEECS

  88. Mark:

    I liked your blog article on peering. It was very detailed and informative for those people who do not know how residential Internet services actually work. Most the people claiming to support net neutrality have no clue what it really means or the technical details of how different traffic types need to be treated differently. Bandwidth is not the only answer to the problem.

    I wish you would have shown the traffic graphs for both directions of traffic on those ports. As I recall the direction to you is not as saturated showing an unsymmetrical peering arrangement. As you know that asymmetry is at the crux of the issue between ISP and its peers. The 5 U.S. ISP feel that they are carrying the burden to support this traffic which is only a half-truth. Peering arrangements were made based on the value of connectivity to as many nodes as possible and near equal sharing of traffic. Residential ISP traffic is anything but equal, but the desire for connectivity is still valuable.
    The value in their service is based on decent interconnectivity; that IS their service. By allowing the connections to remain saturated they diminish the value of their service to their customers, but fixing the problem requires real capital expenditures. Ironically by reducing saturation they will improve the quality of service for their competitors. I’ve work with these ISP (and Level 3) and I know the reluctance that they have to add peering bandwidth because they see it as an expense that they cannot offset. They don’t accept it as the cost of doing business.

    Throwing bandwidth at video will not solve the problem alone. Bandwidth and frame loss ratios are only two parameters of the equation. Latency and jitter play into good quality service as well. Since today everything is best-effort, all traffic is queued in the same fashion only the video more quickly fills up queues than web browsing and e-mail. If we continue to treat all traffic the same the cost of expanding the queues will surpass bandwidth costs. Proper traffic management is necessary to provide a good QoE for all services. Traffic prioritization is offered today with managed Ethernet services, and it should be propagated down to residential services. Giving a higher priority to all voice then video traffic will ensure a good experience for OTT services while improving the flow of best-effort traffic. The result will be a more efficient use of bandwidth; thereby, reducing saturation and better equipment utilization. This is the component that we desperately need to give OTT providers a chance to compete with incumbents.

    I’d like to see the ISP step up and upgrade their peering connections to the major backbone providers and implement QoS on time-sensitive services utilizing the MEF framework. Yes some OTT providers may have to pay an incremental bit more to have their QoS marking honored, but we will all end up paying more if we attempt to just throw bandwidth at the problem.

    Best Regards,

  89. HI Mark,

    In the peers’ table, where is “Sao Cristavo” located? The name looks like São Cristóvão, a borough of the city of Rio de Janeiro. Since Rio is not listed and seems to be Level 3’s entry point in Southeastern Brazil, I assume it means Rio? Perhaps Level 3 has offices in that borough?

  90. Pingback: How Net Neutrality helps your pocketbook. | Cluui

  91. Pingback: Net Neutrality: How the Internet Works | iamronen

  92. Pingback: Level 3 claims six ISPs dropping packets every day over money disputes | Tek-Dude Systems

  93. Pingback: Comcast impondría límite de 500 GB de consumo dentro de cinco años en EE.UU. | Enfoque IT

  94. Pingback: Comcast impondría límite de 500 GB de consumo dentro de cinco años en EE.UU. | Misiongeek

  95. Why is it that CDN providers have no problems paying last mile ISPs for access? Why was Level3 not willing to do the same when they attempted to become the CDN for Netflix? Why did Netflix choose Comcast over Level3? Was Level3 just not willing to pay for the infrastructure to properly support its own CDN services for Netflix?

    • Jonathan, my post attempts to describe the scale and scope of our network. Yes we do operate a CDN but that is only part of what we do. We own and operate a huge global infrastructure that both sends and receives traffic through our network interconnection points. Traffic that we carry all over the world. We are willing to bear our fair share of the cost of moving bits. Our suggested mechanism for assessing fairness is something we have referred to as bit miles. I have described this in response to other questions above. We believe this is a fair and honest way to assess whether two network operators receive equal business value from interconnecting their networks. And this is particularly important when one of those networks is asynchronous in nature – a simple ratio of send to receive traffic (which I think is what you had in mind when posing your question) cannot be used to assess equal business value. It simply isn’t possible to get in balance. And it is also worth pointing out that with our bit mile construct, owning a CDN works against me as I end up carrying those bits less far.

  96. Pingback: ... and points beyond » Observations of an Internet Middleman

  97. Pingback: Thoughts of a Raving Geek | Net Neutrality and the Similarities to US Health Care Reform

  98. Phenomenal post, Mark. I’m especially grateful for your persistent engagement with the questions that arose after your article. I’m much more informed after reading all of this. Thanks. (By the way, are you on Google+?)

  99. Pingback: Observations of an Internet Middleman | Level 3 | Backfill for 'Note to Self'

  100. Pingback: When the Middleman and ISP are Aligned | Beyond Bandwidth

  101. Question I live in one of the few markets in the us where there is at least a little competion albeit a duopoly (fios and optimum) how can I test who has a better connection for me in terms of packet loss and congestion and In your view are these regions in the us better even with the limited competion that there is.

  102. Pingback: Internet non va? Level 3: colpa degli ISP. Ma è tutto vero?

  103. Pingback: Yes, Your Internet Is Getting Slower | VantageWire

  104. Pingback: Google Backs Netflix in Epic Battle With Comcast | News In Marketing

  105. Pingback: Google Backs Netflix in Epic Battle With Comcast | Brain Cage

  106. Pingback: Is the FCC purposely making their comments section unavailable? | inventropy

  107. Pingback: The Internet has stopped growing. | Beno Chapman

  108. Pingback: Verizon Won’t Use Netflix’s Hardware to Boost Streaming Speeds - TIME

  109. Pingback: Verizon, Netflix Spar in Epic Battle Over Who Should Pay for What | HELIK.ES

  110. Pingback: Peering agreements should be transparent | ARTS & FARCES internet

  111. Pingback: Four short links: 7 May 2014 - O'Reilly Radar

  112. Pingback: What Everyone Gets Wrong in the Debate Over Net Neutrality « Reviews Technology

  113. Pingback: What Everyone Gets Wrong in the Debate Over Net NeutralityCoaster World News | Coaster World News

  114. Pingback: What Everyone Gets Wrong in the Debate Over Net Neutrality - Wired - Unemployed Computer Geeks California (714)975-3656

  115. Pingback: What Everyone Gets Wrong in the Debate Over Net Neutrality – Wired | News Update

  116. Pingback: What Everyone Gets Wrong in the Debate Over Net Neutrality | The Washington Observer

  117. Pingback: The biggest flaw in the net neutrality debateGet World News | Get World News

  118. Pingback: The Father of Net Neutrality Returns to Do Battle With Comcast – Wired | Everyday News Update

  119. Pingback: The Father of Net Neutrality Returns to Do Battle With Comcast – Wired | NEWS HEADLINES

  120. Pingback: The Father of Net Neutrality Returns to Do Battle With Comcast - TOP NEWS

  121. Pingback: The Father of Net Neutrality Returns to Do Battle With Comcast – Wired | News Station Online

  122. Pingback: The Father of Net Neutrality Returns to Do Battle With ComcastFind Latest News | Find Latest News

  123. Pingback: Heads ISPs Win, Tails You Lose (And a way to fix it) | Beyond Bandwidth

  124. Pingback: SDN: ¿Cuál es el próximo paso? | Beyond Bandwidth

  125. Pingback: SDN: O que ainda virá? | Beyond Bandwidth

  126. Pingback: SDN: ¿Cuál es el próximo paso? | Bienvenidos al Portal de la Revista IT/USERS

  127. Pingback: Regulators Want Comcast to Cough Up Lots More Data About Its Internet Policies | Re/code

  128. Pingback: Heads ISPs Win, Tails You Lose (And a way to fix it) - Beyond Bandwidth

  129. Pingback: SDN: ¿Cuál es el próximo paso? - estamos en linea

  130. Pingback: Fast hyperlinks | PakUltimate

  131. Pingback: Net Neutrality in Africa | Many Possibilities

  132. Say i wanted to start an isp and offer gigabit how much would it cost to lease access to your network and where would i connect to it ?

    • Ryan, while I cant talk about pricing on a blog, the opportunities to connect are huge across most of the world. In the United States, Level 3 has fiber in a very large number of cities.

      • Yes, I’ve noticed this in the past on Level Three’s main website as well while doing research for business concepts. Is there some legal reason why so many peering providers follow the seemingly shady professional practice of refusing to publish their service prices in any easily accessible location?

        • Sean peering isn’t a product that is sold. It is a relationship between two network operators who decide to allow traffic between their respective customers to flow through a network interconnect. Each network gets paid by their own customer. But the traffic flowing through the peering interconnect is usually settlement free. Each network has slightly different rules they use to assess whether or not to peer. And those are usually found on their web site.

  133. Hi Mark,

    Thanks for that detailed post and patient responses in the thread. I have a question regarding the traffic.

    Isn’t is true that the same big providers you didn’t name use the same peering connections to carry their business class traffic with more stringent SLAs?


    • Anujitam, yes. However the Internet service is sold (consumer, business, any speed etc) by an ISP, all traffic associated with those services goes through the same interconnections to other networks.

  134. Thanks Mark for the informative post. I have a friend in europe that pays about $8/month USD. And they have like 12 ISPs to choose from, they get 100Mbps download speed, contrast that with $60/month is US for a dismal 10Mbps.

    Bottom line is we need competition to improve things. Now comcast merging with TWC will make even less choices. We as consumers are suffering in the US because of these monopolistic companies, if we had 12 ISP choices I bet we would have better, faster networks and a much better price.

  135. Pingback: Why America’s Internet Is So Shitty and Slow | VoxBox

  136. When talking about peering and the large ISPs are you including last mile providers like the Cable Companies who for the most part are paying customers of Level3? In most cases the cable companies are not reselling their service as “transit” but normal internet connectivity or access to end users. Meaning little traffic is going out of network and the bulk of the traffic is coming in.

    In a case where more then 50% of transit traffic purchased by a last mile provider is OTT-video why is it incumbent on the last mile provider to spend the money to increase transit connectivity to Level3? That increase at the transit level has to be matched all the way down to the customers access. For the CDNs and the transit provider its simply a matter of adding a port to meet the demand. This doesn’t even consider the fact that the added bandwidth will be eaten up by variable bit rate video streams.

    Why are consumers not being asked to pressure the OTT-video operators to figure out a better arrangement for their product into these last mile providers networks? And why shouldn’t they pay for the access they are receiving into those networks? I don’t believe this is simply an issue of last mile provider unwillingness to upgrade transit connectivity.

    I also believe its disingenuous for Level3 to call out these companies and not disclose the financial benefit Level3 would receive by last mile providers upgrading all of their transit connectivity around the country. It would certainly include leasing “waves” from Level3 to ensure their network is not saturated. For Level3 pushing this means better business and higher revenue, but calling other businesses who have other costs to way chickens is slanderous at worst and half the story at best.

    • Johnson, I’m talking about the largest broadband providers. They all sell transit too.

      Yes you are right traffic is more heavily weighted towards the paying customers of those companies rather than the other way. That’s because broadband is designed that way. It isn’t actually possible for the traffic to reverse. And as you say that is also how people are using that service. That is what they pay those companies to do.

      However, imagine that the traffic did reverse. Ignoring for a moment the structural asymmetry of broadband, the ISP cost remains the same. Because what matters to a network operator isn’t which direction traffic goes it is the distance it is carried. That’s what drives costs.

      When that ISP connects to Level 3 there is a network on the other side. The Level 3 network spans the globe, crossing land and oceans. We carry traffic that is requested by the ISP’s paying customers. And we hand it to that ISP at the interconnect point for delivery of that last mile. Yes it is simple for both of us to add that interconnect. But both of our networks also have to carry that traffic on either side of that interconnect. Again direction doesn’t drive our cost. Distance does.

      So if we are trying to have a balanced business relationship. One where both of us share the burden equally what matters is how far we EACH carry the traffic. Something we call bit miles. And in every case we carry traffic at least as far as every ISP we connect to. Even taking account of the traffic sent from our CDN, which as you rightly say can in many cases be very close to the interconnect point.

      And don’t forget traffic can only flow through those interconnects between a paying customer of Level 3 and a paying customer of the ISP. We both get paid. While it is true that an ISP bit mile costs more to build than a bit mile in the Level 3 network that also is partly why a broadband consumer pays a huge amount more for their bandwidth than a Level 3 customer does.

      And yes I do think an ISP should invest in their network to deliver the service they sold to their customers. What is also overlooked by many people is that the cost of the equipment needed to build all these networks has been falling faster than the increase in traffic. But ISP broadband prices haven’t fallen. Where you can see the profitability of ISP businesses in public filings they have been increasing.

      On your last point an ISP may buy service from Level 3 to build their network as an alternative to constructing some of their own network. But they buy those services in a massively competitive market. One in which we are happy to compete. That is in stark contrast to the lack of competition for the consumer broadband services that the ISPs sell. Most consumers in the U.S. only have a single choice of high speed wired broadband. And that’s no competition at all.

  137. Mark,

    great article! thanks for pointing out this problem.

    I am a researcher thinking some ideas about congestion at peering links and I wanted to ask you a clarification. Does this problem appear in settlement free peering links over Internet eXchange Points? Or is it just on direct peering links (not in IXPs)? In IXPs it could presumably affect 3rd parties too.


    • Fontas, I have very little data for IXPs as we make very little use of them. All of the data I have made available relate to private peering relationships. Some of the physical connections make take place in an IXP facility but are not made through the public peering fabric.

  138. Pingback: Tim Wu,The Father of Net Neutrality Returns to Do Battle With Comcast - itbusinessdirectitbusinessdirect

  139. Pingback: Internet Over Packet Loss | TechSNAP 162 | Jupiter Broadcasting

  140. Pingback: Google Backs Netflix in Epic Battle With Comcast - Lunarsoft

Leave a Reply

Your email address will not be published. Required fields are marked *

Time limit is exhausted. Please reload the CAPTCHA.