Verizon’s Accidental Mea Culpa

David Young, Vice President, Verizon Regulatory Affairs recently published a blog post suggesting that Netflix themselves are responsible for the streaming slowdowns Netflix’s customers have been seeing. But his attempt at deception has backfired. He has clearly admitted that Verizon is deliberately constraining capacity from network providers like Level 3 who were chosen by Netflix to deliver video content requested by Verizon’s own paying broadband consumers.

His explanation for Netflix’s on-screen congestion messages contains a nice little diagram. The diagram shows a lovely uncongested Verizon network, conveniently color-coded in green. It shows a network that has lots of unused capacity at the most busy time of the day. Think about that for a moment: Lots of unused capacity. So point number one is that Verizon has freely admitted that is has the ability to deliver lots of Netflix streams to broadband customers requesting them, at no extra cost. But, for some reason, Verizon has decided that it prefers not to deliver these streams, even though its subscribers have paid it to do so.

The diagram then shows this one little bar, suggestively color-coded in red so you know it’s bad. And that is meant to be Level 3 and several other network operators. That bar actually represents a very large global network, and it should be shown in green, since, as we will discuss in a moment, our network has plenty of available capacity as well. In my last blog post, I gave details about how much fiber and how much equipment we deployed to build that network and how many cities around the globe it connects. If the Verizon diagram was to scale, our little red bar is probably bigger than their green network.

But here’s the thing. The utilization of all of those thousands of links across the Level 3 network is much the same as Verizon’s depiction of their own network. We engineer it that way. We have to maintain adequate headroom because that’s what we sell to customers. They buy high quality uncongested bandwidth. And in fact, Verizon admits as much because they conveniently show one direction across our network with a peak utilization of 34%; almost exactly what I explained in my last blog post. I can confirm once again that all of those thousands of links on the Level 3 network are managed carefully so that the peak utilizations look very similar to those Verizon show for their own network – IN BOTH DIRECTIONS.

So why does Verizon show this red bar? And why do they blame Level 3 and the other network operators contracted by Netflix?

Well, as I explained in my last blog post, the bit that is congested is the place where the Level 3 and Verizon networks interconnect. Level 3’s network interconnects with Verizon’s in ten cities; three in Europe and seven in the United States. The aggregate utilization of those interconnections in Europe on July 8, 2014 was 18% (a region where Verizon does NOT sell broadband to its customers). The utilization of those interconnections in the United States (where Verizon sells broadband to its customers and sees Level 3 and online video providers such as Netflix as competitors to its own CDN and pay TV businesses) was about 100%. And to be more specific, as Mr. Young pointed out, that was 100% utilization in the direction of flow from the Level 3 network to the Verizon network.

So let’s look at what that means in one of those locations. The one Verizon picked in its diagram: Los Angeles. All of the Verizon FiOS customers in Southern California likely get some of their content through this interconnection location. It is in a single building. And boils down to a router Level 3 owns, a router Verizon owns and four 10Gbps Ethernet ports on each router. A small cable runs between each of those ports to connect them together. This diagram is far simpler than the Verizon diagram and shows exactly where the congestion exists.


Verizon has confirmed that everything between that router in their network and their subscribers is uncongested – in fact has plenty of capacity sitting there waiting to be used. Above, I confirmed exactly the same thing for the Level 3 network. So in fact, we could fix this congestion in about five minutes simply by connecting up more 10Gbps ports on those routers. Simple. Something we’ve been asking Verizon to do for many, many months, and something other providers regularly do in similar circumstances. But Verizon has refused. So Verizon, not Level 3 or Netflix, causes the congestion. Why is that? Maybe they can’t afford a new port card because they’ve run out – even though these cards are very cheap, just a few thousand dollars for each 10 Gbps card which could support 5,000 streams or more. If that’s the case, we’ll buy one for them. Maybe they can’t afford the small piece of cable between our two ports. If that’s the case, we’ll provide it. Heck, we’ll even install it.

But, here’s the other interesting thing also shown in the Verizon diagram. This congestion only takes place between Verizon and network providers chosen by Netflix. The providers that Netflix does not use do not experience the same problem. Why is that? Could it be that Verizon does not want its customers to actually use the higher-speed services it sells to them? Could it be that Verizon wants to extract a pound of flesh from its competitors, using the monopoly it has over the only connection to its end-users to raise its competitors’ costs?

To summarize: All of the networks have ample capacity and congestion only occurs in a small number of locations, locations where networks interconnect with some last mile ISPs like Verizon. The cost of removing that congestion is absolutely trivial. It takes two parties to remove congestion at an interconnect point. I can confirm that Level 3 is not the party refusing to add that capacity. In fact, Level 3 has asked Verizon for a long time to add interconnection capacity and to deliver the traffic its customers are requesting from our customers, but Verizon refuses.

Why might that be? Maybe we should ask David Young.

The following two tabs change content below.

Mark Taylor

I work as VP of Content and Media here at Level 3. English expat and passionate new tech energy evangelist.

Latest posts by Mark Taylor (see all)

239 thoughts on “Verizon’s Accidental Mea Culpa

  1. Seems like the Verizon post clearly states the reason for the “bottleneck” called out here. Seems like they want to get paid when there is a traffic flow imbalance. Makes sense to me.

    > … took steps to ensure that there was adequate capacity for their traffic to enter our network. In some cases, these are settlement-free peering arrangements, where the relative traffic flows between an IP network provider and us remain roughly equal, and both parties invest in sufficient facilities to match these roughly equal flows. That is the traditional basis for such deals. In other cases there may be traffic imbalances, but the networks or content providers have entered into paid arrangements with us to ensure connections and capacity to meet their needs for their out-of-balance traffic.

    • Sanjay if the traffic flow through that Los Angeles router is 40Gbps from Level 3¹s customer to Verizon¹s customer and 10Gbps in the opposite direction then both networks carry 50Gbps. What matters then is how far we each carry it. If Level 3 carries it 800 miles and Verizon carries it 80 miles then Level 3 incurs a higher burden of cost – ten times in this example. If the direction of traffic then reverses our respective costs are unchanged as we both still carry 50Gbps, and Level 3 still carries it 800 miles and Verizon still carries it 80 miles. If the distance reverses, however, then our respective costs do change. So bits multiplied by distance (bit miles) determine costs. Level 3 is more than happy to incur its share of that cost. I appreciate that traffic ratios were used as a proxy for cost equality between backbone network peers historically. And that can work well because both networks are synchronous and likely have the same business model. But it completely fails to work as a proxy for shared costs when a synchronous backbone like Level 3 connects to an asynchronous broadband consumer network like Verizon. It simply isn¹t possible to get in balance even if balance was a measure of cost equality – and it isnt. Enforcing balance in these circumstances is, in our view, a way of arbitrarily raising a toll.

      • “I appreciate that traffic ratios were used as a proxy for cost equality between backbone network peers historically … Enforcing balance in these circumstances is, in our view, a way of arbitrarily raising a toll.”
        In other words, you want to change the way interconnect business arrangements have been done in the past because it disadvantages you because you are the primary source network for probably the biggest sender of data in the history of the world. Not to mention that Netflix is paying you a lot for all that access bandwidth to your network.
        In this case, Verizon is quite explicit in that post that they either want you to pay them for the traffic imbalance or have Netflix buy access directly to their network.
        This does not seem unreasonable to me.

        • John, interconnect agreements between backbone networks used traffic balance as a proxy for each network equally sharing the cost of delivering traffic between their paying customers. When two synchronous networks interconnect and have much the same business model (sell the same tyes of serviecs to the same types of custoemrs) then this proxy generally works.

          But when a syncronous network connects to an asynchronous network, traffic balance no longer works as a proxy for an equitable sharing of costs between those two networks. So if two such networks want to interconnect, and genuinely want to find a way of eqitably sharing the cost of delivering traffic between each of their paying customers, then a different methodology is required. We have suggesed that ³bit miles² should be used. That measure combines how much traffic is exchanged through the interconnect point and then how far each network carries it.

          That is what should be in balance if, as I say, the goal is an equitable relationship. Level 3 is most certainly not asking for a free ride here.

          We fully recognise we have to use our network to carry traffic flows between our paying customer¹s and an ISP¹s paying customers. And we should bear at least half of that burden.

        • Netflix is not a “sender” of traffic. Netflix does not arbitrarily spew traffic on to the internet. Verizon’s customers request this traffic from Netflix and pay Verizon to deliver it to them. Verizon told their customers “You can have this much bandwidth and use it all you want!” and now they don’t want to maintain the infrastructure to support it.

          Verizon (and almost every last mile ISP) sells their customers asynchronous connections, so if every customer was at 100% utilization there would be far more traffic flowing in to their network than out… duh. Don’t be a troll.

      • “If Level 3 carries it 800 miles and Verizon carries it 80 miles then Level 3 incurs a higher burden of cost – ten times in this example.”

        Level 3 has long and narrow networks from data center to data center, Verizon has wide and shallow networks. Yes, it’s 80 miles to the specific end user, but multiply that by the number of accessible users that Verizon needs to be ready to deliver that traffic to and I think you’ll find that the comparative number of miles is vastly different than 10:1.

        • Tom that’s a reasonable point. But don’t forget we both have a different product and a different commercial relationship with our respective customer. The architectures that each network operator deploys maybe different. In our view that should be factored into how Level 3 charges for Internet Services and how the ISP charges for consumer broadband services.

          • While you see Verizon as playing ‘dirty’ against CDN’s such as Level 3 etc Verizon are also leveraging this situation of THEIR making to upsell their service to customers. I have what would be a fairly fast data service of 75/35 but I get buffering all the time. I was told that I didn’t have enough bandwidth for the number of devices I had on my network at home and needed to pay for higher speed service from them.

            I pointed out that at most Netflix uses at maximum 6 Mbps and that I can only watch one thing on Netflix and can only use one mobile device at a time or one PC at a time and that forcing customers to use their router so that they can access the router’s settings etc and know how many devices are connected and making wild claims about how people actually use their internet to try and force them to purchase a more expensive package. Heck I pay $95 a month just for internet and that people that use the only local rival for Broadband, Time Warner Cable, who have a maximum locally of just 20 Mbps can have several people using multiple items and streaming to multiple devices can all use it without buffering but myself alone, no other person there, with 3 3/4 times the bit rate for downstream cannot have an iMac on idle just doing email, a tablet sitting on standby and a Nexus 5 just sitting there and a single Chromecast streaming a movie BUT myself with all that ‘extra’ available download bandwidth gets buffering not only on Netflix but on other content streams and feeds such as dailymotion, youtube, cachefly etc I’ve even suffered buffering on the Netflix stream that is fed separately through an Apple TV.

            Their response is purely “No we know that you are using straining the capacity of your internet package and need to upgrade”. In fact 5 of the devices they listed or claimed as actively using the internet at my home at the time of the call were with their owners in the UK or devices that had been given away months before!

            Sorry for the rant, Verizon have been a sore point for the past 5 months with the games they have been playing. I thought I was alone but have spoke to others who have had the same issue with Verizon FiOS and ‘mystery’ buffering.

          • Stew, just a correction to the first sentence. We do own a CDN. Verizon does as well btw. But we are a global network operator that built one of the biggest IP networks spanning the globe. Billions of dollars have been spent digging trenches, laying fiber, putting cables under the world’s oceans and then lighting that fiber to carry internet traffic around the world.

          • So you want them to push that cost to their customers, regardless of whether or not that particular customer cares to use Netflix. Wouldn’t it make more sense to force the cost on to Netflix, so that Netflix’s customers pay for it? More specifically, shouldn’t this be a negotiation between the companies involved (such as yours) and not an issue for Congress or the FCC?

            The only real issue I see here is the monopoly so many companies such as Verizon have on the last mile.

          • Ken, what I described was an equal sharing of the burden of moving bits from the point of origination to the point of consumption. As the prior poster pointed out, the cost per mile for the last mile network operator may be higher than the cost per mile for a backbone network. And my response was that partly explains why the price of backbone bandwidth services are a tiny fraction of the price of consumer broadband services. Of course the other reason is that the backbone market is extremely competitive. But in the consumer broadband market 77% of Americans have a grand choice of a single provider. Hardly the greatest check on prices.

            Another interesting fact, forgotten by many, is that the cost of the equipment needed to build those networks has actually been falling faster than the growth in traffic. Where public data is available it is clear that, despite all the protestations from the consumer broadband operators, their profitability has been increasing as their capital intensity decreases.

            Prices in the highly competitive backbone market have fallen by at least 20% every year for the last decade. Prices for the somewhat less competitive market of consumer broadband only seem to go up.

        • I think that is why they are using “bit-miles” not just miles. Let’s simplify. Say Level 1 carries 1Gbps for 100 miles, or 100 Gbit-miles and Verizon has 1000 customers paying for 10Mbps 10 miles from their router. They would each carry the same amount of “bit-miles”, as verizon would then carry 1000x10Mbps*10miles=100Gb-miles.
          That means that while they have different distribution models for data, they carry the same amount of data.

      • Can you clarify more on how bitmile billing would work? It almost sounds like you’re saying that Verizon should be writing *you* a check, since you’re covering so much more distance than them.

        Generally the goal is to decrease physical distance. i.e. static content like Netflix movies shouldn’t necessarily be making an 880 mile trip when they could instead be served from a CDN node right at the edge of Verizon’s network, 80 miles from the end user.

        I’m afraid bitmile billing might set up a bad incentive to do (or continue doing) things wrong.

        • Mike good question. My example was simply to expose the problem with ratios, not to suggest that they are actual numbers. A bit mile agreement, however, should contain obligations on each party to ensure that there is always balance. That there never needs to be an invoice raised by one party to the other. If a network is carrying less bit miles there are things each network can do to rebalance the bit miles such that the network that was carrying less now carries more. But it is also important to make that a choice. There are instances where one network may not want to make the change. Adding extra interconnect points is one way of re-balancing bit miles. It maybe that one network operator doesn¹t want to do that. So each agreement does include the option to pay something if they choose to do so rather than get back into balance. We think that choice is an important part of making these agreements endure through unforeseen changing circumstnces.

          • All bit miles were not created equal. It probably costs an ISP many times more to transfer a bit across town than it costs you to transport it across the country.

          • Ray, that may be correct. But we also sell different services. Level 3 sells Internet Services to businesses. Verizon sells broadband to consumers. Our commercial models are different. Each of us should have a model that enables us to provide the network we need for that service.

            That removes the necessity to get into the argument over how a network operator decides to build their network.

      • This part…
        where the relative traffic flows between an IP network provider and us remain roughly equal
        …is merely a policy. It is a common policy for internet backbone carriers. In this case where Verizon is a last mile ‘consumer eye balls’ network it would be ridiculous to assume the traffic would ever be equal and the only plausible reason to have such a policy would be to hide behind it to provide justification for extorting additional fees.

      • As the founder of a small ISP, I am dumbfounded why these guys think they can charge carriers. I hope they keep these charades up as it is business for me.

      • I have some interest in Verizon’s network performance as it relates to Netfilx.
        Mr. Dave Young, is Verizon’s chief analyst on this matter and available to discuss it with anyone directly interested in contacting Verizon.
        Although I cannot vouch for its content: Please read the following public policy article which was published on the Verizon website back on July 10, 2014:

        It appears to be a business decision not to upgrade their own equipment just to suit competitors needs. Is that the issue?

        • Yes, that is the article that my post was referring to. It certainly is a decision taken by Verizon but one that damages their customer¹s Internet experience. I believe that Verizon¹s broadband customers expect that their monthly payments give them unfettered access to everything on the Internet on a non-discrimnatory basis.

          • In your position you must know that the monthly payment for a 100mpbs line from Verizon is nowhere even close to the cost of a dedicated line. That expectation is just plain ridiculous, if Verizon customers expect that their monthly bill is going to go up by an order of magnitude.

          • Ken, it certainly is my expectation as a broadband consumer that I should be able to access all (legal) content. And that I should be able to do that on a non-discriminatory basis. I don’t want my broadband provider deciding what I can see or deciding to give priority to content they like (or maybe that they even produce) over other content. And that expectation in no way requires a dedicated 100Mbps connection.

    • Also keep in mind, Level 3 isn’t just randomly sending Netflix traffic to Verizon. Verizon’s customers are asking for the Netflix traffic to consume. Verizon’s customers are paying for Verizon to deliver any and all traffic from the Internet. If Verizon can’t deliver the Internet to its customers at the speed promised by Verizon, then Verizon should be the one trying to fix the problem.

      • Exactly. The fault here is Verizon’s unwillingness to shoulder the costs of delivering the quality of service they advertise to their customers. Unless their terms of service specify exactly what type of content they are willing to provide, they should treat all customer requested inbound traffic equally, regardless of origin.

    • My ISP sends way more data to my home network than my home network sends to my ISP. Maybe my ISP should be paying me for access to my network.

      I love this “imbalance” logic.

    • Maybe Verizon should stop selling asynchronous bandwidth packages with massive download speeds and smaller upload speeds if they want relative traffic flow to remain even. If you sell a bandwidth package that looks something like 75 Mbit down/15 Mbit up, you shouldn’t be alarmed when you have more download than uploading going on.

      Also, keep in mind that this is the nature of the Internet in general. Most people who use the Internet are content consumers, not content producers, so naturally most last mile customers will be pulling more download than upload.

    • Every consumer ISP will have traffic flow imbalance, because consumers download more than they upload. That’s the nature of consumer network usage, and does not in anyway justify charging backbone networks to transmit traffic to the end consumer over the ISP’s local network. That local network was paid for by the consumer.

      Additionally, P2P applications such as bittorrent are the one thing that can cause consumer traffic flow to become symmetric, and ironically large ISPs complain about that too.

  2. Wait,
    V says, “In other cases there may be traffic imbalances, but the networks or content providers have entered into paid arrangements with us to ensure connections and capacity to meet their needs for their out-of-balance traffic.”

    L3 says, “Could it be that Verizon wants to extract a pound of flesh from its competitors, using the monopoly it has over the only connection to its end-users to raise its competitors’ costs?”

    So is this a case of extortion?

  3. Pingback: Internet congestionata (ISPs vs Level 3)

  4. Excellent post, Mark. I work for a major LA telecom and it’s a pretty open secret that we throttle all of our customers with the exception of the highest payers. It’s sad, because we could easily offer the same level of service across 95% of the customer base (there are some exceptions where bandwidth will be constrained based on location or use), but really it’s a money grab.

    • Do some good for the world – gather evidence, blow the whistle. It’s ridiculous that greed is so bad that so much capacity is being shut off from people that could very easily be provided. You can do it anonymously if your evidence is sound.

  5. Spot on, Mark.

    Zero technical reason why this problem exists – it’s being entirely manufactured by the ISPs.

  6. I hear what you’re saying Mark. But you don’t address the second half of David’s article:

    “What it boils down to is this: these other transit and content providers took steps to ensure that there was adequate capacity for their traffic to enter our network. In some cases, these are settlement-free peering arrangements, where the relative traffic flows between an IP network provider and us remain roughly equal, and both parties invest in sufficient facilities to match these roughly equal flows. That is the traditional basis for such deals. In other cases there may be traffic imbalances, but the networks or content providers have entered into paid arrangements with us to ensure connections and capacity to meet their needs for their out-of-balance traffic.

    That has not recently been the case with Netflix and the networks carrying its traffic. Netflix sends out an unprecedented amount of traffic. Sandvine recently noted that Netflix now accounts for more than one-third of all North American downstream traffic during peak hours. For whatever reason (perhaps to cut costs and improve its profitability), Netflix did not make arrangements to deliver this massive amount of traffic through connections that can handle it.

    Instead, Netflix chose to attempt to deliver that traffic to Verizon through a few third-party transit providers with limited capacity over connections specifically to be used only for balanced traffic flows. Netflix knew better. Netflix is responsible for either using connections that can carry the volume of traffic it is sending, or working out arrangements with its suppliers so they can handle the volumes. As we’ve made clear before, we regularly negotiate reasonable commercial arrangements with transit providers or content providers to ensure a level of capacity that accommodates their volume of traffic.”

    So when you say “But Verizon has refused.” in regards to adding more links between border routers, aren’t they really saying “You have to pay us more” to deliver such an imbalanced load? How is their desire to do this different from you stating “We have to maintain adequate headroom because that’s what we sell to customers. They buy high quality uncongested bandwidth.”

    If they allow those those 4 unused ports to be connected doesn’t their network off that border router jump to 100+% utilization and the traffic flowing from that increase the load on it’s interconnections?

    What’s the debate here? Is it that Verizon should have to freely carry Netflix’s traffic to consumers because consumers pay for use of Verizon’s bandwidth?

    Let me give a comparison: I pay out of my taxes for the government to maintain roads and bridges that I drive on. This doesn’t provide me 100% free use however, I still pay tolls. Businesses want to use these roads for shipping and to do that they use 18 wheelers to deliver. Because these 18 wheelers have a bigger impact on the roads, they pay a higher toll. If all of a sudden the waters dried up the sky turned to fire and all businesses could rely on for transport was the roads to ship, there would be so many more 19 wheelers on the road. Do you think the average citizen would want to pay for the roads to be repaved monthly due to the stress of this traffic? No…they would expect the companies to pay for it and for the governments to somehow regulate/arrange that. Is that fair for the company? After all, they say “The customer is paying their taxes for us to use the roads to sell them goods they want.” I say no dice. The companies should pay and, should therefore likely have to raise their prices of goods to the consumer.

    Netflix should raise their prices to pay for more bandwidth. Will they loose customers? Maybe. Where will those customers go? Maybe to the next company that can figure out how to compress high definition video to 1/3rd the size that Netflix uses. That will spur innovation and competition the things that drive the economy.

    Is the problem here that L3 promised a certain level of access to Netflix through its provider that it can’t no deliver based on the sheer capacity of it? If so, L3 signed a bad contract, and likely have to honor it until renegotiation time. L3 shouldn’t be the one paying in the longterm for this…or even throwing their hat in the ring. Netflix should pay you to deliver their content just like I have to pay for my upstream bandwidth for my remote connections, you pay that to Verizon to increase your link with them.

    It isn’t like Verizon isn’t paying money to deliver services to users…they have to deliver their infrastructure to the end user and that costs them. Netflix shouldn’t think it doesn’t need to increase it’s distribution costs as it increases users.

    • BH, we did address that issue in both comments to my prior posting on this subject as well as Mike Mooney’s last post. If two parties are genuine in wanting to share the cost of moving traffic between their respective customers then traffic ratios or blanace of traffic are irrelevant when it comes to connecting to a last mile network. In fact it isn’t even possible to be in balance with a network operator who sells an aysnchronous service. What matters is how many bits you each carry and how far you each carry them. As my very simple diagram shows there is a network on both sides if that interconnect. There are 40Gbps of traffic exchanged there. If Lelev 3 carried that traffic an average of 800 miles and Verizon carried that traffic an average of 80 miles does the direct of traffic matter? No it doesn’t. Level 3 is absolutely not asking for a free ride here. Far from it. We are simply saying that network operators who are genuine about sharing the cost should do just that and not use an irrelevant metric to obfuscate why they are really not adding extra interconnect ports.

      • Mark,

        I just found and read which I think is what you are referring to. I wish I had seen that one first (I don’t normally frequent the blog), as it would have changed my explanation/questions somewhat.

        I’m with you in the spirit of most everything written in that post. I just don’t know that it is realistic.

        What if, tomorrow, 3 new companies sprang up that were going to deliver services at the level Netflix requires. Based on the numbers thrown around about current load being close to 40-50% and with out knowledge that Netflix accounts for like 60%+ of all traffic, I think it safe to say that the current infrastructure on any end would be pressed to deal with Netflix x4!

        What you are asking for is full access through Verizon’s network in turn for you giving them full access through yours, right? What happens when those 4 extra ports aren’t enough and they need to install another 8 ports and quadruple their current influx through their networks from L3 while utilizing only 7% of that back channel? I’m not sure where the balance comes in that.

        For Verizon to provide that it would require consumers paying the cost.
        Personally, I’d rather make sure that my ISP charges are fair and regulated and that those providing the content charge what they feel they need to deliver it.

        It’s a chicken and egg scenario because where would you guys be if Verizon didn’t deliver service to customers? Then again, where would they be if L3 wasn’t delivering any content from the providers? That’s what makes it a 2-way street, but I’m not sure I’m fully on board with both parties paying the same for disparate use.

        That being said, there shouldn’t be additional charges above and beyond what it should take to carry that traffic. That is, Verizon can’t charge Netflix (via L3) 2x for what it takes Verizon only 1x to deliver for its own content.

        • In closing, while I agree with the spirit of the net neutrality in this issue, I’m not really sure that the end result as it scales out will actually be fair to the consumer. Will I have to pay 5x the cost from my ISP even though I am not streaming high bandwidth content all day?

          I would like *someone* to give an argument that encompasses both the philosophical and economic implications of how “net neutrality” is going to play out 5 years from now when more people are connected and there is much more content to deliver to many more endpoint.

          Realize that most people care a lot more if thier ISP rates get raised 40% than if Netflix/L3/Verizon make 50% less profit 🙂

          • BH, what most people overlook is the cost of the equipment required to provide network bandwidth. It is mostly driven by Moore¹s law and as such we have seen dramatic reductions in the “cost to deliver a bit.” That reduction is ongoing and pretty predictable. It has been going on for years. And it has been decreasing faster than the increase in traffic.

            This is also pretty apparent in the public financial results from ISPs.

            Their profitability has been increasing over time.

          • “I would like *someone* to give an argument that encompasses both the philosophical and economic implications of how “net neutrality” is going to play out 5 years from now”

            The traditional ISP’s (verizon, comcast, etc) desperately want to crush net neutrality because they want to serve their own content. By forcing Netflix to pay to stream over their last mile, their own streaming services will have an advantage (cheaper and/or faster). Do you like paying $150 per month for 500 channels you never watch? or would you prefer $10 per month to stream only what you want when you need it. Which price model do you think the ISPs want to protect?

      • “What matters is how many bits you each carry and how far you each carry them.”

        So if Level 3 send the traffic from their CDN a few meters to an ISP and that ISP carries the traffic 200 miles, then the burden is on the ISP and not Level 3.

        Or if Level 3 sells to a CDN and delivers the same way.

        Flip side is if an ISP delivers traffic to Level 3 and Level 3 needs to carry it around the world.

        This makes sense, but what % of traffic these days is CDN and really only carried a few meters?

        • Col. Jessep the way you have described the impacts of various types of traffic is absolutely correct. Two networks adopting this methodology would add up all those respective bit miles and see if there is an imbalance. The question then is what do you do about that imbalance? It turns out that there are lots of network changes that can be made to create a balance. And we have found that in every instance where we have done this it is relatively easy if two networks have that goal in mind.

      • I’ll add that BH must live in the East, out West we don’t have toll roads. Our major roads are paid for by taxes on each gallon of gas or diesel purchased. Consequently those who use the roads more pay more. The state also has yearly vehicle registration fees that go toward our transportation system.

        Seems to me the best way to catch up with the rest of the world who has faster Internet is to break up the local monopolies the ISP enjoy in most places. Our last mile connections have become a mandatory utility service. It’s time for We The People to add our regulation over the ISPs to keep them honest and to provide the Net to us.

        • Lived up and down the East coast and now in the mid-west and toll roads have always been a way of life. We also have state tax on fuel (not as high as CA, but we are in the top 10…add city tax onto that and we might be higher still). We also have yearly registration fees. I think most if not all of the country has a combination of these.

          Point being…apart from one time registration fees, everyone is going to pay more the more they use the roads. Why shouldn’t Netflix?

          Currently, in-state drivers get toll road discounts for having lane-pass systems for their vehicles. If we had “road neutrality” this wouldn’t exist because it means out of state drivers are being charged more for use of roads. It makes some sense because out of state drivers don’t put in state registration fees to subsidize.

          I think it is a little bit of a convoluted argument to layer vehicle infrastructure on top of the net neutrality. But, really, why shouldn’t I be able to drive through NY’s tollways at the same rate a local can? Isn’t that somewhat of the same argument here?

          What net neutrality is saying is that *should* be the case. And, I agree, Netflix shouldn’t pay more to transit Verizon’s networks then it costs Verizon to transit their own data. That doesn’t mean that they should only pay the same amount if they are transiting 10x the data.

          As I mention above, it also doesn’t have a logical conclusion when we scale out. What happens when Netflix accounts for 95% of the bandwidth as their subscription base increases. That leaves only 5% for the rest of content providers. Should Verizon suffer not being able to deliver their content because Netflix is taking the whole pipe?

          I’m not saying I know the right answer, but I don’t think anyone is giving us the full picture as to how this plays out.

          • The basic flaw in the “Netflix should pay more argument” is that it’s not Netflix who is deciding how much traffic is being called up – it’s the end users. We already pay for “unlimited” bandwidth.

            The end user is going to pay for it in the long run, regardless. This is basically an abuse of near monopoly gatekeeping.

            For awhile, you had hundreds of small ISPs using the phone lines. It’s time for cable to be broken up and split the content provider business from the utility of cable.

          • Point being…apart from one time registration fees, everyone is going to pay more the more they use the roads. Why shouldn’t Netflix?

            Your analogy makes sense. Goods shipped to Target are paid by the manufacturer, distributor, or Target. When you go to Target and buy the merchandise, you pay the costs to drive back to your house.

            So following that analogy, Netflix pays to deliver content to your ISP (which they are in fact paying) and you the customer pays your ISP to deliver the content to your house.

            Which, I believe, is exactly what L3 is asking for.

      • So, wait, then Verizon is trying to monetize the wrong party then. They should be charging Level 3, instead of Netflix! Problem solved.

        • A_Pickle, we have a better suggestion which is that both Level 3 and Verizon use their respective networks to equally share the cost of delivering traffic flows between our paying customers and Verizon¹s paying customers. No arbitrary interconnection fees are then needed.

          • This sounds a lot like that scene from Jerry Mcguire when he says he wants to make less money, and is instantly fired, shunned, and ridiculed.

            What Verizon will hopefully figure out eventually is that the cost of holding out trying to make more money due to the Netflix situation will ultimately cost them more in customers. What with monopolies the way they are, and the difficulty of calculating that, it’s not surprising that they would go the “make more money” route.

            This entire thread is extremely fascinating and I thank you for posting it. Ultimately if a customer pays Verizon for the service and they can’t make it happen, regardless of who they blame, it’s their fault for not being savvy enough businesspeople to make the deals necessary to deliver what their customers want. The customer is not to blame for interpreting an ISP’s service as a pipe to everything on the Internet, because that’s how ISP service has been positioned and advertised since day one.

        • Dave no I¹m not saying that. I’m saying that our network is fully synchronous. But an broadband ISP’s isn’t. Therefore, trafic flows through an interconnection point between those two types of network will not be in balance. And to try and suggest that they should be in order to achieve settlement free interconnection is a fallacious argument. Because even if there were in balance it is the distance that each network carries the traffic that determines their respective costs. Traffic ratios in this context are in fact a smokescreen hiding the fact that an arbitrary toll is being raised.

        • Dave,

          Verizon is effectively saying “If we send 100 packets to them and they send 100 packets to us, we will connect for free to them, but if they send us 180 packets to us and we only send 20 to them, they should pay us to handle their packets.”

          The problem with this is twofold. One was addressed in comments above: The direction of the packets doesn’t matter. In both cases, both networks are dealing with 200 packets between two locations. Whether the packets are going from the the content provider to the ISP customer, or from the ISP customer to the content provider, all the same hardware does all the same work and costs the companies who have the hardware the same amount.

          As for “synchronous peering arrangements”, I’m scratching my head. “Synchronous” means “Happening at the same time”, and I believe the term that should be used is “Symmetrical”, which means “The same amount in both directions”, as in SDSL (Symmetrical DSL, or for example 3Mbps up and 3Mbps down; equal in both directions) vs ADSL (Asymmetrical DSL, like 7Mbps down and 1Mbps up, or not equal in both directions).

          That nitpick aside though, yes, it’s impossible for an end-user ISP to have a synchronous/symmetrical arrangement with a content provider. If the user gets 75Mbps down and 10Mbps up, then obviously the ISP is going to request more data for its customers from the interconnects than it is going to have sent from the customer to the interconnect endpoint. So Verizon’s claim is basically using fancy words to say “We want everybody on BOTH ends of our network to pay us”, since symmetrical peering arrangements are not really possible for ISPs.

          I think one important thing to remember is that as big as Verizon and Comcast may seem to their users, they are really just docks and piers at the end of a huge network of roads. If you went to an internet cafe and paid $5 for a WiFi connection for an hour and the WiFi was slow, should the cafe say “Well, that’s because the internet service provider we connect to is not paying us enough to give you data they have.”? No. They pay an ISP to give them the data that they give to you.

          If you consider it directly, the end user pays the ISP and is given a link that they can download the data they want over. If this all worked in the “Logical” direction, then Verizon would PAY Level3 for the connection to download the data that their customers are paying for. If Verizon’s customers want faster speeds, then Verizon would have to pay for a bigger connection to the bigger network.

          This is not the way it works though. “Peering Arrangements” mean “Peer”, or equal, even though Verizon, big as it is, is a small fish in the network world. The internet works the way it does because these companies agree to connect and pass information to each other, and agree to do so for free because you literally get everything you give in the world of networking. Again, the direction of the packet doesn’t matter. It hits a router, it goes through a pipe. Both are used no matter what direction.

          Technically, the internet is not even working as efficiently as it was originally envisioned to. The original idea behind the internet was that it would be resilient and route around problems. Verizon connects to Datachron and Level3 for example. Level3 and Datachron also connect to each other. In the original idea of the internet, if Verizon’s link form Level3 was down completely or working badly – as it is now, because Verizon is literally saying “We don’t want to connect more to meet our customers’ needs unless the ISP that graciously allows us to connect to them for free and get data at effectively unlimited speeds as much as we want pays us to get their data” – then Level3 should be able to send the packets to Datachron and then Datachron sends them to Verizon because that route is working better.

          In reality, the hardware that handles the packets adds to the cost of handling the data. That is what is implied in the 80 miles versus 800 miles concept. The reason Level3 is able to NOT have to charge Verizon to connect to them and get the data from them is because they are able to set up CDN systems. They can connect the source of the content directly to the router that talks to Verizon and thus have minimal costs in letting Verizon give that content to Verizon’s customers.

          Were it not for things like this and long haul fiber (and a few other, more-complicated factors), Verizon would be paying Level3 because its customers are sucking so much data out of the the connection. If you want more data speed, you pay more to the people you connect to to to download from. Verizon should pay more to suck so much data from Netflix for its customers.

          • Hit Verizon where it hurts. If everyone ends up just using a VPN to bypass the throttling, the traffic could end up costing Verizon more to deliver than via Level 3. Customers need to start thinking like MBAs.

          • Major flaw in the cost argument L3 is putting up ( yes I know I’m late to this debate ) but the cost to run 100 gibt 800 miles is FAR less than the cost to connect 1 million subscribers in an 80 mile metro like LA. First, 100 gbit is 10 lines of fiber, using the latest tech as the author constantly argues, you have a distance of 150+km per repeater but we will use the argument that your using low grade fiber and decline to use any error correction thus need a repeater every 20 miles. That will run you 40 repeaters for the trip. Now The largest switch I could find by Cisco is 768 ports in a 13 u chassis. To service 1 million subscribers, you would need 1300 of those devices.. And let’s be honest here, we both know networks are far more complicated than just a switch to physically plug the subscriber into. So for that 80 mile LA area, the cost to deliver bits is several magnitudes higher. Also, your long haul fiber is pretty straight forth from hub to hub running along either a high way or rail way, in areas like LA, you have to deal with zoning, access tunnels/pipes, specific location to make your hubs. It is a logistical nightmare that costs a ton. This discussion seems like smoke and mirrors to disguise profit and a bad agreement. Netflix has a multi year agreement for traffic which turned to be a bad deal. L3 cant breach the contract so is trying to blame everyone else for the bad deal with netflix as they are the middleman between ISP and netflix. Just like the gas tax, the company who uses the road the most should have to pay the most. This net neutrality is going to push ISPs to data capped services just like the cell phone market, companies and individuals who cant see past tomorrow are driving the nail in that coffin. Charge Netflix for their ridiculous utilization or watch in horror as ISPs have to datacap just to cover their infrastructure costs ( noted above ). Ever wonder why most telco companies who offer end services are net profit negative? Check a public expense report… infrastructure kills…

          • Tony, I agree that the cost per mile differs between a backbone-long-distance network and a consumer-last-mile network. But the price that each of those networks charge for the services they sell to their customers is also very, very different. When Level 3 interconnects its network with Verizon’s network then only traffic between each of our paying customers crosses that interconnect. Level 3 get’s paid by its Internet Transit and CDN customers. Verizon gets paid by its broadband customers. The difference in the prices of those services (when compared as a $ per Megabit per second) is enormous. In fact those price differentials more than allow for the different cost per mile of the two networks.

      • Your diagram shows network on both sides of the peering but only shows one utilization figure for each of the interfaces. Is this a unidirectional peering where traffic flows only from Level3 to Verizon (which would imply transit, not peering), or is traffic flowing both ways? If there’s traffic flowing both ways, does the figure mean that the entire 20 Gbps bidirectional capacty of the interface is in use, or are we only being shown utilization in one direction?

        Michael Mooney confirmed in comments I exchanged with him in his blog post back in March that Level3 does indeed enter into contractual agreements with its peers. He also went on to confirm that the terms of those contracts are confidential. I’m okay with that; it’s a necessary business practice. Personally, I have doubts that those terms include unlimited bandwidth for both sides whether or not the exchange of traffic is equitable. But it’s all confidential, so we have to ignore the man behind that curtain.

        Or do we? You’ve seen fit to reveal the things about this peering that bolster your side of the argument, so at least some confidentiality is out the window. How about providing the two other pieces of data required to evaluate whether or not there’s any validity to your argument that Verizon is being a bad peer? One is the utilization figure for traffic in the other direction. Any router big enough to handle a Tier-1-sized peering keeps count of inbound and outbound traffic, so Level3 is in posession of that information. The other is whether or not the current state of things is within the definition of “equitable” per the agreement you have with Verizon. In the interest of maintaining confidentiality, I won’t even ask how “equitable” is defined. Of course, you could say “yes, it is,” and it still wouldn’t prove a thing. But at least you’ll have said it in public and Verizon can throw the BS flag if it disagrees.

        I bring this up because, despite your assertion that despite having lots of available network capacity, Level3 de-peered Cogent in 2005 for pretty much the same reason that Verizon isn’t upgrading the peering. The reason — and this is verbatim from a Level3 press release — was, “without paying, Cogent was using far more of Level 3’s network, far more of the time, than the reverse. Following our review, we decided that it was unfair for us to be subsidizing Cogent’s business.” What I read into this is that Level3 does actually care about equitability in its peerings, at least when the imbalance of traffic isn’t in its favor. I’ve been very critical of Cogent for its recent behavior, and Level3 deserves just as much scorn if it’s doing the same thing.

        By the way, I’m not a Netflix customer, but I am a Verizon residential customer. (My only other choice is Comcast, and that’s a non-starter.) I also have some sentimental love for AS701 becuase I helped run it in the 1990s, but not so much love to automatically favor Verizon if they’re being buttheads. What’s amusing is that I lease and use a server in a distant data center which buys transit from both Level3 and Cogent. The routing from where I am works out such that traffic traverses Level3 in one direction and Cogent in the other, so I get the shaft if either or both of you happens to be wedged at your Verizon peerings at any given moment. So thanks for that.

        • Mark, I did say in the post that the utilization figures for those ports were for traffic in the direction from Level 3’s customers towards Verizon¹s customers. You are correct these are syncronous links – the utilisation in the reverse direction is lower and is shown on the Verizon diagram in their blog post.

          Yes we did have a disagreement with Cogent many years ago. I¹ve also written about that elsewhere too. I agree that neither party handled that well but we resolved the problem within days and put an agreement in place that has stood the test of time ever since. Our issue was one of equitable sharing of costs. We have remained absolutley consistent in saying that what matters in assessing that is the amount of traffic that each network carries to and from the interconnect point multiplied by the distance that it is carried. We call that bit miles. I have given examples of how that works in several of the other comments here.

          • btw bit miles

            is it geographical miles or actual cable length miles?

            If cable length, what’s to keep an interconnect from simply wounding a wire around a spool.

          • Jeff, great question. It is the straight line distance between the originating and terminating points. And we suggest that partly to overcome any gaming issues, as you suggest. But mostly we did this so that network operators don¹t have to argue with each other over how they decided to build their network.

          • Here’s the thing, Mark:

            Since it’s been convirmed that Level3 enter into peering agreements with other carriers, it’s a pretty safe bet that those agreements set out each side’s responsibilities and what determines whether or not those responsibilities are being met. In other words, somehwere in a file drawer is a contract signed by Level3 and Verizon that defines what “equitable sharing of costs” means and how close to that definition equitable the arrangement must be to keep it settlement-free. I understand how bit-miles works, but if it isn’t part of your agreement, it’s as relevant to this dicussion — and your relationship with Verizon — as the number of cat pictures transferred per hour between IP addresses with all-even-numbered octets.

            If Verizon is violating the terms of your agreement by not adding bandwidth to the peering, plant your flag in the ground and say so. If your agreement prevents you from doing that even when it’s true, it’s perfectly okay to say “our agreement prevents us from commenting on that.” And at that point, all the public can know for sure is that the probability of one of you or the other being full of it is about equal.

          • Mark, what I can say is that Level 3 is meeting ALL obligations we have under the agreement in place.

      • Let me add a little bit here.

        Home users are data consumers. They stream movies, download software and browse web sites. For the most part home users won’t distribute data out to the Internet except when they’re doing things like video conferencing, backup or peer-to-peer file sharing (which most home network providers hate).

        So, as Mark says, it’s impossible for there to be balanced traffic between a home network provider like Verizon and a long-haul backbone like Level 3. The idea of “balanced peering” is relevant between backbone providers, not between a backbone provider and a home network provider.

        Furthermore, Verizon’s customers have already paid Verizon to deliver data to them. Verizon simply wants to get paid twice.

      • Thank you! I’ve been hearing about this peering debate for ages, and I’m so tired of the “asynchronous traffic” argument. Apparently it makes just enough sense that anyone who doesn’t really think about it finds it acceptable. It’s particularly sad when, while making an argument, people get confused over whether it is the person who sends data or the person who receives it who should pay, but that doesn’t clue them in to the fact that they’re using the wrong metric.

        So Level 3 should pay Verizon since they’re sending Verizon more data than they’re accepting? OK, so then I guess Verizon should be paying their customers when they finally deliver that data to their customers, right? I mean, if the rule is “he who sends the data pays” then Verizon owes a lot of money to their customers who watch Netflix. …or is the rule “Verizon gets the money no matter which direction the data is flowing?”

        You’re right, it’s absolutely about who carries the data the furthest. I pay my ISP for internet access. They either need to provide that by running wires between me and all of the web sites I want to visit, or they can just run it a few miles, then pay someone else to take it the rest of the way around the world. For them to cover only half the route, then insist that someone else pay them for the privilege of taking it the rest of the way is absurd. This is nothing more than Verizon telling Netflix “we own your customers, so you’re going to pay us.”

      • I payed Verizon to deliver data to me at a certain speed / capacity. Netflix paid Level 3 to deliver data at a certain speed / capacity. I didn’t pay Verizon for a package that provided full access to AT&T links, but only partial access to Level 3 links.

        Of course Verizon also competes with Level 3 as a tier 1 provider, so they have a reason to want to get large customers (Netflix) to connect to them directly and pay them. If Verizon owns the connection to users and to content providers, they can get paid for the bandwidth on both ends.

    • > What’s the debate here? Is it that Verizon should have to freely carry Netflix’s traffic to consumers because consumers pay for use of Verizon’s bandwidth?

      FIOS already has the internal capacity to carry the streams, but they are refusing to open the tap all the way.

      And yes, that is exactly what a last mile ISP should be doing. This is how it works. This is how it has always worked. No customer, no one ever in the history of the internet. has said to themselves “Gee, I sure do wish I could selectively partition my bandwidth so I can pay different rates for different services. Wouldn’t that be nice?”

      The analogy of an 18-wheeler doesn’t really hold up. Using a network connection doesn’t cause it to wear out more quickly, and in this case it isn’t even some faceless entity driving the big rigs, it’s the customers themselves. If I paid my taxes and bought the right to drive my truck on a road 100 times a month, does it matter if I choose 90 of those trucks to be Netflix trucks?

      • The analogy of the 18-wheeler DOES hold up. If 9 of every 10 trucks on the road are Netflix’s trucks, and if you’re congested because the highway isn’t big enough, so that you can only get 20 miles in an hour, are you going to just accept it everyday for 10 years, or are you going to get upset? If you get upset, are you going to demand that the road gets made larger if there are no other alternatives?

        Of course you are.

        Now that you’ve accepted this, ask yourself if everyone should equally chip in, or if this one company abusing the system by throwing 90 trucks for every 100 on the road when, had they chosen another method such as rail (aka their own content delivery network, their own private road for example), the public road traffic would be fine.

        Netflix is abusing the system.

        The lawyers and everyone else can point fingers at eachother here and elsewhere all over the Internet, but the fact remains that Netflix is the root of this problem. Netflix is collecting revenue that is directly inflicting costs on the last-mile providers.

        The FCC absolutely should address this, as it’s an unsustainable model that abuses existing relationships, stifles innovation, and negatively impacts everyone’s Internet experiences.

        And while it’s Netflix abusing the system, and the last-mile and middle-miles pointing fingers at eachother, the real culprit here is the cowards in the FCC who refuse to step up to the plate and DO THEIR JOB, one way or another, and MAKE A DECISION.

        • The analogy of the 18-wheeler DOES hold up. If 9 of every 10 trucks on the road are Netflix’s trucks, and if you’re congested because the highway isn’t big enough, so that you can only get 20 miles in an hour, are you going to just accept it everyday for 10 years, or are you going to get upset? If you get upset, are you going to demand that the road gets made larger if there are no other alternatives?

          The point is that despite the analogy holding up, the roads are only half full. Fine. Four out of every five trucks on the road belong to netflix. The point level3 is trying to make here is that there is no “bandwidth crisis” that everyone seems to think there is. The roads are not full. That’s what he picks apart in the first half. Level3’s roads aren’t full yet, so they aren’t that annoyed at netflix. Verizon’s connection with level3 is overfull, but verizon wont expand it even at level3’s prompting.

          Verizon is the one with the small toll.

          The second part of the article is about balance peering.

          Those connections between level3 and verizon aren’t one-way. An “imbalanced network” is one that, given 50% for upload and 50% for download, utilizes all 50% of the download and 1% of the upload. In this case, when level3 sends netflix or verizons data along, it is wasting a good 40+% of its pipe.

          Netflix is not abusing the system yet. They may have to pay extra for the pipe they are wasting but the point is that what netflix pays for is between neflix and its provider, and netflix’s provider and level3. Not between verizon and netflix.

          If verizon gets its way, verizon will get paid by its customers for access, and by netflix for traffic, and will pay level3 for access.

          Level3 will get paid by verizon for access and netflixs isp for access.

          Netflix will get paid by customers, and have to pay its isp for access and verizon for access on the other end.

          Customers pay twice, access and netflix. Level3 gets paid twice, once for each connection. Verizon gets paid twice, once by customers, once by netflix, and pays once. Netflix pays twice, once to its isp, once to verizon.

          That’s just greedy.

          Customers should pay twice. Netflix should pay once. Verizon should get paid once (by customers) and should pay once (to level3). Level3 should get paid twice (for 2 connections) and pay twice (to verizon and netflix’s isp). Netflix should get paid once and pay once (by customers, to its isp).

          The POINT of the article is that there is already an agreement in place to deal with this, and verizon sidestepping it is just a way for them to lie their way to more money. If verizon has an issue, they should charge more for their network to improve it and/or get more space from level3. If level3 gets congested, they should balance out both ends. If netflix’s isp is congested, they should charge netflix for the space netflix is using on its network.

          Once traffic enters level3, it is now traffic from level3, not from netflix, therefore you should negotiate with level3 like you have been instead of trying to charge around level3.

          • meritasix the “who pays who” today, for delivering the bits, is a lot simpler than you are suggesting. Level 3 gets paid by Netflix. Verizon gets paid by the broadband consumer that makes the request for Netflix content. The interconnection in the middle (between Level 3 and Verizon) is settlement free. And in our view that’s OK provided both networks, Level 3 and Verizon, carry the same number of bits roughly the same distance; we share the cost of carriage.

    • BH> “Netflix sends out an unprecedented amount of traffic.”

      Excuse me, above sentence sounds like a big, fat, abominating lie.
      Isn’t it a traffic that Verizon customers _requested_?
      Isn’t it a traffic that requesting it customers paid Verizon for?

    • >Is it that Verizon should have to freely carry Netflix’s traffic to consumers because consumers pay for use of Verizon’s bandwidth?


      Because their own network doesn’t handle it. Specifically, their peering with other ISPs (like Level3).

    • BH, your comparison falls a little short. To make it a more proper comparison, end users are the car drivers that use their cars to go shopping. They pay their registration fees for their vehicles which all include a roads and bridges tax (Their ISP fee). That tax allows them to use the roads. They can make one trip a month to the store (content provider) or they can make a million trips to the store (content provider), but they have PAID. You don’t turn around to the content provider and demand they pay more because half of the cars on the road are going there.

    • BH sounds like he works for verizon.

      At the end of the day, Verizon has many customers that PAY for high speed internet, and Verizon is failing to give customers what they pay for. They are trying to double dip (ie. get money from the customer AND Netflix!)

    • All right, but Verizon has a near-monopoly position on consumers who pay them for Internet service. If they want Netflix, the obligation is on Verizon to build out their network to accommodate what their own paying customers are using (ie, Netflix). That there’s an imbalance in the network is unavoidable, given that Verizon is a last-mile provider and does not allow users to run servers on residential connections. Free peering agreements shouldn’t come into play for last-mile providers, as they’re not providing transit as Level 3 does; they provide the pipes into the customers’ houses.

      Customers don’t have a lot of options, either. It isn’t as if they can tell Verizon to stuff it and leave – there’s nowhere to go. Where I live, my options are CenturyLink DSL (max 5 Mbps) or Comcast. If I need more than 5Mbps, I can only go to Comcast, who will charge way more, try to wheedle me into cable service I don’t want, then mock me with fliers in the mail twice a week showing they’re going to charge new customers half what I’m already paying them.

    • “Is it that Verizon should have to freely carry Netflix’s traffic to consumers because consumers pay for use of Verizon’s bandwidth”

      That’s exactly right. There’s nothing “freely” about it. Netflix isn’t “sending out an unprecedented amount of traffic” as much as Verizon’s customers who have paid for it, are requesting it. It’s Verizon’s job to deliver what their customers are paying for, regardless of where that traffic comes from.

      If our political system wasn’t already so corporately corrupt Verizon would have already received a huge smack down for their extortion and greed.

      • My addition to this thread is to refer to Mike Mooney’s previous blog post ( In order to help solve these interconnection issues we have suggested fair and equitable ways for those connections to be maintained without congestion. Level 3 is not suggesting we get a free ride. We have paying customers on one end of the traffic flows too. And as such we have an obligation, when we interconnect with an ISP like Verizon, to do our fair share of moving that traffic between our respective customers.

    • I like your analogy which equates road traffic, because it really perfectly shows why the attempts to get extra money from content providers is flawed.

      Lets continue to assume that the internet is like a highway, and that high content deliverers cause degradation in the network over time. If this were like a highway, we would choose one of 2 models. The first is the public highway option. This highway is entirely paid for through taxes (Subscribers) and receives funds for maintenance that way. The second option is to create a toll road which receives funds by those transporting the goods over the highway. The costs can be assessed based on the weight of the goods. Both models are capable of fully funding the highway independently. Now, in Verizon’s model, they would get the full revenue from both sides, while also not performing their required maintenance.

      Either model provides more than enough, but in an industry which has monopolized itself out of competition, they have decided to screw both their actual customers, and the providers of the product their customer demands.

      Keep in mind, the only reason that people want the road is because goods are being shipped over the road. If those goods become too costly due to having to pay tolls to send and receive, people will build a different road.

    • “Is it that Verizon should have to freely carry Netflix’s traffic to consumers because consumers pay for use of Verizon’s bandwidth?”

      Freely =\= paid for.

      I realise I’m probably coming to this argument with very few relevant facts, but in my view – if Verizon has customers paying for a given speed of access to content (including Netflix), they should provide that to the best of their ability. Is it the fault of any other company if they have decided to market their service at a speed or contention higher than they think they can provide? No. Restricting access to the product doesn’t seem like the right answer for the customer (bearing in mind that the product is “access to the internet, at speed x, for this cost per period).

      If they feel they can’t provide the services they’re selling without restriction, why are they advertising/selling them at that level?

    • What’s the debate here? Is it that Verizon should have to freely carry Netflix’s traffic to consumers because consumers pay for use of Verizon’s bandwidth?

      Answer: Yes. Verizon should deliver to Verizon customers the content that Verizon customers are paying Verizon to deliver to them. Verizon customers pay Verizon money because they expect Verizon to deliver to them the content they request.

    • “Is it that Verizon should have to freely carry Netflix’s traffic to consumers because consumers pay for use of Verizon’s bandwidth?”
      Yes. US taxpayers paid huge subsidies to have these networks built.

    • “Netflix did not make arrangements to deliver this massive amount of traffic through connections that can handle it.”

      Realistically, Netflix isn’t obligated to ensure that Verizon has sufficient transit from their customer networks to the transit networks that connect them to Netflix’ infrastructure. At no other point in the history of Internetwork peering has any content delivery company ever been made to be responsible for how an ISP gets its transit, but now for some reason Verizon wants to shift part of its own responsibility onto Netflix, with no clear and reasonable explanation as to why this should be. Netflix isn’t sending unsolicited traffic to its customers, rather Netflix is responding to requests from its customers, requests that those customers pay their Internet service providers to facilitate. Those requests are not being facilitated properly due to congestion that Verizon refuses to address.

      As for the trucking “comparison,” you need to understand that cars and trucks and roads is a woefully poor analogy to Internetworking. We aren’t dealing with roads that wear down. We aren’t dealing with public infrastructure. We aren’t dealing with capacity that can be involuntarily constrained. Every bit of exchange between Verizon and Level3 is voluntary, every bit of that exchange is paid for to Verizon by its customers. Every single bit of Netflix traffic that traverses an interconnection between Verizon and Level3 is a bit that Verizon’s customers paid Verizon to facilitate. Verizon does not get to turn around and complain about Netflix because Verizon doesn’t wish to provide its customers with the service that they pay for.

    • Toll roads are private enterprise and not government roads of which you pay a fuel tax to maintain. (see the highway trust fund). Commercial drivers have additional fees and regulations, but that is born only once and is passed on to the customer. Sorry, but I worked in shipping and transportation for nearly a decade and I understand the issues. What Verizon is doing or wants to do is double bill: Bill the customer in the home or office for the bandwidth and connection costs plus profit margin, and then bill the other end of the connection for the same bandwidth and infrastructure. One of those connections is pure profit. Simply greed. So to fix your road analogy: Company X is shipping a truck load of widgets and has to pay for the cost of the fuel, a road tax, and fees to the shipping company before being allowed to use the road. On the other end, the receiver of the goods has to pay the cost of the fuel, a road tax, and fees before delivery can be made by the shipping company. That’s what Verizon wants to do: charge twice for the same movement of stuff.

      • Verizon is maintaining the road. The customers and Netflix are demanding that Verizon expand the road. Are you suggesting the Verizon should shoulder this cost alone?

        • Brian I am not suggesting that at all. My simplistic diagram shows two networks. One either side of the interconnect point. Level 3 is not asking for a free ride. We know we have to have an equitable arrangement where we share the burden of carrying traffic from our customers to Verizon’s customers. I have referred to that sort of arrangement as a bit miles in several of my responses here.

    • Actually, Verizon are getting paid for that traffic…. from the customers. I pay them to bring netflix to my house. I pay them to have the capacity to do that for the last mile. Netflix pays level 3 to bring it the majority of the way. The only way around this is for netflix to set up their own ISP.

      • You are incorrect. You are paying for Verizon to connect to Verizon’s network, and Netflix is paying to connect to Level 3’s network. Level 3 and Verizon have a network connection that is full. They are having a disagree about how to resolve that. The root cause for that connection being full is that Netflix is using a lot of that capacity and wants more. Why are you saying that Verizon is to blame?

        • Brian, stop being disingenuous. The point of “FiOS Internet” is to provide access to the *Internet*, not “Verizon’s network.” I don’t think any Verizon customers care about what’s on Verizon’s network, they’re paying for 75Mbps of access to *Internet content*. That’s the service Verizon is supposed to be providing, and providing it well depends on properly maintaining the access points to said Internet, including their Level 3 peering point.

          Only in a non-competitive market like the US ISP space do you get absurdities like a company willingly degrading a service that’s extremely popular with their customers.

        • Slant 1: Netflix isn’t using the capacity. Verizon’s customers are sucking that data from Netflix. You pay Verizon to be able to get data from Verizon’s network. Verizon should pay Level3 to be able to get the data from Level3’s network.

          Slant 2: Verizon says, “More data is coming from Level3/Netflix to us than we are sending to them! They should pay us to send this data to us!” Answer: More data is going to Verizon’s customers than the customers are sending to Verizon. Verizon should pay its customers to send that data to them.

          Slant 3: A restaurant has a main kitchen and a sushi bar. The sushi bar gets fresh fish from the fish market and the main kitchen gets groceries from the grocer. And through some miracle of some sort, the restaurant gets all of this food for free and just charges the customers to prepare and serve the food. But suddenly a lot more paying customers want main entrees because the veggies are super fresh, and they even pay the grocer to give the restaurant the freshest veggies. The demand on the main kitchen means the restaurant is not sending enough trucks to the grocer every day, and the grocer has plenty of space to give the restaurant more fresh veggies, saying “We’ll make new docks, and even give you new trucks and people to drive them so you can get all the veggies from us that your customers pay for. Half your kitchen staff is sitting around doing nothing, so the only problem is how many trucks you’re sending to get veggies from us”. But the restaurant wants to charge the grocer money before they send more trucks to pick up more free veggies from them for the diners.

          • Slant 4: I’m part of a relay race.
            I pass a paper rod to verizon.
            Verizon passes it to level3.
            Level3 passes it to netflix’s isp.
            Netflix’s isp passes it to netflix.

            Congrats your packet just showed up.

            Netflix then passes back a heavy metal rod and a few paper ones.
            Netflix’s isp charges netflix extra to move more weight. Netflix pays their isp.
            The iron rod and paper rods are passed to level3. Level3 negotiates with netflix’s isp because carrying paper one way and iron another is annoying. They come to an agreement.
            Level3 lugs the rod back, and gives it to verizon. Verizon drops it and complains it weighs too much. Then they completely ignore level3 and yell at netflix for handing back your iron rod.
            This post is level3 yelling back at verizon for being a sissy by not only going back to yell at the guy way back by the starting line, but also ignoring the fact that level3 did all this work and verizon is wasting their time. Level3 is also pointing out that if verizon had put shoes on they might not have dropped the rod. They point out that all this yelling is wasting race time. They also offer to buy verizon some good shoes.
            Verizon proceeds to throw a hissy fit and say that level3 is too slow. Then they call the referee over and give them money, asking them to yell at level3 and netflix and allow them to change the racetrack so that netflix has to do more work. This would also allow them to shrink the lanes of other racers too, but never mind that.
            During the hissy fit, verizon loses your paper rods.

            When verizon finally gives you the iron rod you wanted, its bend and dinged up, and they want more money for the hard work of transferring it so far.

            Except you already paid verizon to move an iron rod. And are wondering where the paper ones went.
            You complain to verizon.
            Verizon responds: “You paid for up to an iron rod and 3 paper per day. You got a damaged iron rod. That’s under an iron rod and 3 paper. Yep, you are getting what you paid for. You didn’t pay for a guarantee. We can’t offer a guarantee, that would make us liable for freak storms.”
            Verizon then offers you business class (with a guarantee!) for 10x the price.

        • Brian, why are you defending Verizon? Do you have a monetary stake in the matter? It’s clear a lie was told. There is capacity available. If anything, we need an end to the ISP monopoly.

    • Your road/toll road analogy has some flaws, and your point is weakened (even reversed) if you correct it. To stick with the shipping analogy:

      1) Netflix is like a ship-to-home retailer (think Amazon).

      2) The companies that maintain the roads are also the companies that run trucks across them – the trucks are full of Netflix’s goods, but Netflix isn’t running the trucking company.

      3) L3 provides trucking and roads until it “peers” with Verizon’s roads. Netflix isn’t paying for use of L3’s roads, it’s paying L3 to use their fleet of trucks to bring stuff across their roads to where it meets Verizon’s.

      4) At the peering point, Netflix’s goods that have been purchased by their customer are loaded from L3’s truck on L3’s road to Verizon’s truck on Verizon’s roads.

      5) The customer has already paid for Verizon to deliver up to 10 thousand kilograms of goods per second, no matter the source. Verizon has already received payment from their customer for their shipping service/road use.

      6) The toll-road part of your analogy makes no sense because Verizon owns and maintains the road and trucks – they aren’t charging a toll to themselves – thus that part of your analogy needs to be eliminated.

      Now that I’ve corrected the analogy, you have a customer who purchased a shipment of goods from Netflix. Netflix has paid L3 for the delivery of those goods to as far as L3 can bring them. The customer has already payed Verizon for them to continue carrying those goods to the customer’s house.

      The situation we have here is that when L3’s trucks show up at the peering point ready to load the goods onto Verizon’s trucks, Verizon knows the goods on that particular shipping line are likely from Netflix, and puts up “road closed” signs on half their lanes and “Truck out of order” signs on all the trucks in those lanes, so L3 has to load all of the goods from Netflix onto a fraction of the trucks only capable of carrying 1 kilogram per second of goods to the end customer (1/10th of what they are actually paying Verizon to carry!)

      Verizon is telling L3 “If you want our other lanes open and trucks in service to bring Verizon customers the goods they have already paid to have delivered, you or the retailer who sent those goods need to grease our palms!”

      Also, Verizon just happens to run a competing ship-to-home retailer that doesn’t have to deal with this cost because they will never close roads and shut down trucks inside their network to cause congestion. In fact, goods shipped from Verizon’s stores eventually won’t even count against your 10 thousand kilogram per second shipping limit.

    • You don’t pay taxes to use a toll road, as they are generally funded by private parties (at least in Texas). The 18 wheelers already pay a good bit more then you do for taxes, mostly since they drive more and thus pay more gas tax. Your vehicle is much more efficient also, so you pay less gas tax per mile. The government doesn’t restrict what you drive on the public roads (as long as properly licensed). If you choose to drive a Prius instead of a Peterbuilt everywhere, then that is your choice.

      The real point being, that like our public roads, once you pay the gas tax, the government doesn’t say you can’t drive down main street, that you have to stick to the more congested 2nd street because of which store you bought the gas from.

      Verizon is basically saying that they don’t want to upgrade the interconnects because it will allow more Netflix traffic in from L3. They would rather that Netflix pay them AND L3 to move their traffic to the customers that are already paying Verizon for that traffic too. At what point does it stop? Does Netflix need to pay every mom and pop ISP in the world (like me) to move their traffic also? Does Verizon need to collect a “toll” from every single website in the world?

      How about we swap this scenario around to give it more perspective. You pay for internet already. You also pay for a Netflix subscription. Netflix also pays for internet access on their side. Now what if instead of Verizon wanting Netflix to pay them for the traffic, what if they wanted it the other way. They wanted you to pay them an extra $10 per month to be able to access Netflix, even though you already paid for internet and Netflix yourself. Now you know how Netflix feels. Why stop there though. What if they decided that Google searches or Youtube uses a lot of bandwidth and now they want you to pay extra to access them also.

    • If you took the George washington bridge as an example.

      you don’t pay tolls to go to Jersey,
      you pay tolls to go to NY.
      truckers do pay more.
      Chris christie decided he wanted to add a traffic pattern test during times when the bridge is heavily being used and closed off 3 lanes on the GW bridge.
      the bridge had plenty of bandwidth, businesses are still paying more for their usage, people are still paying tolls to utilize the bridge, but the bottle neck was from jersey.

      Guess what happened to Chris christies cronies? You know why? Because there is only 1 resource in town and it’s owned by 1 person. You can’t exactly build another bridge right next to the GW bridge if you wanted to.

      I’m all for business making their monies. Verizon has a contract wiht me as an end user to provide “blazing speed” They fulfill that as their speed test always show that I’m getting 50mbps down and 35 up. Except the issue is that we all know it’s kind of pointless if there’s no content being delivered to me due to their interconnect to level3 (and that’s ALWAYS the case). So if there were more competition in the market, I would HAPPILY say goodbye to verizon and move to another carrier. Except as Mark has pointed out, they’re a monolpoly and i’m kind of stuck. So, everyone in my area on verizon, is paying a premium to verizon, for them to skimp on their infrastructure costs to L3. Considering they have increased revenues on inet by 15% year on year and also generate 1.15$ earnings per share of profit, i’d have trouble believing that paying a few extra thousand dollars to upgrade their interconnect pipes would cripple their profits.

    • I don’t understand your car analogy. The government hasn’t agreed or advertised to give me a certain amount of road usage. They don’t advertise unlimited use of every road.
      Verizon however does advertise unlimited usage and advertises an expected level of bandwidth for the price. People pay Verizon for that full bandwidth. I’m sure Netflix pays their ISP an exorbitant amount as well for their bandwidth. It makes no sense to me that you want Netflix to pay because I’m using more of my (already paid for by me) bandwidth. If Verizon does not have the infrastructure to support the what they’ve advertised to their customers, then it’s absolutely Verizon’s problem.
      If that means that Verizon gets raked across the coals because of a poorly thought out peering arrangement with L3, I’m sorry. Be more careful with your terms when the contract comes up for renewal. In the meantime, they need to provide what they’ve advertised.

    • Netflix does not and should not pay Verizon a dime.
      I as a Verizon customer and every single verizon customer am paying Verizon to provide me with internet access at the speeds they promised me. Netflix is not thier customer I am. If I want to use my internet connection to stream on 6 different computers 6 different netflix shows I should not have my Netflix throttled.
      If I want to down load 25 games off steam, stream all of the Simpsons and play world of warcraft at the same time I should be able to do it. Netflix is not Verizon’s customer, I am its my internet connection not Netflix’s.

      • You might want to read the contract you signed with verizon and inform yourself of what you’re able to do.

        No way in the contract does it gaurantee you’ll get your desired content at the desired speed you think you should get it at.

        what your contract does semi stipulate is that the your connection from verizon is 50/35 or however fast you pay for. Unless your fiber connect is broken or your relay is busted pretty sure verizon delivers on that promise.

        • jeff, the entire point of this article is that Verizon is not getting Netflix data to you at the 6 Mbps needed to watch uninterrupted, far less than 50 Mbps download speed promised.

    • The fact that 1/3 of the traffic come from Netflix is meaningless – that means there is 1/3 less come from all of the other sites on the Internet. If all of the traffic came from one site…Verizon would sell “unlimited access to that one site” and get their $50/month. The fact is Verizon is selling a product – unlimited access to the Internet. The fact that the Netflix service exists drives DEMAND for Verizon selling such connections. Verizon markets their great network and its great capacity so you can safely visit sites like Netflix. Verizon makes MORE $$ because Netflix exists.

      You can bet if Netflix had a monopoly (like cable networks do / ESPN) they would be extracting a toll. That’s why we have regulation for monopolies….unfortunately with a bought and paid for Congress we don’t manage these issues, we sell off monopoly rights to the highest bidder.

    • Thanks for the clarity – its like a breath of fresh air.

      To cover the extra money Verizon wants, maybe Netflix *should* charge Verizon customers more, and automatically add a ‘Verizon peering surcharge’ to itemized bills sent to Verizon customers.

      Better still, Netflix customers could opt-in to this surcharge; their traffic flows would then be prioritized with the help of the new Verizon-Level3 interconnects Verizon wants to charge for.

  7. Thank you for explaining how this works. Please pardon this naive question, but why don’t they solve this problem simply by using colocation ? In the case of netflix, everyone is watching the same titles from one common catalog. Why even use a CDN at all, and simply just pay for a dedicated server that physically hosts the whole catalog in all major US cities ? Then it never really goes over the “internet”, at least not out of the user’s town. It just seems horribly inefficient to send down a separate copy over the internet every time someone watches something.

    • Mike, this is actually how a CDN works – including Netflix’s OpenConnect platform. And all good CDNs will have servers in all population centers. But there is plenty of other Internet traffic not delivered over a caching CDN.

  8. “Hail Mary” throw here: but without seeing the per link raw throughput data *per 10Gbps link*, is there *any* chance that an etherchannel load balancing algorithm keeps hashing to the same 10Gbps pipe due to repeated endpoint proxies and load balancer VIP IP SRC and DST IPs (whereupon SRC/DST ports should be used too) or *is* the actual 40Gbps per the diagram?

  9. So cool to see you people hammering home the point of failure and who’s control it falls under. It is too easy for some public facing PR pro to absolve a company of guilt and simply claim innocence. “The glove appears to fit!”

  10. This is a great blog post Mark, it appears that Level 3 is ready to connect additional ports with Verizon, but Verizon is refusing to upgrade their own equipment. This leads to a question that I have: since Netflix has already paid Verizon for interconnecting, is Verizon not holding up their end of the deal? Should other content companies look at that Netflix / Verizon deal and not feel comfortable about signing a contract?

    Verizon is greatly sacrificing their own customer’s experience for bottom line profits. Other companies need to be aware of the games that Verizon likes to play before doing business with them.

    • Netflix has paid Verizon for direct, Netflix-to-Verizon-and-back transit. Netflix has not paid Verizon to upgrade its peering with Level3.

      • That’s probably the best comment I’ve seen here

        I know Mark has posted that it’s an “agreement” between two interconnects. It sounds more like a gentlemen’s handshake that Verizon may not be agreeing to, but the main issue as you say is

        “Netflix has paid Verizon for direct, Netflix-to-Verizon-and-back transit. Netflix has not paid Verizon to upgrade its peering with Level3.”

        Because it brings up a more poignant issue which is, other cable companies like comcast/ time warner probably also do the exact same thing when routing through level3.

  11. Pingback: Did Verizon accidentally admit it's slowing down Netflix traffic? Level 3 thinks so | Content Generator

  12. Thank you for clarifying the issues and for pointing out the details the Verizon neglected and/or omitted. Glad I ditched them years ago.

  13. This back and forth is getting really tiresome, neither this post nor the ones from Verizon are giving anywhere close to a complete picture of how the business relationship has been conducted in the past and present. Offering to pay for Verizon to upgrade their ports is like one of your customers offering to pay for the ports / cross connects to your network without any ongoing usage costs. No, its not an exact comparison, but it’s close enough for this grand-staging effort being conducted by all three levels (content producer, transit, residential ISP).

    As to the “suggestions” that this is an intentional effort by Verizon to thwart NetFlix, the simple questions can be asked… Why are you selling a service (transit) to NetFlix that you know very well in advance that you cannot fully fulfill? If NetFlix is being targeted by Verizon (or Comcast, AT&T) unfairly, why have they and you elected not to pursue any legal remedies to date but rather instead wage a public “public relations” campaign?

    The latest tactic appears to ask the US government to intervene, which may or may not be a bad thing, but remember most consumers only realize their NetFlix experience sucks and doesn’t understand or care about peering in any shape or fashion. Regulation may come, but the government is not known for subtle or light gestures, its quite possible all parties involved will find themselves in a much worse position afterwards. Let’s also not forget, the regulation may not stop at residential ISPs, interconnections are far more important than any single ISP.

    To summarize: I believe all of the parties involved are simply trying to look out for their own interests and profits, rather than any genuine concern about how the Internet works. None of the parties to date have released any meaningful data publicly to support any of their claims, but rather have released highly summarized data that can not be analyzed or verified by any uninvolved party.

    • Aaron, we have a settlement free peering agreement with Verizon that has been in place for many, many years. Each party has an obligation to augment the capacity at those interconnects to avoid congestion. As my diagram makes clear we remain willing to meet that obligation. When we sold services to Netflix we most certainly did not know in advance that Verizon would no longer continue to act as the good partners that they had for many years prior to that event.

      • Yes, the settlement free connections were established years ago for mutual connectivity benefit, and it appears from both you and Verizon that these links are sufficient for normal bi-directional traffic.

        You sold Netflix service on the assumption that you can abuse these settlement free links with a dramatic and completely predictable imbalance of traffic and now complain that Verizon isn’t rolling over and, at no cost to you, helping create further imbalance by opening up additional ports which will effectively be entirely unidirectional.

        Now, if you can tell me that Verizon has refused to consider opening up non-settlement-free ports that Level 3 will pay for the traffic on then I think we can consider that to be deliberately obstructive. As it is, it looks like Level 3 is abusing the settlement-free links in a way that was never intended when they were created.

        The fact that your customers, like Netflix, are stepping up and paying ISPs for links that you aren’t willing to pay for only reinforces this perception.

        • Tom, the traffic imbalance is a smokescreen. We have always said Level 3 should share the cost of moving traffic between our paying customers and Verizon¹s paying customers. If we do that why should there be an additional arbitrary toll?

      • Hello Mark, Can I assume that this settlement free peering agreement was in place before Netflix existed? Can I also assume that when the peering agreement was put in place the balance of traffic was equal? Now that Netflix streaming exists how has this balance of traffic changed? When does the peering agreement end do you expect it to renew a settlement free?

        • Scott, I’ve mentioned elsewhere in these comments how a simple send to receive ratio between a broadband ISP and a last mile network does not work if the goal is to determine that each party shares the burden of exchanging traffic between their respective paying customers.

  14. This congestion only takes place between Verizon and network providers chosen by Netflix. The providers that Netflix does not use do not experience the same problem. Why is that?

    This crosses the line of credibility. The amount of traffic Netflix generates in the US dwarfs any other source on the Internet. You know that.

    Of course you want Verizon to give you more ports so you can charge Netflix more money for sending more data. Especially when the alternative is Netflix cutting out the middle man, i.e. you.

    • Nick the statement that congestion only occurs with providers chosen by Netflix was actually made by Verizon in their blog post. I simply repeated it and asked how could that be? Of all the Internet backbone providers in a massively competitive market (unlike the broadband ISP market) that only those connected to Verizon carrying Netflix traffic are blocked, but those that don¹t are not blocked.

      • “Of all the Internet backbone providers in a massively competitive market (unlike the broadband ISP market) that only those connected to Verizon carrying Netflix traffic are blocked, but those that don’t are not blocked.”

        Maybe those other backbone providers don’t push hugely unbalanced traffic onto their network because they aren’t a source network for Netflix the biggest source of send data in the history of the world???

        • Every interconnect “pushes massively unbalanced traffic onto Verizon’s network” by that definition. Though you need to replace “Pushes” with “provides for free to Verizon to deliver to their customers who pull it from the other networks” for it to be accurate. Verizon has millions of customers who on average will request many times more coming onto Verizon’s network from the interlinks than Verizon sends out over those interlinks.

    • If Verizon and Netflix implemented TOECDN within their networks they would get less traffic to/from Level 3. Less traffic less pay and with TOECDN you would have happier customers as well.

      Since no one is interested in solving the main issue here, that is, distributing static content, we are going to see these kind of blog post between ISP:s every now and then.

      (The concept of TOECDN is released in public domain. There is no middle hand, only content provider and the ISP directly).

      • Fredrik, I applaud your concept. We have looked at similar things over the last few years. User acceptance and copyright laws have precluded a wide deployment though. I wish you well.

    • Netflix is not trying to force unsolicited traffic to verizon’s customers (or anyone else). The fundamental issue here is that Verizon’s customers are ‘asking’ for all of that content – to which, ‘yes’ — both Netflix and Level3 are happy to oblige.

      It seems to me like Verizon is trying to double dip here — as both a last mile internet provider and a tier 1 backbone provider. Verizon – or any other ISP, should not be able to charge for data entering it’s network — that was requested by it’s network!

  15. Pingback: Did Verizon accidentally admit it's slowing down Netflix traffic? Level 3 thinks so | Tech Auntie

  16. Pingback: Level3: Verizon Intentionally Causing Netflix Congestion - »

  17. “This congestion only takes place between Verizon and network providers chosen by Netflix. The providers that Netflix does not use do not experience the same problem. Why is that? ”

    Could it be because those other network providers aren’t serving by far the single largest user of bandwidth on the Internet?

    • If you do the math on it, it still doesn’t make sense. If Netflix is providing a third of the data on the internet, then the rest of everybody is providing two thirds. That means 200% as much. That 200% as much is consuming 44% of the links with “everybody else”, which means that everything else is getting over 4.5 times as much space for only 2 times as much traffic. Eg, if Level3 is allowing Verizon to connect 10 cables to them, other people are allowing Verizon to connect 45 cables to them combined. But 23 of those 45 cables are unused on average.

      Those cables are already there. Any few of them could be moved to connect to Level3 instead, and Level3 would not charge Verizon anything to do it. Adding another cable to anybody at all would cost Verizon as much money as they make from about 20 customers in a month and cost it only one time. Basically, 100 customer bill payments out of the millions of customers Verizon has in Los Angeles would fix the problem, and would not do anything bad to the rest of Verizon’s network at all, since the rest PLUS Netflix is using only 56%. Even 33% more would be 89%, and there is not 33% more anyway.

      The agreement with Level3 says that if Verizon says “I want to connect more lines to you”, Level3 will let them do so for free and will make sure the places to plug that in are on the Level3 side. So Verizon just has to spend the payments of 100 customers one time and everything is fixed.

      But simply put, if people can get and enjoy Netflix, why would they pay $150 a month for TV from Verizon to have to wait until Thursday at 3 to see the movie they want to watch? Make sure people can’t get Netflix easily and they have little other choice but to pay that $150 if they want to see that movie or any others. So just don’t bother to spend a hundred bills to fix things and make more money. Bottom line is better. Customer is paying $150 to Verizon to wait five days for the movie instead of paying $10 to Netflix to get it when they want to. Verizon is happy. Customer? Who cares? They can’t exactly get high speed internet anywhere else.

  18. Hey! This was an amazing article and wanted to thank you for providing me some extra information before I switch providers, but at the same time, I think ISPs (in general) all should upgrade their networks, at least making USA a competitive in terms of the speeds against other ISPs in other nations. Its just sad that we have to spend all this time with arguments, debates and blame game, when we could all just focus on increasing the amount of data from every stand point (Datacenters to users).

  19. I’m on the east coast, sysadmin, I’ve been proving to Verizon that their connections are failing for us at times from their network to Level3. The last ticket they admitted there was an issue with a port card and that it was replaced. I was told ‘alter’ is an old name for some of their equipement from MCI I think.

    4: ( 5.729ms
    5: 0.xe-7-3-0.BR1.IAD8.ALTER.NET ( 14.520ms
    6: no reply
    7: ( 14.390ms asymm 11

  20. Clearly is Netflix fault for pushing that imbalanced amount of traffic. The poor Verizon users have nothing to do with it. They didn’t requested that traffic, so Verizon has no obligations here. Netflix is sending unsolicited imbalanced traffic to verizon users, forced to see videos, so totally ok to request more $$$

    Hint for netflix: start sending upstream packets from clients filled with garbage to balance the up/down traffic, and then they will improve the connectivity.

  21. Perfect solution. Maybe we can just start a kickstarter for Verizon’s cards. I’m sure they’ll love the resulting PR. Maybe we can get the potato salad guy to chip in some lunch for the 5m it will take to install this stuff.

  22. Thank you for the excellent post, some bells need to be rung over at Verizon. Verizon’s BS posts are damning to anyone with experience in the industry.

  23. Mark, surely there is a group within Verizon that is responsible for provisioning these ports. Is it Level3’s belief there is someone very high up at Verizon who was engaged on this before the ports filled up and said “oh this is Netflix traffic don’t provision more ports?” That group has to be foaming at the mouth with wanting to provision these ports and level off the traffic. Level3 should offer a bounty for an engineer at Verizon who will work with Level3 to provision more ports, assuming that means they lose their job for doing so. Offer a bounty for anyone who can take a picture of Verizon’s equipment showing they have unused ports sitting there.

  24. I work for a Canadian ISP. Maintaining a lack of network congestion is difficult, but it’s entirely doable. I don’t work on the back end but I know that we have to install a lot of hardware and often have to run more tiber optic cable to various areas so that the various areas aren’t saturated. It’s not cheap, though relative to the cost that we charge customers, it’s not bad.

    TV providers don’t like Netflix, mostly because they offer customers a lot more content (not necessarily always the content you want mind you) on demand for a really reasonable price.

    What providers are afraid of is Netflix’s continued growth of popularity, and the depreciation of value of their TV options. Netflix makes the expensive hardware, back end and customer service reps that Verizon has already paid for seem VERY expensive in comparison, partly because it is.

    Everyone knows that they’re doing this to make Netflix’s inexpensive service seem shoddy, they’re not remotely interested in the pennies they could get from Netflix, verses the dollars they can continue to charge consumers.

  25. While I agree with your dissection of the “jeeze, just plug in a couple more ports” stupidity, there is one piece of data I’d like to get out of level3 – how much delay and packet loss does a 100% congested port on level3 actually have in this case?

    (while I also agree with overprovisioning always, in the core! as a member of the ietf aqm working group I’d like to know what is happening when you are not overprovisioned, as in this case.)

    5ms? 100ms? what?

  26. Once again all you’re doing is talking which will change NOTHING at these scumbag companies. We need Google Fiber to come in and destroy them. HELP GOOGLE FIBER DESTROY THEM, PLEASE

    • Morgan we have been doing a lot more that talk. We have be active participants in the debate with regulators and we have spent a lot of time an effort to negotiate agreements with many, many interconnect partners.

      Your point, however, about having more competition for broadband consumers is certainly one I do agree with. I don’t want anyone destroyed I just want there to be more choice.

      • ok so when you are talking with regulators do you just completely ignore the fact that they are bought and paid for by Comcast? Tom Wheeler being appointed FCC chairman was a calculated move by Comcast. The FCC is full of past and future Comcast employees. Comcast spent $18,810,000 on lobbying in 2013 and so far this year Congress has received $2,981,976 from Comcast. A Comcast PAC gave money to every senator examining the Time Warner Cable merger!! Why do you even bother? Are you really this naive? Comcast has spent more money buying politicians than they’ve spent on infrastructure. If you want to actually help, take those ports you offered to give to Verizon and give them to Google Fiber. Call Milo Medin right now and ask him what you can do to help them roll out nationwide. That is what we want to hear from Level 3. Why can’t that happen?

        • Morgan, we have to work within the legal and regulatory system that exists. Verizon’s customers are the ones requesting content through those congested links. Moving them would make matters worse for both our customers and Verizon’s.

  27. I spent half of the 90’s fighting battles like this. Sadly level 3 (or better said their acquisitions) were on the other side of the conversation.

    For example:

    Level3 released a press release in 2005 saying that Cogent communications was sending “far more traffic to the Level 3 network than Level 3 was sending to Cogent’s network”

    Now Level3 claims that it is wrong for Verizon to do the same thing..

    • Steve, we are being consistent here. We had a disagreement in 2005 with Cogent over how far we each carried the traffic flowing through our interconnects. We resolved that within a very short period and traffic has been flowing through unfettered ever since. In Verizon¹s case we have also said that we certainly do not expect a free ride. That in fact Level 3 should share the cost of carrying traffic between our paying customers and Verizon’s paying customers. We reached an agreement quickly with Cogent to do that. We have been unable to get that agreement with Verizon.

      • Hi Mark,

        Does this mean you know how Verizon’s network is engineered and that you are carrying traffic x distance further where x = the multiple of the traffic imbalance? That would mean you are providing cold potato traffic distribution for Verizon, taking traffic to the closet peering point to the destination customer (based on information provided to you by Verizon). If I am correct in my translation, this would be a reasonable contractual agreement to deal with a traffic imbalance.

        • Steven, both networks involved in a bit mile agreement have to share (with audit rights) the data for their respective network. There is no need for one network to know how the other network is architected. It is, however, fair to say that in most cases the source and destination IP addresses mean we have a pretty good idea. And yes, one of the ways you rebalance miles is to move away from hot potato to cold potato routing.

          Two co-operating networks can change how destinations in one network are “announced” to the other network. That has the effect of increasing bit miles on one network while reducing them on the other. It is one of the adjusting factors within a bit mile agreement that keeps an equitable sharing of the costs associated with moving traffic between the two network¹s respective paying customers.

  28. Pingback: End of the Internet (or end of net neutrality)

  29. I’m not sure what the likelihood of this being read is but from my network perspective (Verizon FIOS) this article is largely accurate. As of recent whats been confusing me is is the unreasonable and unusable routes taken.

    It is well known that Netflix is in the eastern AWS region. Any routes my traffic takes that come remotely close to crossing that area, including IAD, takes a route through a ‘peering arrangement’ (sarcastic use of quotes) with comcast! I’m no CCIE but I have never seen two carriers who compete in the same market for the same subscriptions exchange traffic in this way. It is not until my traffic hits the comcast network that it becomes latent. To me, this arrangement is not an accident.

    I am at a massive disadvantage here. As an engineer:
    – I need frequent and fast access in and out of the AWS cloud to maintain my products.
    – I need frequent and fast access to github to do my work! There are nights where it will take upwards of an hour to sync ~800kb just to continue working.
    – I pay a premium on Fios to eliminate these problems.

    Unfortunately, it takes carriers the size of Level3 to say something. I applaud level3’s position and feel better about the situation knowing level3 cares about the issue.

  30. 1) Most last mile technology is inherently asymmetric, even FiOS GPON. Traffic can NEVER be balanced to most last mile networks.

    2) Traffic balance rules were established long ago to protect long haul networks from the effect of hot potato routing by BGP. Such rules are being incorrectly applied to traffic flows to last mile network operators.

    3) Geolocation systems today select the source server closest to the requesting client. So, in many cases, the content is being handed to the last mile operator (eg Verizon) IN THE SAME CITY as the destination client requesting the content.

    In the literal sense, Verizon is refusing to carry traffic from one side of Los Angeles to the other, unless some one pays them a bribe.

    The (not insubstantial) payment Verizon receives from their own FiOS customers is not enough for VZ management. They want to tax content suppliers even though VZ is not bearing the cost of long haul transport of the content.

    4) Maintaining congestion on network interconnections causes traffic flows to experience unequal quality. Suppliers who don’t pay a bribe to VZ don’t get through. This is certainly NOT net neutrality in action. In any other setting this would be clearly identified as extortion.

    5) FiOS customers are being significantly mislead by Verizon. This seems like a golden opportunity for yet another customer class action lawsuit against Verizon. If government worked right, The FTC and state Attorney Generals would begin criminal proceedings against abusive last mile operators.

  31. Pingback: Level3 (network backbone) Confirms Verizon is PURPOSELY Throttling Netflix Traffic. | My Blog

  32. This constant back and forth between all providers and netflix about congestion is getting old.

    Instead of pointing fingers, someone should address the market dynamics leading to this situation. Settlement-free peering works, because it’s usually tit-for-tat.
    But once one customer represents 1/3 of Internet, it clearly starts to feel unfair if one party gets paid to transport it, other party does not get paid to transport it.

    The fundamental problem is, that ultimately there is strong incentive to drop bits, as every delivered bit eats into your margins, because no one is getting paid for the bits.
    Because the bits actually have a cost and delivering has a cost, this makes situation unviable. Right now solutions how the unviable situation is handled

    1. low/average users subsidize heavy users => unfair
    2. users are indirectly billed via content providers (netflix/youtube have to pay premium to comcast/FT etc to access their network) => inefficient
    3. packets are dropped instead of delivered => low quality

    We could give financial incentive to deliver bit, and all these disputes would be gone, because you’d love to get traffic, since you could bill more.
    But for some reason metered INET is curse word, because when we’ve had had it, charges have been unfair, fair charge today would likely be 0.1cent < 20cent per GB, you'd also need low fixed base price to address various fixed costs, which could contain some GB of traffic as base-product.

    Sure there is alternative solution, which US political climate will never-ever accept. Monopolize infrastructure, monopoly stake-holders would be service providers, with no own employers. Monopoly building costs would be covered by customer-share of the network, if you have lot of customers, you pay larger share of the infra. Incentive to build fast and act fast exists, because stake-holders can't sell without monopoly building.
    Monopoly should not be profitable, stake-holder costs should reflect actual building costs by regulation. What service-providers charge their end-customer would be free.
    Anyone building new access network would have to connect to nearest monopoly pop, get and give access to use the monopoly infra at same rate as everyone else.
    This way flat-rate MIGHT still work, because everyone has same cost ultimately and consumer can choose any provider.

  33. I appreciate what BH is saying but his analogy is false. According to his reasoning, the consumer has already paid their ISP for the right to drive a big rig at the maximum allowable speed limit on the highway but their ISP is only allowing bicycles up on ramp from some providers.

    The consumer is being being treated unfairly by their ISP in this situation. Consumer level ISP connections are asymmetric as consumers are net downloaders of data. The ISP knows that their customers are net consumers of download bandwidth as the ISPs sell asymmetric connections and their terms of service usually do not permit uses that would increase net upload traffic (such as running your own commercial server off your household access). ISPs also price internet connection to consumers based on the speed of the connection and not on the type of data or volume of data on the connection. If the consumer is not being delivered the speeds that they have been contracted to receive from the ISP, it is the ISP’s responsibility to fix the problem. Whether the data is video or text, it is all the same to an IP connection. Aside from attacks on the IP infrastructure, there is no basis in reality for the ISP throttling the consumer’s connection based on the type of data traveling through the connection.

    It boils down to the fact that the consumer is paying Netflix for a license to view content and paying their ISP proportionately for the speed of their connection. If the consumer paid for a highway, they should not be provided a dirt road.

  34. Hey, if they want to pull the balanced traffic card to explain why they want to be paid to transfer third-party traffic into their network I have a perfect solution: their customers should do the same.

    “Well, mr Verizon, I just had a look at my traffic stats and noticed you’re pushing an awful lot of traffic into my network, while I hardly send anything back. I think we should renegotiate this here broadband contract, the numbers just don’t add up. How about you pay me to deliver broadband to my network instead of the other way around?”

    • If you look at all the regulations around cable system transmission and retransmission fees, made recently relevant by the USSC Aeros decision, you will quickly find that you are incorrect.

      Further, if you look at Google Search Engine and Gmail, you will see that nothing is free; Google is being paid by advertisers, and the product is you.

  35. As a FIOS subscriber in the Mid Atlantic, I’ve noticed the effect of this between Verizon and Netflix.

    What can *I* do about it?

    Because anything on Netflix, 24/7 is affected.

  36. Mr Taylor,

    You’re right on. I’ve been in the datacenter business for years, and recently attempted to explain this in a reddit comment to an earlier post about this, but no one understood how easy this is to fix. OK, if there are no remaining ports it’s a little more to than just adding a SFP and a cable, but still – even in the worst scenario, it’s a new router on either or both ends to add the peering capacity. The problem can easily be solved technically. The problem won’t be solved, if their business people insist on trying to avoid fixing the problem. Honestly, this is quite shameful for Verizon. Peering agreements are supposed to work to provide mutual benefit. I get it – “mutual” is blurry in this situation, but you have to take the good with the bad. It’s always been that way.

    I hope more and more technical people who can articulate the facts like this will get more screen time. The problem can be fixed. Thanks.

  37. The value of a network is its ability to connect unfettered to as many other points as possible. I would think that a last-mile provider would want to ensure that their customers receive the best possible service to all potential end-points. That is a great theory but it loses a bit in practice. First is the fact that Verizon provides its own video service. Could it directly or indirectly allow Netflix services to degrade in favor of its own service? Possibly. Another more likely possibility is that some District Manager or Director just doesn’t want to spend the money to provision another 10G port? It seems like such a simple solution to add more peering capacity but I have seen it all too often where peering is left congested because working out the port assignments, changing routing tables, etc. is too much work. The operations people have to get network engineering involved and then they have to contact their counterparts. It simply is too much work to be bothered. I’m just reporting a possible explanation not excusing it. I have seen this at telco and MSO too many times not to bring it up.

    As a consumer this is frustrating because they don’t have the tools or ability to solve these issues. I recently did experience some congestion reaching a certain provider in CA that co-incidentally transited through Level 3. Using ping it looked like my traffic was transiting through Paris but when actually sending TCP and UDP packets I determined the real route. It was much shorter. The access provider tried to blame the out-of-network providers, but I was able to show congestion on one of their local links. It took 2 months of me reaching into network engineering of this company to get them to add another 10 Gbit/s link, but it was done and my round-trip latency dropped to 18 ms. Now the typical consumer can’t do this so what is the solution?

    For too long residential broadband has been a best-effort service with some minimum bandwidth requirements. If the carrier could do a TCP throughput test and show that they could deliver that bandwidth to some point in their network then they were off the hook. I believe that service level agreements should be added to residential broadband services that specify not only minimum bandwidth, but maximum latency, jitter, and packet loss ratios to points just outside their networks. That solution would not address specific throttling issues with certain end-points though. This is why I advocate open access last-mile networks so we can have true facility competition in the last mile. Level 3 can’t get by with this behavior because customers have a choice of other long-haul carriers. If cities would build the fiber infrastructure then they could sell access to any service provider on a non-discriminatory basis to provide true service competition.

  38. When you initially set up shop with an ISP, do you not sign some kind of contract that enables you to scale up your bandwidth for an agreed upon cost?

    The “hey, just go plug in another cable” suggestion sounds simple enough, if you ignore some realities of how business works. For example, if I go to Whataburger, it would be pretty easy for the guy to throw an extra patty on my burger. He won’t do that without charging me a little extra though, for obvious reasons. It’s the work that his company sells.
    Your offer to pay for the cost of a new card conveniently doesn’t include ongoing maintenance costs associated with that card, which is where I suspect some of the contention comes from.

    • Mike, a peering agreement, like the one we have with Verizon, contains obligations on both parties to augment (add) interconnect capacity before there is any congestion. The language typically says something like “each party shall add interconnect capacity if the utilization of the ports at a location exceeds 70% over five or more days. That capacity shall be turned up within 30 days.”

      • Mark, if Verizon is breaching the peering agreement as you say then why not sue?

        One of the top comments on reddit is
        Hey Level3, you want to prove your point. Send out a message that you will be at the interconnect in 5 days with 4 10gbps cards and the cables. Live stream your guys waiting there for Verizon to simply allow you hook it up.
        Reddit will watch. Others will watch.

        • “Mark, if Verizon is breaching the peering agreement as you say then why not sue?”

          Exactly, and I’ll tell you why L3 won’t: because Verizon is not violating the terms of their agreement. I guarantee that their agreement stipulates that the upgrade obligations are conditioned on there being roughly balanced traffic flowing between the networks.

          “One of the top comments on reddit is”

          Which completely misses the actual point and demonstrates why L3 is playing this shady PR game very well. Yeah, upgrading the interconnect itself wouldn’t cost too much, but then the relationship is no longer an equitable peering: L3 is pushing much more onto Verizon than vice versa. So, not only will Verizon need to upgrade the interconnect but also a good bit of the downstream network as Netflix continues to grow. Meanwhile, L3 gets paid more and more for Netflix’s usage while Verizon sees none of that revenue but incurs the resultant costs.

          • John, there are two networks involved here. An equitable arrangement would be that the two interconnecting networks share the burden of carrying the traffic from one customer to the other. Level 3 does not want or expect a free ride. We are more than willing to carry as much traffic as the ISP at least as far as they do.

      • Are those peering agreements not legally binding documents? If so, isn’t Verizon’s refusal to add capacity a breach of those contracts and legally actionable?

        • So you’re going to suggest that two of the largest telecommunications companies in the US start suing over 4x10GbE links? That’s the quickest way to ensure that costs for customers skyrocket.

          The most efficient way is handle this is a PR campaign when at least one company has enough humanity to recognize when they should be ashamed of their own actions. Sadly, this is lacking in the US right now, and so we see these little back-&-forths over blogs. Neither of these companies have the wisdom to step back from that ledge that they’re trying to throw the other guy off of.

  39. Pingback: Verizon blaming Netflix for slow streaming speeds is an ‘attempt at deception’ says Internet backbone provider |

  40. Pingback: Did Verizon admit it's slowing down Netflix? | Android Authority

  41. For those suggesting Level 3 should pay, remember that the consumer already pays their ISP for the “last-mile”.
    If Verizon doesn’t have enough bandwidth to handle all of it’s customer’s streaming Netflix then it’s Verizon’s fault as the customer is paying Verizon for the last-mile usually for x GB worth of usage.

    Verizon is trying to double dip.
    Level 3 is only handing off traffic to Verizon, that Verizon’s customers have asked (and paid) for.

    Imagine if Level 3 turned to Verizon and asked Verizon for more money to deliver Netflix to their customers?

    Netflix pays Level 3, The customer pays Verizon.

    • Level 3 should not pay; Netflix should pay. The problem here is that, if Netflix pays to properly solve this problem, Level 3 will take a direct hit to their revenue streams. Whereas, if Verizon solves this unilaterally, then Verizon will take a direct hit with increased costs. No winners here, just losers all around.

  42. Pingback: Peering Dispute: Level3 Says Fix Would Cost Verizon A Few Grand Per 10G - Telecompetitor

  43. Pingback: droidultimate » Did Verizon admit to slowing down Netflix?

  44. Pingback: Level3 called out Verizon on their dickishness by DMAUL - TribalWar Forums

  45. Netflix sends out an unprecedented amount of traffic.

    People seem to be missing the point about what Netflix is sending to Verizon. Netflix is not “sending” data to Verizon on just a whim or on speculation. At the lowest level, Verizon customers are initiating TCP connections to Netflix servers and requesting content. Verizon customers, as part of their usage of the service they paid for, are requesting content and Netflix just happens to be popular. Netflix also pays a fair amount to get their “unprecedented amount of traffic” on the wire from their end – and are paying for that bandwidth delivery.

    Mark has clearly pointed out that Verizon is intentionally limiting the bandwidth through Level-3. Verizon may publicly blame Netflix, but anyone who really understand networking knows that they are really throwing Level-3 under the bus. There’s no reason why Level-3 should let that stand if they are willing to add capacity and it’s Verizon keeping that from happening. Should Netflix send data using other exchanges, it wouldn’t surprise me to hear Verizon making the same claims, yet now the problem is at another interchange point.

    Given that most consumer network activity (especially the web) has a higher download than upload ratio, Verizon is positioning in this can be used against any content provider, not just Netflix.

    • “Should Netflix send data using other exchanges, it wouldn’t surprise me to hear Verizon making the same claims, yet now the problem is at another interchange point.”

      Verizon would prefer that Netflix buy access directly from them so that what Verizon is paid scales with the usage that Netflix is causing. If what they are paid does not scale with the usage of Netflix (e.g. – the way L3 would prefer it), then Netflix can drive them into the ground or force them to continually raise end-user prices to handle the unprecedented load Netflix is causing.

      “Given that most consumer network activity (especially the web) has a higher download than upload ratio, Verizon is positioning in this can be used against any content provider, not just Netflix.”

      Not really. Netflix truly is a singular, special case right now. Netflix accounts for something like 33% of all Internet traffic during peak hours. That’s unbelievably ridiculous. Youtube is a distant second at around 10%.

    • “When you owe the bank a million dollars, you have a problem. When you owe the bank a hundred million dollars, the bank has a problem.”

      If you’re willing to destroy your credit rating over a pixelating TV show, be my guest.

      Netflix and Verizon are waging a PR war over some calculus about who has the most to lose. Level 3 is helping Netflix because it has a hell of a lot to lose.

  46. My issue with Verizon (FIOS) is rather different. I notice that speeds get progressively downgraded as usage increases during a short window of time, a few hours, for example. This seems like congestion, if I hypothesize other users.

    But then, I open a web tab to speedtest.

    In about 30 seconds, the speed of all the connections on the machine increases to its original rate and stays there for about 10 minutes, after which it collapses again.

    That requires some explanation.

  47. Hey Level3, you want to prove your point. Send out a message that you will be at the interconnect in 5 days with 4 10gbps cards and the cables. Live stream your guys waiting there for Verizon to simply allow you hook it up.

    I guarantee you — people will watch. You might even get a crowd.

    (First part is copied from Reddit comments/ second paragraph is me.)

  48. Pingback: Verizon blaming Netflix for slow streaming speeds is an 'attempt at deception' says Internet backbone provider – Health and Fitness

  49. I just wanted to say thank you so much for writing this up. Sad to see how much flac you are catching in the comments.

    Perhaps one go argument to show what BS to all the crying about a downstream provider like Verizon getting more downstream traffic off your network than they send up is that other companies in other countries and the US that you work with don’t seem to find the traffic imbalance that debilitating.
    Not to mention the fact that Verizon sells imbalanced network access to their customers, with higher download speeds then upload, and by blocking ports to prevent web and mail hosting on many of the connections they sell. Given that they do this they should EXPECT that their upstream traffic would be lower than downstream and what they charge customers and how the design their network should take that into account.
    And seriously, 10 Gbs ports? I would have thought we would be running at least 40 Gbps SFP+ if not 100 Gbs connections by now, we certainly do in the internal datacenters I work in.

    • Scott, I don¹t consider this flack. One of the reasons we decided to have this open discussion is to engage in a conversation over a complex topic.

      On your point about port speeds; 40Gbps port cards on edge routers cost more than 4 x 10Gbps cards so we haven¹t deployed them. We have now reached the point where 100Gbps port cards are just less expensive than 10 x 10Gbps cards. But, both sides of the connection need to be at the same rate. Our first implementation of 100Gbps for Internet interconnects was with Cablevision as I mentioned in a prior post (

      • Thanks for responding, glad to see you take it all in stride.
        Something a bit silly you guys could consider to fix this problem would be to setup a system just on the other side of your peer link with Verizon that accepted traffic in a bittorent client (you) and server (Verizon customers) sort of fashion that took the traffic and immediately dumped it to dev null.
        This would have the effect of balancing the BS traffic asymmetry Verizon is complaining about. Once there is no more asymmetry they don’t have anything to complain about and it wouldn’t add any significant load to your network. I am willing to bet there are many many Verizon customers (myself included through my landlord) who would happily help you out by uploading to you to balance the asymmetry.

        *correction for above I meant QSFP, not the SFP+ used for 10Gbps.

  50. I’m no network engineer, but could Verizon’s reluctance to increase their Level3 interconnect capacity be partly due to the need to maintain network overhead? Yes, Verizon has plenty of capacity now, but if they add in those other 10Gbps cards, how much will it increase? The diagrams from both Verizon and Level3 seem to indicate the respective networks run about 50% capacity at peak usage, while the interconnect is maxed out at 100%. As Mark Taylor seems to indicate, Verizon could double the interconnect capacity so presumably it would also run a little over 50% (since it’s over-congested now). How much would that increase Verizon’s network traffic, if in fact Netflix accounts for 1/3 (or 2/3 or whatever the statistic is) of peak traffic. Presumably, Verizon wouldn’t want that utilization number near 100%, so there’s some truth to the fact that this specific interconnect may represent special problems for Verzion’s network (compared to other peers), but it depends on the details. What’s the ideal number purely from Verizon’s own technical needs?

    All this is beside the point that others have made that customers are paying Verizon to deliver that content, and the reality of the internet as it is used today is that delivering high-definition video from Netflix, YouTube, Amazon, Hulu, etc. etc. is the biggest bandwidth hog and will probably stay that way for a while. Furthermore, home broadband has ALWAYS had asynchronous speeds, because home users always request more data than they send, so Verizon’s arguments a la “traditionally, peering agreements…” don’t make much sense in the context of network-to-ISP interconnect. Is Verizon really claiming that bits going one way cost more than those going the other? From this perspective, Level3’s argument makes sense.

    Also, doesn’t Netflix specifically offer a caching server to install in data centers so ISPs can reduce traffic coming from Netflix? Or is that not relevant in this architecture?

  51. Pingback: Why the consumer is still held hostage in peering disputes — Tech News and Analysis

  52. So I’ve been referenced quite a bit in some follow up posts here and at first I must say while there has been disagreement, the entire debate has been 99.9% civil which is overall refreshing for a comments section. To that I congratulate all!

    My posts were not meant to be the answer, and I am trying to formulate my own opinion on the matter starting with how I see it. Many of you have offered counter arguments or clarifications that have helped to sway me, but I’m not sure I am fully there. As to the accusations I am a Verizon shill, I assure that’s not the case, I honestly don’t care who pays and gets paid in this scenario, as long as it doesn’t hurt the consumer. Isn’t that what “net neutrality” is supposed to be about and what the government is stating…that no harm should come to the consumer.

    So, for sake of argument, let’s say I agree 100% with Mark and his fervent supporters. There are two things that I think need to be brought up.

    The first is the idea of the peering arrangements and equitable exchange. As others pointed out, I’m sure that no one expected a single company to monopolize 2/3rd of the Internet bandwidth and for such an asymmetrical exchange of data to be occurring. It would seem to me that there should be some “best-faith” clause in interpreting this. The very fact of the Cogent situation in 2005 shows that contracts aren’t always made with growth in mind.

    This brings me to the second question. Presumably it isn’t hard for Verizon to connect up a few more ports to double the throughput, but remember that increase in traffic affects all pipes down the line. If they opened up that additional bandwidth to every peer, their link becomes saturated.

    So (and here I hope any suspect of my being a shill for Verizon disappears) – what I am concerned about as a consumer is this… What would really happen if every single person right now tried to maximize their guaranteed downstream bandwidth? Could the ISP support it? Could the backbone providers? I think not.

    So let’s think about how this actually affects consumers. If we have unfettered peering that only increases the used bandwidth on the ISPs services. Even if we say that Verizon, et al are the ones responsible for this, the simple fact is they can’t handle it. They will fall apart. Consumers will suffer in outages and in higher ISP fees so they can build out their infrastructure.

    You think Verizon should pay for all this? Fine. Realize though that *does* hurt consumers because consumers are the ones paying the ISP.

    Point being, let’s not look at this in isolation, but what it means in practice and how the cards will fall when all those links get connected to all the content providers and from all the backbone providers.

    Maybe the federal regulation should also be focusing on ensuring that ISPs don’t oversell bandwidth and can support 100% x 24/7 on the rates they sell to customers along with caps on what these providers can charge consumer’s to subsidize this network.

    In the end however I still have this nagging feeling that if a company is going to make its money by selling goods to consumers then that company should share the largest burden in paying to get those goods to the customer. Netflix is trying to sell a good over an infrastructure that wasn’t designed to distribute it with such quantity and I don’t believe they should expect that infrastructure to be subsidized by everyone else for them so they can make more money. If we take the argument that it’s just what the consumer is asking for, then I think we also have to accept that the consumer is going to have to pay more to get what they want. Regardless of what you think you pay for now from your ISP, the reality is they can’t support what we pay them for.

    So how do we solve it? Do we double our ISP fees? Does Verizon loose all its profits as it expands it’s network capacity? I think this is where the government should be focusing its efforts. In the meantime, I think we all should be a bit patient about opening up the floodgates, less we be drowned.

    • BH let me comment on two of your points.

      Yes, a large problem with historical peering agreements is that they never considered what would happen if things changed between the parties. And that left peers with some difficult situations when one party considered that the relationship was no longer equitable. The only option was for one party to disconnect and walk away. As we¹ve seen that isn’t good for customers who pay to use the Internet. So, a key part of our bit mile peering methodology is to include actions that each party is obligated to perform if the bit miles get out of balance. These new agreements are meant to be long term and meant to endure even if the business model of one of the parties changes or if some outlier like YouTube or Netflix comes along. The goal of these bit mile interconnect agreements is to maintain uncongested interconnects but also for each party to equitably share the cost of delivering traffic between each of their customers.

      Secondly, we often overlook the cost of delivering a bit of traffic – or more importantly how that cost has changed over time. Most of the equipment used in networks follows a trajectory very similar to Moore¹s law. There has been a dramatic reduction in the cost of carrying a bit over the years. That is ongoing and fairly predictable. And, despite the fact that higher speed services are being sold by ISPs, and despite the fact that we are all enjoying a richer experience as we use the Internet, most ISPs have seen an increase in their profitability. I have been involved in this business for over a decade and the one question I always get is related to the ability to continue to grow. No one seems surprised that the storage on our computers just gets bigger and bigger every year.

      We get more for less. But people do seem worried about that when it comes to the Internet. And yet for well over a decade thousands of network operators have figured out how to add capacity at between 50-100% a year. I see no reason for that to stop.

      • So the main question that arises then out of your clarifications here, and as you seem to allude to in some other responses:

        You seem to state that your interconnect agreement already has in place language that is supposed to prevent these congestion issues. You infer that contractually Verizon already has an obligation to connect up those additional interconnect ports. So why isn’t this simply breach of contract? Why is there not a law suit to enforce that they abide by their side of the agreement?

        It would seem (as others have mentioned) that Verizon could simply connect those ports to be in compliance with your contract and then implement some traffic shaping to their end users. Of course, if they did this they could be more directly accused of giving priority to certain types of traffic, but then it becomes a policy dispute and not a breach of contract.

        Why are they risking not abiding by your agreement terms now? Again, you seem to state that both sides have signed agreements that are meant to provide equitable exchange rates even in light of content providers as large as Netflix.

        To the other point – that of consumer bandwidth. Yes, of course, our needs increase and so has bandwidth. We certainly get higher speeds for the same $$ then we did 10 years ago, but 10 years ago there was no Netflix (streaming) and a year ago their were likely no 4K video streams.
        I still believe that the larger picture here is to assure that the ISPs can provide their sold bandwidth to all customers 24/7. Opening up more congestion to their networks if they can’t provide that is asking for consumer disruption.

        Unfortunately, while I still could advocate your “unlimited” peering arrangements, I think that following them to their logical conclusion is going to disrupt the ISP infrastructure and I think we need assurances from them (likely with government oversight) that consumers will not be affected by performance or price. I would like to share your nonchalance that the networks will just handle it, but let’s just say I’m from Missouri and L3 interests are in getting Netflix’s data uninhibited to Verizon, not to my computer.

        I think my main concern now is the myopic view being taken by each side. I will tend to agree that your argument is the stronger argument, but I still see some wiggle room. I definitely think that any federal regulations that seek to offer a guiding hand in what Verizon can and can’t do with interconnects like L3 have to also spill over into a higher degree of consumer protection, both in pricing, availability, and freedom to information.

  53. It’s a pretty clear cut case of gouging by Verizon, any way you slice it.

    Case : Verizon (and Comcast, and take your pick of provider) spams the airwaves with “Buy our service, enjoy movie downloads at blazing speeds and streaming content”

    Customer buys the plan, then lags terribly trying to access content that they paid both Verizon and Netflix to deliver.

    Verizon cries that Netflix is sending all this data, when in reality, the Verizon customer (whom paid for access to the Internet from Verizon, and then access to licensed content from Netflix) REQUESTED it.

    In my business I run, I either make good on what I’ve promised and contractually signed with a customer to provide, or I lose the business and/or get sued if the customer made any investments to receive the service(s) I failed to deliver. Verizon should be ashamed, and make good on what was promised to their customers to deliver, OR, stop advertising said access to services and refund customers that signed on expecting to get services they can’t reliably receive.

    Having owned and worked for many a colo provider myself with services, Mark is spot on. Provide the 10G cards to the Carrier Hotel facility staff, install it, and if there’s capacity issues either stop offering the service or increase capacity to continue offering the services, PERIOD.

    One other note, I see Netflix periodically being blamed (and, etc) for sending traffic, when in fact that is about the most clueless statement to ever be made. The servers that host the content send information REQUESTED of them. They don’t just start scanning the internet for addresses to blindly send the traffic. You can’t claim (as Verizon actively does in my market, along with other ISPs) to offer high-speed service to streaming content, and then on the other hand claim that Netflix is to blame. Arguably, without Netflix and the plethora of services available to be streamed (Apple TV, Netflix, Twitch, etc), no one would be buying the bandwidth (service) to being with if all we needed was “static” .html pages on some random website.

  54. Pingback: Netflix partner says Verizon slows traffic |

  55. I agree to that Verizon is double dipping. The consumer already pays for the bandwidth. Verizon network capacity has been finaced by the profits paid them by the consumers they are serving. The question is how much profit should Verizon get. Since there is limited competition, history has repeatedly shown that coporations will try to maximize profit not by competivie ratios but by how much the law and consumers will put up with. Consumers have spoken and dont want to shoulder the additional burden of Verizon’s selective pricing. The law must follow the people it serves. Simple.

  56. Bottom line here is that Verizon collects untold amounts of money from its customers who purchase a level of service (sold by throughput generally) to the network content of their choosing. Verizon is holding the internet hostage by effectively throttling data throughput that their customers are requesting in order to extort more money from backbone networks and/or content providers. This is classic double dipping enabled by the limited competition among ISPs and dirty pool in my book.

  57. There is indeed an imbalance here. Verizon should be paying *Level 3* for access. I worked in various large and small ISPs in the 90s through 2000. I negotiated peering for my clients/employers . I remember clearly that when I worked for a large dialup access provider that had no webhosting customers that we were mostly told to suck transit unless we were truly a peer or had significant (in 2000, 10 megabits/second sustained was significant) traffic to a particular network or had a T3 (45 megabit/second) down each coast and were at e.g. MAE EAST and MAE WEST. Everyone was scrambling to put T3s into Microsoft to take the pressure off the interconnect points, this was an example of private peering.
    Verizon isn’t primarily in the hosting business. They provide access. *THEY* should be paying Level 3 so that Verizon customers can get the content that Verizon customers want which is hosted on Level 3 network. Netflix already pay Level 3 to haul traffic. The same is true of Time Warner and Comcast. These are ACCESS companies. They do not provide web hosting companies with bandwidth. That’s what AT&T and Sprint among others do.

    The thing is that Verizon is so large an access provider that they feel they can extort money from the companies whose content their customers want. Is Verizon demanding money from Microsoft so that all the Verizon customers can get the latest patches for Windows? How about Apple, all that iTunes content is ginormous consumer of bandwidth, is Verizon holding a gun to Apple? No of course not, because Apple would squash Verizon like a bug if they tried (e.g. suddenly, Verizon Wireless would no longer have iPhones and iPads to sell to their customers who who would all jump ship to AT&T or T-Mobile…not to mention all those Apple stores preaching — excuse me, educating — consumers who would be told why iTunes doesn’t work on Verizon).

    • Why should they pay Level3 when their customers already (obivously) payed Netflix?

      Perhaps Netflix should pay Level3 so Level3 can pay Verizon? 😉

  58. The question here is: would Verizon successfully sell high bandwidth internet services to consumers if Netflix and other streaming services did not exist? I argue no because there would be no benefit to higher connection speeds. The reason I have a 20Mbps service from Time Warner Cable is in part to get unfettered access to Netflix and Hulu for a six person family. I expect with a 20Mbps down service, I get 20Mbps regardless of the source. Without Netflix or Hulu there’s no reason for the bandwidth we have.

  59. I work for a University in Virginia whose packets pass through Level3’s network. I live about 25 minutes from the University and have Verizon FIOS’ 75/35Mbps plan, and at night I cannot access educational content faster than 20KB/s (it usually bounces around from 4KB/s to 20KB/s). I’m guessing it’s due to users in my area accessing Netflix and saturating the ports, but it annoys me that other students paying for FIOS broadband access are being prevented from accessing their educational materials at even remotely decent speeds.

  60. I guess Verizon has other transit partners other than Level3.

    How come Netflix doesnt get those as transits aswell (unless they want to private peer with Verizon) and start to loadbalance between their uplinks?

    This way whatever cannot fit in the 4x10G link in LA will be offloaded through other transits…

    • None, they do as was shown in Verizon¹s diagram on their blog. The problem is all the providers chosen by Netflix are all 100% utilized where they connect to Verizon.

  61. A question for Mark:

    Now when this blog entry is roughly 24 hours old, any responses yet from Verizon?

    Did they accept your offer of getting 4 more 10G interfaces along with cabling which you also can install for them?

  62. I see this as a customer service issue. Verizon customers want to use Netflix. They pay for more expensive packages because they plan to stream video. It seems to me that due to a lack of competition, Verizon doesn’t need to worry about providing access to the sites that people pay them the money to access. If there were competitors and if there was more information available to consumers, a large chunk of their customers would leave them to go to another service.

    Since we don’t have much competition in that industry, the Internet is becoming like cable where you pay $90 for a bunch of channels you don’t want, a few you do, and a bunch you want but don’t get. Similarly, your Internet access gets you tons of access to sites you aren’t interested in and a lot of ISP apathy about getting you access to what you actually want.

    I would like to see Netflix publish recommended ISPs on their website to inform users.

  63. simple, don’t use verizon…
    there are alternatives, if you look for them…
    and I don’t pay anything for high speed access in the middle of nowhere…

  64. Ok maybe I missed it somewhere, but who decides how fast the link has ot be between Level 3 and Verizon?

    You’re insinuating that no one does, that all Verizon has to do is link up more ports in between your two routers, but I don’t think it would be that simple.

    Who chose the speed that it’s currently at, and why?

    • Wendy, the speed, or bandwidth, changes very regularly. Interconnect agreements contain language that obliges both parties to keep those links uncongested. We try and keep the utilization to 70% or lower. So its the growth of internet traffic at any particular location, driven by all of us as consumers of the Internet, that determines the speed of the interconnect at any moment in time.

  65. This is absolutely fantastic! I’m an L3 customer and I had no idea all I had to do was ask for bandwidth!

    Let’s see… does my network have capacity? Check.
    Does Level 3’s network have capacity? Check.

    Awesome! Who do I speak to about getting my free circuits for the cost of the equipment? My client’s customers are suffering, so if you could step on it I’d appreciate it. Bravo guys, seriously.

    • The interconnects are set up to only exchange traffic between the customers of each network. The interconnect should be set up such that traffic flows on each side of that interconnect travel about the same distance. In that case both networks have an equitable relationship; they both have paying customers that want to communicate with each other, they both share the burden of noving the traffic. No one is asking for free bandwidth or a free ride.

  66. John, is Level3’s preferred policy, distilled, that you want settlement free peering with residential/retail ISPs like Verizon? If so, that would be a significant development for smaller ISPs, wouldn’t it? What do you tell them when they come to level3 and say that they would like that arrangement, too?

  67. This makes one heck of an argument for unplugging from the internet altogether. Reading about the corporate greed all around takes the joy of the internet away. The internet used to be a place for information sharing and collaboration, even some anonymity. Now, it’s a place simply to extract money and sell people’s private information or warehouse it for sale. Reading the nonsense above makes me only so happy to unplug completely. Americans should get outdoors and be more active anyhow. This might just be my impetus to do so.

  68. I am guessing these links are for all traffic to/from AS3356? Thus actually this will be impacting Verizon customers trying to access other Level 3 customer sites (and visa versa).

    As a smallish but heavily peered provider I would horrified if any of our edge connections were congested.

    I do wonder why Verizon wouldn’t sort the peering capacity out and then if they so wanted restrict the Netflix traffic elsewhere, this it wouldn’t impact other services for them.

    We are peered directly with Netflix in the UK, so I also wonder why Verizon don’t just have direct PI with Netflix, that would give them an easier opportunity to shape if needs be.

    Anyway my point is that deliberately causing an interconnect with another network to be overloaded is IMHO extremely bad practice and cutting your nose off to spite your face.

  69. Pingback: Level 3: streaming slowdowns are all Verizon's fault - Computer Business Review - Computer Repair Review Orange County California

  70. It seems painfully obvious Verizon (and not other carriers) is leveraging it’s best-in-nation cellular coverage into so-called “triple play” packages. Customers are then unwilling or unable to switch providers because they cannot afford Verizon arbitrarily charging extra for internet service sans cable.

    Verizon is hoping customers will excuse their bad behavior in the market in exchange for excellent cellular coverage.

  71. Pingback: Verizon caught throttling Netflix traffic even after its pays for more bandwidth | On-line World News

  72. Pingback: Frustrated FiOS Subscriber’s Video Exposes Severity of Verizon’s Netflix Throttling | Rizenet² -UNDER CONSTRUCTION

  73. Pingback: Verizon promises to improve upload speeds as it pretends Netflix and YouTube don’t exist | PandoDaily

  74. 1) A nice summary of the debate over historic peering-ratio traditions can be found here.

    2) The fundamental problem is a lack of competition. Verizon having poor connectivity to Netflix’s ISP wouldn’t be an issue if a customer who liked having high-quality Netflix service could switch to an alternate broadband provider. This is beautifully illustrated in this video which goes on to make the case for ISPs to be treated as common carriers. Of course given the ongoing lack of leadership by the FCC this won’t likely happen…a rant by a famous comedian notwithstanding.

    In my view, it is precisely this lack of last-mile Internet competition that makes Network Neutrality a necessary band-aid. And one need look no further than a ranking of country by download speed to realize just how poor the competitive situation is in America. There has been a vacuum of regulatory leadership by the FCC in fostering competition since the (I would argue failed) 1996 Telecom Act…though surely this situation is itself a byproduct of the toxic intersection of lobbying dollars and politics in the US.

    The consequence of limited peering is poor network performance: besides congestion it also causes traffic to take less direct routes. I would point to my first-hand knowledge of the traffic patterns of a large residential-targeted streaming service as an example of the impact on US users: roughly twice the percentage of customers in Europe are directly connected vs. in the US. For latency-sensitive applications like multiplayer games and telepresence such indirect routing can have a huge impact on quality.

    3) Setting aside the politics of peering, Netflix ought to prefer to purchase some of its IP transit directly from Verizon. After all, Level 3 is merely a middleman for the traffic destined for Verizon and Verizon thus ought to be able to undercut Level 3 on price (in fact, I suspect such basic economics are the explanation for Netflix’s recent deal with Comcast). Unfortunately, based on past negotiations with the largest residential network providers in the US (Comcast, AT&T, Verizon), I can confirm there is widespread denial that wholesale IP transit pricing has been dramatically falling for years. Surely it is these companies’ monopoly (or duopoly) position that emboldens them to ask more than the market rate.

  75. Pingback: ViewQwest Fiber Broadband Official Thread - Part 3 - Page 31 -

  76. -ironic mode on-
    Clearly is Netflix fault for pushing that imbalanced amount of traffic. The poor Verizon users have nothing to do with it. They didn’t requested that traffic, so Verizon has no obligations here. Netflix is sending unsolicited imbalanced traffic to verizon users, forced to see videos, so totally ok to request more $$$
    -ironic mode off-

    Hint for netflix: start sending upstream traffic from your clients filled with garbage to balance the up/down traffic, and then they will improve the connectivity! 🙂

    The internet companies are selling a service to their subscribers. If their subscribers want to see netflix, it’s verizon the one that should ensure there is enough BW for all of them. Same if they want just to visit wikipedia. And they should be very happy because the clients will buy high speed internet, and level3 and other netflix providers have many interconnection points to ensure they don’t need to route the massive traffic over expensive cross-country fiber lines. But the issue here is that they are greedy, they are a oligopoly and extra bonus: they have their own video offerings.

  77. Pingback: Verizon caught throttling Netflix traffic even after its pays for more bandwidth

  78. Thank goodness I live in a town without an ISP monopoly. Our two cable providers are listed in the top 10 in Netflix’s ISP speed rankings. When ISPs have to compete, they usually are forced to provide better service for lower rates.

  79. Pingback: Technophilia 123: This Man Loves Rabbits » The MakeUseOf Technophilia Podcast

  80. I’m somewhat skeptical of the idea of “bit-miles”. In particular, if the Netflix data were being served from a Level 3 CDN, that would mean it will have effectively 0 miles on your side of the equation. You certainly can’t pretend it traveled 800 miles through your network when it really came from a server in a cage across the room.

    • Rob, you are exactly right. And in this case we would not be able to “claim” any bit miles. Level 3 has chosen to invest in this sort of technology. We are effectively passing on the benefit to Verizon because we have to do more work elsewhere in order for us to fully share the burden of carrying all the traffic between our paying customers and Verizon’s paying customers.

  81. Is Verizon doing this in the Baltimore / Washington, D.C. area as well? I live there and regularly interact with remote machines at my employer in North Carolina. I have the Verizon FIOS 25/15 plan. During normal business hours, it works fine. But starting right around 6 pm every night, round trip times go to crap. The route often looks like this:

    3 15 ms 13 ms 16 ms []
    4 14 ms 12 ms 12 ms []
    5 * * * Request timed out.
    6 * * * Request timed out.
    7 27 ms 19 ms 19 ms 0.ae2.BR2.IAD8.ALTER.NET []
    8 111 ms 101 ms 101 ms []
    9 107 ms 101 ms 101 ms []

    The ALTER.NET (a.k.a. Verizon) to level3 hop always introduces the latency. Thanks for any thoughts you can provide.

    • Yes. Washington is a bigger interconnect point but also needs additional ports activated between our two networks.

  82. Creating an artificially exorbitant problem here gives Verizon pull in the neutrality debate. We all know that’s what this is about. If Verizon, or anyone else, can create a powerful enough argument for the need for “fast lanes” the FCC could likely follow the mob mind and give companies what they need to legally establish the pricing tiers they really want. I believe that kind of regulation can only hurt the internet using population and any plans to expand that base into lower income areas and third world countries. High bandwidth users, read $$$, won’t be there, so why roll out? As much as I like some of the things we have in our culture, I’m really getting tired of capitalism and the greed behind it. I live in Texas and get my connection from a co-op leasing lines from ATT. I pay $130/mo for 700Kb/s|2Mb/s and if Verizon wins here, I won’t be able to pay any more than I do already.

  83. Pingback: Verizon Plays DirtyFree Internet Tutorials | Free Internet Tutorials

  84. Pingback: thEndUsr Episode 178: Pants Based Personal Flying Transportation System | The Digital Media Zone

  85. Pingback: News - Internet Slowdown Day will see top websites join in protest

  86. Pingback: Why Free Marketeers Want To Regulate the Internet » De Civitate

  87. Pingback: Guard Frequency » Guard Frequency Episode 031 | Big Mamajama

  88. Thank you so much Level 3 for putting Verizon in there place. It’s so sad to see someone like Verizon who is a last mile ISP do this to their customers, one shouldn’t have to go out and buy a VPN just so they can watch 1080p video. So thank you for standing up for us and giving cold hard facts.

  89. Pingback: Year in Review | Top 10 Blogs of 2014 - Beyond Bandwidth

  90. Pingback: FCC Net Neutrality Proposal is good for you and the country

  91. This all boils down to who is going to pay for the excessive bandwidth usage. Why should Comcast/Verizon have to raise rates on ALL of their users to pay for the additional bandwidth servers necessary for the users that use Netflix/level3. The burden of cost SHOULD be on Netflix/Level3 as they are using a disproportionate amount of bandwidth. Netflix/Level3 do not want to raise rates on their customers and are trying to push that cost / burden off to the various ISPs. Calls of throttling and wrapping this in the banner of Net Nuetrality are at best misleading and at worst dishonest.

    Netflix/Level3 want to keep their costs down. It’s sound business (for them), we get it, but let’s stop with the throttling snd Net Nuetrality BS.

    • Valdarez, I disagree.

      Firstly, this blog post was written to show that there is plenty of bandwidth available. Both Level 3 and Verizon have the bandwidth. Both have confirmed that fact. It is already there waiting to carry traffic. There is simply a bottleneck at the interconnection points between the two networks. A bottleneck caused solely by one of the parties. As my diagram shows we have interconnection ports ready and waiting for ports to be made available by the other network.

      Secondly, unit costs for the provision of bandwidth have fallen consistently and dramatically. And in fact faster than the increases in consumed bandwidth. And because of this conveniently overlooked fact the profitability of most broadband ISPs has been increasing. There is no burden. In fact the opposite is true.

      Thirdly, traffic exchanged through the congested interconnection points I mentioned above has to be carried to and from that point on both sides of that interconnection. Level 3 carries packets across its network as does Verizon. When we add up the total traffic and the total distance we each carry it, it turns out that “burden” is an equally shared one.

      Fourthly, ISPs sell their broadband services to consumers on the basis of speed and access to a hugely dynamic variety of content. The ISP’s customers then make use of the service they pay for by requesting that content be delivered to them. That content is unable to be delivered because that same ISP is throttling access to its network. I would suggest that your comments about the misleading nature of this problem lie here.

  92. I Maybe off key on this subject.
    From what I gather, this all but nothing but a blame game on a breach on settlement agreements between the two. Who is going pay the difference? The consumer will end up paying. I’m installer on the last mile. Over the years I have seen speed increase’s on the consumer end (nowhere near the european model but we do have the capability) along with price. On some of the trouble tickets and upgrades I go to is somewhat stupid..I don’t blame the consumer for the lack of understanding of what is happening behind there devices that are connected to their isp. I have had a share of customer complaining about netflix is buffering but they can watch youtube without a problem and I am paying for hi speed internet but my connection drops out when I check email from work.( the company they work for headquarter is oversea)… The ISP’s has lead the consumers to believe faster will be better for everyone therefore cost will increase along with speed (with data caps in place).This is a bogus brainwashing.. A lot of the people are reaching these data cap faster then ever before with the shift of entertainment transferring over to internet…To me this is a unfair advantage for all the consumer and the backbone ISP’s. The only advantage that I see is the last mile provider’s making which the internet was not really intended to be for profits but for information to exchange among it peers. Yes there will be cost involved in expansion for network congestion to be relieved as the demands go up..I think it is time for the cost to be neutral among all the parties of internet..not in asymmetrical way where its unbalance but in symmetrical way where it balanced across the board…..just my thought.

  93. Pingback: Your ISP is Slowing you Down?

  94. Pingback: An Interesting Perspective from an Insider… | Neal Head

Leave a Reply

Your email address will not be published. Required fields are marked *

Time limit is exhausted. Please reload the CAPTCHA.