OTT and paid peering

Isp-Column

Yesterday there was an article in the Indian paper Financial Express with the title “OTTs may have to pay access charge to telcos”.

Quoting a few points from the article:

  • Social media intermediaries like WhatsApp, Facebook and Twitter, and over-the-top (OTT) players like Netflix, Prime Video and Disney+Hotstar may have to pay a carriage charge to telecom service providers
  • Data, particularly video, comprises 70% of the overall traffic flow on telecom networks, and this would grow further with the rollout of 5G services
  • Upon reference from the DoT, Trai is currently studying various possible models under which OTTs can be brought within the purview of some form of regulation
  • According to sources, an interconnect regime is a must between OTTs and telcos because as 5G services grow, there would be immense data/ video load on networks, which may lead to them getting clogged or even crashing at times.

This concept of “OTTs must pay” is not new. This has been argued a few times in past. Exactly ten years ago in 2012 I wrote a blog post about Bharti Airtel expecting Google/YouTube to pay. At that time they could not convince OTTs to pay. Why is this renewed interest now? Well, that has to do with the first SK Telecom (South Kore telecom) Vs Netflix court case in South Korea where SK Telecom claimed that a large part of bandwidth utilization was because of Netflix and hence they should pay a “fair share” of their traffic which they lost. Soon around this multiple of large telecom monopolies in Europe started this discussion in their respective geography. Four of the top EU players - Deutsche Telekom, Orange, Vodafone and Telefonica are of opinion that OTTs should share the burden (news here). And hence Indian telcos possibly looking to renew this debate.



Mis-representation of peering & caching

It’s hard to know the exact arguments in that court case but it seems like it was less about OTT Vs large eyeball ISP and more about “domestic South Korean ISP Vs the mighty American content player”. Technically many of these demands are wrong and often the telco view hides many important technical facts which people outside of our industry may not realise.
When telcos claim OTT is using over 70% of their capacity, where are all those massive bits flowing to?
(Hint: their “paying” customers). When telcos claim that OTT must bear the cost of infra - aren’t they doing that already by building a mix of network PoPs for peering and caching CDN servers which are placed “inside the” ISPs network PoP to push bits to the end users?



Understanding present state of caching & peering

Let’s take a step back and understand how content is being delivered presently from a company like say Google to users in India.

  • The actual content for Google is stored in their datacenter. There are 23 datacenters for Google (source here). The nearest one to India is Singapore and Taiwan.
  • Bits are transported from these datacenters to Google’s network PoPs in India which are established inside well-connected datacenters. For Google that’s 8 datacenters - Airtel Santhome (Chennai), Equinix GPX MX1, NTT Netmagic Chennai, Sify Greenfort Noida, STT/Tata Comm in Delhi, Mumbai & Chennai (source: AS15169 peeringdb)
  • Origin datacenters are connected to these network PoPs over very high capacity (multi-terabit backbone) over n x 100G or n x 400G optical wavelengths
  • Next, Google does PNI - Private Network Interconnect for networks with a “reasonable” amount of traffic (which is typically levels beyond 3-4Gbps and the number updates over time). In this model, Google would order a “cross-connect” between their rack and an eyeball ISPs rack to connect at 10G or more of capacity.
  • For smaller ISPs with lower traffic, they push the traffic via internet exchanges like Extreme IX, NIXI, DE-CIX and AMS-IX Sify in Mumbai, Chennai, Delhi etc.
  • A large part of YouTube traffic happens to be popular videos which are cached on Google GGC nodes. These are thousands of servers placed inside a large number of ISPs. The “inside” is important. Thus all the YouTube or Facebook videos, popular WhatsApp videos, and Netflix shows - are cached and are sitting right inside the ISP’s network PoP.
  • ISP is responsible for maintaining these PoPs, connecting them to their distribution network and ultimately handling last-mile delivery of bits to their paying customers

Google (and other OTTs) are effectively paying various large telcos for layer 1 circuits to pump that sort of high capacity within Google’s own PoPs. That adds CAPEX in terms of submarine cable buildouts and Opex as part of leased capacity on layer 1 on these OTTs. Further, there is massive Capex to deploying caching servers & Opex to maintain them (send replacement drive, other hardware etc). Thus when telcos argue that OTT is not paying for the bandwidth - it is simply incorrect and a misrepresentation of facts. What instead they are asking for is that OTTs pay for “paid peering” or simply the middle mile & last mile cost of the network.

Another misleading part here is about where the traffic is going. All this traffic ultimately is going to the paying users of these networks who are bearing the cost. If end users sitting on these networks stop accessing OTT, traffic will drop to zero. So instead of “fair share” this is a question about “getting paid on one leg” Vs “getting paid on both legs”.



Isn’t paid peering already a thing?

Actually yes. There is quite a bit of paid peering between OTTs and eyeball networks. Typically peering between OTTs and small to mid-sized ISPs is very often settlement free i.e no one pays each other. Peering between OTTs and large telcos depends. Some part of that is settlement-free e.g Google AS15169 in India on their peering with Airtel, Jio, BSNL etc while some pay e.g Amazon AWS typically pays to the same set of networks in India. While some networks have a different relationship with Google, for instance, Tata Comm where Google is a customer on many of their cable systems & as a part that has a transit agreement with AS6453 to reach a small % of the non-peered world for them.

An interesting talk about this topic in the EU context was recently given by Rudolf van der Berg at NLNOG embedded in this post below (direct YouTube link here)



Who pays whom when you access this blog?

So this blog has videos, pictures and text. Not a traditional OTT but say hypothetically if I start posting my video explaining this, in that case, this can vaguely be an OTT. Who pays to whom in that case?

  • Blog itself is hosted by Google Cloud firebase hosting & hence actual files reside inside Google’s datacenter in Singapore
  • In the front end, firebase (ironically!) uses Fastly CDN instead of Google’s CDN (probably migration never completed since firebase purchase in 2014) and hence you are hitting Fastly CDN nodes which are pulling data from Google’s origin in Singapore.
  • Most static images of blog posts are hosted on Google Cloud object storage in Delhi
  • Heavy static content like screen share videos are hosted on Backblaze B2 in Amsterdam. This is the type of content where I wish to save on egress money instead of paying Google (or any other top cloud provider for that matter) a premium.

So this involves Google, Fastly and Backblaze interconnects & their agreement when accessing this. So who ultimately pays whom?

  1. Google has settlement-free peering with almost all Indian networks (while keeping the context that they are an investor in Jio (reference), an investor in Airtel (reference), a major customer of Tata Comm outside India etc.
  2. Fastly in this case has a mix of peering & paid transits to the best of my knowledge. They are paying large telcos while maintaining settlement-free peering with a large number of mid to small-sized ISPs.
  3. Backblaze Amsterdam on the other hand has no peering, it has IP transit from Telia, GTT and Cogent. None of those networks peers with any of the Indian networks for settlement (except Tata Comm) and hence Airtel/Jio/VI etc do pay to access the content hosted on it.

Ultimately it’s the end user which is always paying the ISP. In some cases (like Backblaze), the other network is also paying and there are good reasons for it. Backblaze excels in ultra-cheap, high-density storage (read about Backblaze pods here). For now, they do not have the requirement of building their network in Europe which will enable them to peer with various networks (including Indian telcos sitting on those European exchanges). So they have traded off Capex in building the network, Opex in maintaining it and having a team of network engineers against a trade-off of slightly expensive paid IP transit with Telia, GTT and Cogent. Though I would have loved if Backblaze had a non-transit free, open-to-peer ISP in Europe as their transit (e.g they have Unwired LLC in the US for their California datacenter and that openly peers locally).



Want to understand more about peering?

Check out Internet Peering Playbook.

Additionally, check out these two podcast shows on APNIC PING on a discussion between my friends Robbie & Geoff Huston. Original source is here.

Korea Vs Content Provider


Be careful what you wish for


Disclaimer:
This is my personal blog and hence posts made here are in my personal capacity.
These do not represent the views of my employer.