13 Nov

Understanding the game of bandwidth pricing

I thought about this long back – “Who pays to whom in case of internet bandwidth?” I have been working in this domain from sometime and so far I have learnt that it’s really complex. I will try to put a series of blog post to give some thoughts on this subject.
Firstly we have to understand that when we talk about “bandwidth price” it’s often layer 3 bandwidth which you buy in form of capacity over ethernet GigE, Ten-GigE and so on (or STMs if you are in India). As we know from back school class in networking – layer 3 works over layer 2 and so to deliver “bandwidth” on layer 3, one needs layer 2 physical circuit. Price paid by companies on layer 2 Vs layer 3 varies significantly based on their location, type of business, their target goal etc. E.g a content heavy company like Google pays hell lot of money on layer 2 circuits while it is strongly believed among networking community that Google is a tier 1 network and hence a “transit free” zone and they do not pay any amount on layer 3. In general the trend is pretty much as big networks have larger network footprint and connected “PoPs” over layer 2 (leading to a higher layer 2 bill) while relatively lower layer 3 bill while small networks depend significantly just on transit bandwidth (in form of layer3) and have very low layer 2 footprint.
What’s more interesting here is the trend on “Who pays for the layer 2?”¬†
Well it has relatively less clear trend and entire setup is based on fact that internet started in US and a significant content was hosted in US for a long time. Now overall that’s changing fast due to ways web is evolving but still if you consider traffic is flowing to/from Asia, you will often find that it’s on Asian telco’s layer 2 and not the Western telcos. E.g traffic from New York to Mumbai say between AT&T in New York to Tata Comm in Mumbai will be handed over to Tata’s PoP in New York and they will carry it all way down to Mumbai over their layer 2. While for packets travelling from Mumbai to New York, they will be carried over Tata’s layer 2 (yet again!) till New York and will be handed off to AT&T in New York. Now considering fact that Tata and AT&T both are Tier 1 settlement free ISPs, why it’s that only Tata pays? Same applies on relation between say NTT & Level3. And same even applies for traffic between small regional networks which eventually go via big tier 2 networks like say traffic flow between Level3 and Airtel etc (Airtel pays for layer 2).
Here’s list of reasons I can think off and have found so far in my numerous discussions with people around in industry:

  1. It has lot to do with sense of power. US is believed to be “Center of backbone” and over time that has changed a lot but still image stays like that.
  2. Except Singapore & Hong Kong most of other major regions in Asia do not much of content & infact consume significant content from Western world. E.g in case of India 70% of content still comes from outside (but don’t assume it to be 70% bandwidth since 30% of domestic hosted content has lot to do with CDN’s and way more % of traffic out on them).
  3. Most of countries in region are very restrictive in letting foreign player to get there and work (India, China, etc).
  4. There is no major exchange point acting as center of regional/global traffic exchange as like that of DE-CIX in Frankfurt or world famous 60 Hudson Street in New York.
  5. Somehow Asian businesses like this to continue and they have no issues in paying at layer 2 as long as they can keep their rivals away from domestic market.
  6. Western players like AT&T, Verizon, Century Link, Level3, Telia etc still stay in more dominant position as compared to Asian majors.

Will share more on this subject in upcoming blog posts.
Sidenote: Visited Amsterdam Botanic Garden today and ending this post with some of pics from there. ūüôā

Apart from that I also saw snowfall at Klagenfurt, Austria which was very nice experience. Here’s how it looked like:

16 May

Backend of Google's Public DNS

And finally academic session over. Done with all vivas and related stuff.
Next will be exams likely in June. Time for me to get ready for travel. ūüôā
Anyways an interesting topic for today’s post – Google Public DNS. Lot of us are familier with popular (and free) DNS resolvers and I have covered reason in previous posts on why it tends to fail with Content Delivery networks like Akamai which rely on anycasting at bottom DNS layer and simple unicasting on application servers. Anycasted DNS nodes point to application servers based on various factors like distance, load, cost etc out of interesting algorithms these CDN networks use for load & cost management.
Anyways today’s post focus is not CDN issues with these resolvers but Google Public DNS itself. Are these servers located in India and everywhere else where Google has PoPs?
Let’s do a simple trace to get forward path from Airtel to Google’s

Type escape sequence to abort.
Tracing the route to google-public-dns-a.google.com (
1 [MPLS: Label 550027 Exp 0] 0 msec [MPLS: Label 550027 Exp 0] 4 msec [MPLS: Label 354133 Exp 0] 0 msec
2 0 msec 0 msec 0 msec
3 44 msec 44 msec 48 msec
4 [AS 15169] 52 msec 56 msec 52 msec
5 google-public-dns-a.google.com ( [AS 15169] 52 msec * 116 msec

50ms latency. Clearly destination is within India and based on my experience with latency values, I strongly guess that’s Chennai.

Location of Google Public DNS servers

Anyways so does that means Google’s DNS server is within India?
A clear answer is no. This is just a DNS caching server and Google does not use it for originating actual queries further to root, TLDs nodes and authoritative DNS servers. This seems like a interesting distributed setup.
As per Google Public DNS FAQ page, there are quite a few locations from where DNS servers originate queries but India is not in the list yet. Google has PoPs in Delhi, Mumbai and Chennai and they peer with pretty much every Indian ISP out from there.
We can actually test which node is serving us here in India.
This can be achieved in multiple ways:

  1. Running a authoritative zone on a server with basic BIND installation. I tried this with my own Linux server by having testing-google-dns.anuragbhatia.com. DNS zone. I delegated NS for this zone on auth. DNS servers for “anuragbhatia.com” zone. Next I sent a DNS query with dig @¬†¬†testing-google-dns.anuragbhatia.com. a +short to ask my DNS server for IP and this gave me source IP of Google’s resolver. 
  2. The other easy way out is to simply use Akamai’s “whoami.akamai.net” service. It is designed in a way to return A record of DNS resolver which queries it. This gives IP of Google’s server which sent the DNS query for resolution.


Anurags-MacBook-Pro:~ anurag$ dig whoami.akamai.net a @ +short
Anurags-MacBook-Pro:~ anurag$

In both cases I saw IP was¬† It belongs to¬† announced by Google’s AS15169. As per Google’s FAQ page (which has IPs too!) the prefix¬† belongs to¬†Kuala Lumpur, Malaysia. So that’s the actual DNS resolver node which serves users here in India. Machines with IPs and are just caching replies and more over taking the IP traffic to Google within India.
Now one can ask why Google is not having DNS resolver within India?


Guess work time!

I don’t know exactly but I can do a strong guess work here. Google is a tier 1 transit free network. It relies on paying on layer 2, building PoPs and connected them together. It does not pays on layer 3 for bandwidth to any ISP. So Google’s routers in India learn traffic from just peering sessions with all major telcos (except BSNL). Google is peering with Tata-VSNL AS4755, Reliance AS18101, Airtel AS9498, MTNL AS17813, Spectranet AS10029 etc. One interesting thing here is that these are all tier 2 networks. Tata Communications is a tier 1 network but their domestic backbone VSNL AS4755 is technically not a tier 1 network and technically it sits in downstream of Tata AS6453 (which is their tier 1 IP backbone). Thus Google does not gets full global table feed from any of these links and possibly nearest PoP of Google which has full table feed from Tier 1 networks is in Malaysia.
What I am not able to answer from my guess work is that when Google is relying on East Asian PoPs for such stuff and mantaining a backbone between East Asia and India directly then why they could not feed Indian routers routing table with routes learnt from outside? ¬†It could be just to ensure direct delivery in India and avoid routing loops. E.g BSNL has IP port from Tata-VSNL AS4755 within India and IPLC port from Tata-AS6453 to outside Indian PoPs. Thus if tables are combined Google might see paths like ¬†AS6453 > AS9829 and AS4755 > AS9829 which seem identical as per AS path but one is direct India to India traffic while other via India > Singapore > India or India > US > India. It’s not just about BSNL but Sify also lately has weird routing loops going from outside India for Indian destinations.
That’s about it. Can’t do any guess work beyond this point unless someone gives me access to a router of AS15169 to see table! ūüôā