Level3

Lately, I have been struggling to keep latency in check between my servers in India and Europe. Since Nov 2021 multiple submarine cables are down impacting significant capacity between Europe & India. The impact was largely on Airtel earlier but also happened on Tata Comm for a short duration. As of now Airtel is still routing traffic from Europe > India towards downstream networks via the Pacific route via EU > US East > US West > Singapore path. Anyways, this blog post is not about the submarine cable issue.

I thought about this long back - “Who pays to whom in case of internet bandwidth?” I have been working in this domain from sometime and so far I have learnt that it’s really complex. I will try to put a series of blog post to give some thoughts on this subject. Firstly we have to understand that when we talk about “bandwidth price” it’s often layer 3 bandwidth which you buy in form of capacity over ethernet GigE, Ten-GigE and so on (or STMs if you are in India). As we know from back school class in networking - layer 3 works over layer 2 and so to deliver “bandwidth” on layer 3, one needs layer 2 physical circuit. Price paid by companies on layer 2 Vs layer 3 varies significantly based on their location, type of business, their target goal etc. E.g a content heavy company like Google pays hell lot of money on layer 2 circuits while it is strongly believed among networking community that Google is a tier 1 network and hence a “transit free” zone and they do not pay any amount on layer 3. In general the trend is pretty much as big networks have larger network footprint and connected “PoPs” over layer 2 (leading to a higher layer 2 bill) while relatively lower layer 3 bill while small networks depend significantly just on transit bandwidth (in form of layer3) and have very low layer 2 footprint.  

This Sunday I was looking at global routing table dump and found AS1 announcing some very weird prefixes.

AS1 i.e Autonomous System Number 1 belongs to Level3 but as far as I know they are not actively using it. They use AS3356 globally (along with Global Crossing’s AS3549). I noticed quite a few prefixes of a Brazil based telecom provider - Netvip Telecomunicaes being announced by AS1. 

Some of entries in global routing table belonging to AS1 (as picked from BGP table dump of route-views archive):

Quick analysis of BSNL-Level3 bad routing issue

I can see BSNL having pretty high latency again with most of Europe again. It seems like they are using Level3 Communications AS 3356 along with Tata-VSNL for upstream. With Level3 transit BSNL has badly screwed up reverse path causing very high latency and awful bandwidth.

anurag@laptop:~$ ping server7 -c 5
PING server7.anuragbhatia.com (178.238.225.247) 56(84) bytes of data.
64 bytes from server7.anuragbhatia.com (178.238.225.247): icmp_req=1 ttl=52 time=320 ms
64 bytes from server7.anuragbhatia.com (178.238.225.247): icmp_req=2 ttl=52 time=320 ms
64 bytes from server7.anuragbhatia.com (178.238.225.247): icmp_req=3 ttl=52 time=319 ms
64 bytes from server7.anuragbhatia.com (178.238.225.247): icmp_req=4 ttl=52 time=327 ms
64 bytes from server7.anuragbhatia.com (178.238.225.247): icmp_req=5 ttl=52 time=320 ms
--- server7.anuragbhatia.com ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4004ms
rtt min/avg/max/mdev = 319.880/**321.765**/327.384/2.828 ms
anurag@laptop:~$

Expected latency values here should be around 150ms. A packet should not take more then 150ms round trip between Radaur, Haryana to Munich located server.