As4755

Post to determine whether or not NIXI carries full Indian routing table?

After my last post about home networking, I am jumping back into global routing. More specifically how Indian traffic is hitting the globe when it does not need to. This is an old discussion across senior management folks in telcos, policymakers, and more. It’s about “Does Indian internet traffic routes from outside of India?” and if the answer is yes then “Why?” and “How much?”

It became a hot topic, especially after the Snowden leaks. There was even an advisory back in 2018 from Deputy National Security Advisor to ensure Indian internet traffic stays local (news here). Over time this has come up a few dozen times in my discussion with senior members from the Indian ISP community, individuals, and even latency-sensitive gamers. So I am going to document some of that part here. I am going to put whatever can be verified publically and going to avoid putting any private discussions I had with friends in these respective networks. The data specially traceroutes will have measurement IDs from RIPE Atlas so they can be independently verified by other network engineers.

Last month I noticed quite high latency with Cisco’s OpenDNS from my home fibre connection. The provider at home is IAXN (AS134316) which is peering with content folks in Delhi besides transit from Airtel.

ping -c 5 208.67.222.222
PING 208.67.222.222 (208.67.222.222) 56(84) bytes of data.
64 bytes from 208.67.222.222: icmp_seq=1 ttl=51 time=103 ms
64 bytes from 208.67.222.222: icmp_seq=2 ttl=51 time=103 ms
64 bytes from 208.67.222.222: icmp_seq=3 ttl=51 time=103 ms
64 bytes from 208.67.222.222: icmp_seq=4 ttl=51 time=103 ms
64 bytes from 208.67.222.222: icmp_seq=5 ttl=51 time=103 ms
--- 208.67.222.222 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4005ms
rtt min/avg/max/mdev = 103.377/103.593/103.992/0.418 ms

This is bit on the higher side as from Haryana to Mumbai (OpenDNS locations list here). My ISP is backhauling from Faridabad which is probably 6-8ms away from my city and 2-3ms further to Delhi and from there to Mumbai around 30ms. Thus latency should be around ~40-45ms.

And here goes first blog post of 2018. Last few months went busy with some major changes in personal life. :) I looked into Amazon’s India connectivity with various ASNs tonight. Here’s how it looks like. (Note: Jump to bottom most to skip traces and look at the summary data).  

 

Traceroutes

Amazon India to Vodafone India

traceroute to 118.185.107.1 (118.185.107.1), 30 hops max, 60 byte packets
 1 ec2-52-66-0-128.ap-south-1.compute.amazonaws.com (52.66.0.128) 21.861 ms ec2-52-66-0-134.ap-south-1.compute.amazonaws.com (52.66.0.134) 19.244 ms 19.233 ms
 2 100.64.2.200 (100.64.2.200) 14.789 ms 100.64.0.200 (100.64.0.200) 20.731 ms 100.64.3.12 (100.64.3.12) 13.187 ms
 3 100.64.0.193 (100.64.0.193) 14.418 ms 100.64.3.69 (100.64.3.69) 15.469 ms 100.64.3.67 (100.64.3.67) 15.946 ms
 4 100.64.16.67 (100.64.16.67) 0.343 ms 100.64.17.165 (100.64.17.165) 0.312 ms 100.64.17.199 (100.64.17.199) 0.313 ms
 5 52.95.67.213 (52.95.67.213) 1.942 ms 52.95.67.209 (52.95.67.209) 1.967 ms 52.95.67.213 (52.95.67.213) 1.935 ms
 6 52.95.66.218 (52.95.66.218) 4.998 ms 4.694 ms 52.95.66.130 (52.95.66.130) 4.650 ms
 7 52.95.66.67 (52.95.66.67) 1.752 ms 52.95.66.89 (52.95.66.89) 1.850 ms 1.806 ms
 **8 52.95.217.183 (52.95.217.183) 3.111 ms 3.102 ms 3.088 ms <- Amazon India**
 **9 182.19.106.204 (182.19.106.204) 3.426 ms 4.547 ms 4.537 ms <- Vodafone India**
10 118.185.107.1 (118.185.107.1) 2.035 ms 2.059 ms 2.039 ms

 

On weekend  I was looking at BGP Instability Report data. As usual (and unfortunately) BSNL tops that list. BSNL is the most unstable autonomous network in the world. In past, I have written previously about how AS9829 is the rotten IP backbone.

This isn’t a surprise since they keep on coming on top but I think it’s well worth a check on what exactly is causing that. So I looked into BGP tables updates published on Oregon route-views from 21st May to 27th May and pulled data specifically for AS9829. I see zero withdrawals which are very interesting. I thought there would be a lot of announcements & withdrawals as they switch transits to balance traffic. If I plot the data, I get following chart of withdrawals against timestamp. This consists of summarised view of every 15mins and taken from 653 routing update dumps. It seems not feasible to graph data for 653 dumps, so I picked top 300.

So it seems like apart from voice interconnect issues, Jio is also facing routing issues on the backbone. I ran a trace to one of IP’s on Jio network allocated to end customer - 169.149.212.122. I ran trace from all Indian RIPE Atlas probes measurement here. There seem quite a few RIPE Atlas probes which are giving latency on 150ms + range. Seems like they are downstream or downstream of downstream of Tata Comm’s AS4755 and routing is happening via Singapore!    

It’s great to see Amazon announcement two days back about launch of their region in Mumbai. In past I was quite happy to see their Cloudfront CDN PoPs in Mumbai & Chennai (blog post here). Now it’s just great to see a full AWS region out of Mumbai. :) Though it’s going to eat most of important customers from the smaller players still it’s good for industry as industry is too big and we need more & more of such large Cloud players in India to bring more and more content hosting in India.

Today I spotted some routes from Amazon AWS Cloud services -  AS16509 in Indian tables. AS16509 was originating prefixes while sitting in downstream of Tata-VSNL AS4755 and Reliance AS18101. I almost missed Amazon AWS's announcement on their blog about Indian PoPs for their DNS service - Route53 and CDN service - Cloudfront.

New PoP’s of Amazon in India are at Mumbai and Chennai and I see pretty much consistent BGP announcements to Tata and Reliance from these locations. Prefixes I have seen so far:

And finally academic session over. Done with all vivas and related stuff. Next will be exams likely in June. Time for me to get ready for travel. :)   Anyways an interesting topic for today’s post - Google Public DNS. Lot of us are familier with popular (and free) DNS resolvers 8.8.8.8 and 8.8.4.4. I have covered reason in previous posts on why it tends to fail with Content Delivery networks like Akamai which rely on anycasting at bottom DNS layer and simple unicasting on application servers. Anycasted DNS nodes point to application servers based on various factors like distance, load, cost etc out of interesting algorithms these CDN networks use for load & cost management.   Anyways today’s post focus is not CDN issues with these resolvers but Google Public DNS itself. Are these servers located in India and everywhere else where Google has PoPs?   Let’s do a simple trace to get forward path from Airtel to Google’s 8.8.8.8:

Just was looking at routing tables of BSNL. They have a significant address space in /10 - 117.192.0.0/10. Overall this /10 address space is divided into /18 and /20 subnets.

Let’s pick two of such subnets and observe routing tables from route-views:

1. 117.192.0.0/18
2. 117.192.0.0/20 

Routing table for 117.192.0.0/18: 

* 117.192.0.0/18 193.0.0.56 0 3333 3356 6453 4755 9829 9829 9829 i  
* 194.85.102.33 0 3277 3216 6453 4755 9829 9829 9829 i  
* 194.85.40.15 0 3267 174 6453 4755 9829 9829 9829 i  
* 129.250.0.11 6 0 2914 6453 6453 4755 9829 9829 9829 i  
* 128.223.253.10 0 3582 3701 3356 6453 4755 9829 9829 9829 i  
* 4.69.184.193 0 0 3356 6453 4755 9829 9829 9829 i  
* 209.124.176.223 0 101 101 3356 6453 4755 9829 9829 9829 i  
* 69.31.111.244 3 0 4436 2914 6453 6453 4755 9829 9829 9829 i  
* 207.46.32.34 0 8075 6453 4755 9829 9829 9829 i  
* 66.59.190.221 0 6539 6453 4755 9829 9829 9829 i  
* 12.0.1.63 0 7018 6453 4755 9829 9829 9829 i  
* 208.74.64.40 0 19214 2828 6453 4755 9829 9829 9829 i  
* 203.181.248.168 0 7660 2516 6453 4755 9829 9829 9829 i  
* 66.185.128.48 111 0 1668 6453 4755 9829 9829 9829 i  
* 134.222.87.1 0 286 6453 4755 9829 9829 9829 i  
* 157.130.10.233 0 701 6453 4755 9829 9829 9829 i  
* 114.31.199.1 0 0 4826 6939 1299 6453 4755 9829 9829 9829 i  
* 89.149.178.10 10 0 3257 6453 4755 9829 9829 9829 i  
* 154.11.98.225 0 0 852 3561 6453 4755 9829 9829 9829 i  
* 202.249.2.86 0 7500 2497 6453 4755 9829 9829 9829 i  
* 154.11.11.113 0 0 852 3561 6453 4755 9829 9829 9829 i  
* 144.228.241.130 0 1239 6453 4755 9829 9829 9829 i  
* 217.75.96.60 0 0 16150 1299 6453 4755 9829 9829 9829 i  
* 207.172.6.20 0 0 6079 3356 6453 4755 9829 9829 9829 i  
* 206.24.210.102 0 3561 6453 4755 9829 9829 9829 i  
* 195.66.232.239 0 5459 6453 4755 9829 9829 9829 i  
* 208.51.134.254 2523 0 3549 6453 4755 9829 9829 9829 i  
* 207.172.6.1 0 0 6079 3356 6453 4755 9829 9829 9829 i  
* 216.218.252.164 0 6939 1299 6453 4755 9829 9829 9829 i  
* 203.62.252.186 0 1221 4637 6453 4755 9829 9829 9829 i  
*> 66.110.0.86 0 6453 4755 9829 9829 9829 i  
* 164.128.32.11 0 3303 6453 4755 9829 9829 9829 i  
* 202.232.0.2 0 2497 6453 4755 9829 9829 9829 i

Routing table for 117.192.0.0/20: