Broken connectivity to F root server in India
It has been an interesting week at village - dry weather, (ultra) dry classes, (boring) external seminars and more of depressing environment but one can always find some hope out of such depressing environment. Overall life here is colourful but one just needs to lookout for colours. :) One interesting case to report today - F root server has quite bad connectivity in India. Last week a friend asked me for traceroutes to all root servers and here’s what I saw when I did traceroute for F root from BSNL connection:
traceroute to f.root-servers.net (192.5.5.241), 30 hops max, 60 byte packets
1 192.168.1.1 (192.168.1.1) [AS8151/AS28513] 1.508 ms 2.103 ms 2.614 ms
2 117.207.48.1 (117.207.48.1) [AS9829] 27.243 ms 29.811 ms 32.483 ms
3 218.248.173.42 (218.248.173.42) [AS9829] 39.861 ms 40.320 ms 40.755 ms
4 218.248.250.142 (218.248.250.142) [AS9829] 90.778 ms 93.919 ms 95.856 ms
5 * * *
6 * * *
7 * * *
8 * * *
How can an ISP like BSNL can have missing route to one of key part of core Internet infrastructure?
Quick look at F root server:
F root server has a anycasting node deployed in Chennai, Tamil Nadu, India. This server is hosted at National Internet Exchange of India (NIXI) Chennai and ISC is responsible for this root server. F root uses 192.5.5.241 across all anycasting instances and in India this block is being announced by Autonomous System Number 24049 which is of ISC. AS24049 does a BGP announcement for 203.119.18.0/24 at NIXI Chennai and this is “supposed to be” taken by all ISPs participating at Chennai IXP. When I requested my friends across India for traceroutes to F root, it was a very interesting result! We found connectivity works from everyone including BSNL too for people living in or near around Chennai. Here’s a traceroute from my friend in Chennai on BSNL to F root:
traceroute to f.root-servers.net (192.5.5.241), 30 hops max, 60 byte packets
1 192.168.1.1 (192.168.1.1) 3.328 ms 3.311 ms 4.616 ms
2 59.92.64.1 (59.92.64.1) 32.489 ms 36.188 ms 37.448 ms
3 218.248.255.2 (218.248.255.2) 45.312 ms 48.171 ms 51.017 ms
4 218.248.250.142 (218.248.250.142) 63.279 ms 66.086 ms 69.025 ms
5 218.100.48.142 (218.100.48.142) 71.112 ms 72.939 ms 76.107 ms
6 192.5.5.241 (192.5.5.241) 79.131 ms 44.760 ms 46.550 ms
Here we can see hop 4 is core network of BSNL while hop 5 is NIXI Chennai and hop 6 is ultimately F root server. Initially I got feeling that this is because of broken IGP implementation for BSNL network since their border gateway routers are holding different routing tables and they are not syncing them properly but one strange thing here - Hop 4 in this traceroute i.e 218.248.250.142 is same as last hop in 1st traceroute (done from BSNL Haryana). How come same router has no route when looked from Haryana while it works for Chennai (and nearby) users! This gives clue that forward path is there but return path has issues. For some reason ISC router in NIXI Chennai is not able to get return path to reply for non-Chennai region users. I raised this concern with Network Operations Center of ISC yesterday along with mailing lists like APNIC & RIPE Atlas. One of my friend who is expert into these issues pointed us to right direction which is NIXI routing policy.
Reading out policy:
" An ISP at any NIXI node must at a minimum announce all its regional routes to the NIXI router at that NIXI location. All ISPs connecting to that NIXI node are entitled to receive these routes using a single BGP session with the NIXI router. This will guarantee the exchange of regional traffic within a NIXI node. This is referred to as forced regional multi-lateral peering under the policy"
So this is what is happening: Each operator is providing ISC with limited regional routes around Chennai area and not to it’s entire network. This was later confirmed by ISC NOC reply. This is sort of awful situation in terms of policy which is breaking India backbones badly and ISPs not able to even reach root servers instances hosted within country. Worst, I have been told that there is 10 day time for fix of this problem else ISC is prepared to pull off plug from Chennai F root node. Since it’s anycasting, once there won’t be any instance inside India injecting routes, traffic will simply start flowing towards other F root server instances in nearby countries. If this happens, it will be surely a sad thing for India since we will loose one out of three four root nameservers. :( With hope that next update on this issue will be positive, time to end this post for now. Feel free to share your comments below. (Incase your are ISP or datacenter in India - I would be happy to discuss this issue with you. You can contact me directly from here)
Updates
ISC Network Operations Center acted very efficiently on this issue. Special thanks to Mr Leo Bicknell from ISC who is taking care of it. In initial phase full routing table was made available to F root server via STPI Internet transit. This fixed the problem (on test basis) and after a week of testing, F root instance in Chennai has been switched off until ISC & NIXI get into a contract over additional bandwidth and costs. For now Indian traffic is being routed to other anycasting instance of F root. Earlier it was California, US instance and now it is Hong Kong instance.
1 router.local (192.168.1.1) [AS8151/AS28513] 2.536 ms 2.799 ms 3.624 ms
2 117.200.48.1 (117.200.48.1) [AS9829] 29.778 ms 32.488 ms 34.677 ms
3 218.248.173.38 (218.248.173.38) [AS9829] 36.966 ms 43.283 ms 44.119 ms
4 59.163.207.81.static.chennai.vsnl.net.in (59.163.207.81) [AS4755] 189.973 ms 192.398 ms 195.118 ms
5 ix-4-2.tcore1.CXR-Chennai.as6453.net (180.87.36.9) [*] 199.965 ms 202.296 ms 203.396 ms
6 if-5-2.tcore1.SVW-Singapore.as6453.net (180.87.12.53) [*] 206.915 ms if-3-3.tcore2.CXR-Chennai.as6453.net (180.87.36.6) [*] 166.267 ms if-5-2.tcore1.SVW-Singapore.as6453.net (180.87.12.53) [*] 160.231 ms
7 if-2-2.tcore2.SVW-Singapore.as6453.net (180.87.12.2) [*] 161.044 ms if-5-2.tcore2.SVW-Singapore.as6453.net (180.87.15.69) [*] 164.876 ms 167.178 ms
8 Vlan1870.icore1.HK2-HongKong.as6453.net (180.87.15.61) [*] 217.994 ms Vlan1779.icore1.HK2-HongKong.as6453.net (180.87.15.38) [*] 216.922 ms Vlan1870.icore1.HK2-HongKong.as6453.net (180.87.15.61) [*] 218.664 ms
9 isc2-FE.hkix.net (202.40.161.200) [AS2687/AS4862/AS9498/AS10026/AS1221] 206.692 ms 208.920 ms isc1-FE.hkix.net (202.40.161.202) [AS2687/AS4862/AS9498/AS10026/AS1221] 211.561 ms
10 f.root-servers.net (192.5.5.241) [AS55440/AS3557/AS23708/AS8167] 247.908 ms 252.645 ms 253.544 ms
I am looking forward towards permanent fix and will update here once I get updates or find any further changes in the routing tables.