26 Apr

Broken connectivity to F root server in India

It has been an interesting week at village – dry weather, (ultra) dry classes, (boring) external seminars and more of depressing environment but one can always find some hope out of such depressing environment. Overall life here is colourful but one just needs to lookout for colours. ūüôā
 
One interesting case to report today – F root server has quite bad connectivity in India. Last week a friend asked me for traceroutes to all root servers and here’s what I saw when I did traceroute for F root from BSNL connection:
traceroute to f.root-servers.net (192.5.5.241), 30 hops max, 60 byte packets
1 192.168.1.1 (192.168.1.1) [AS8151/AS28513] 1.508 ms 2.103 ms 2.614 ms
2 117.207.48.1 (117.207.48.1) [AS9829] 27.243 ms 29.811 ms 32.483 ms
3 218.248.173.42 (218.248.173.42) [AS9829] 39.861 ms 40.320 ms 40.755 ms
4 218.248.250.142 (218.248.250.142) [AS9829] 90.778 ms 93.919 ms 95.856 ms
5 * * *
6 * * *
7 * * *
8 * * *
 
There’s no route!¬†
 
This is not a overnight problem. I looked at my hosted RIPE Probe #1032 data and found this:
 

 
 
 
How can an ISP like BSNL can have missing route to one of key part of core Internet infrastructure?
 

Quick look at F root server:

 
 
F root server has a anycasting node deployed in Chennai, Tamil Nadu, India. This server is hosted at National Internet Exchange of India (NIXI) Chennai and ISC is responsible for this root server.
F root uses 192.5.5.241 across all anycasting instances and in India this block is being announced by Autonomous System Number¬†24049 which is of ISC. AS24049 does a BGP announcement for ¬†203.119.18.0/24¬†at NIXI Chennai and this is “supposed to be” taken by all ISPs participating at Chennai IXP.
 
When I requested my friends across India for traceroutes to F root, it was a very interesting result!
We found connectivity works from everyone  including BSNL too for people living in or near around Chennai.
 
 
Here’s a traceroute from my friend in Chennai on BSNL to F root:
 
traceroute to f.root-servers.net (192.5.5.241), 30 hops max, 60 byte packets
1 192.168.1.1 (192.168.1.1) 3.328 ms 3.311 ms 4.616 ms
2 59.92.64.1 (59.92.64.1) 32.489 ms 36.188 ms 37.448 ms
3 218.248.255.2 (218.248.255.2) 45.312 ms 48.171 ms 51.017 ms
4 218.248.250.142 (218.248.250.142) 63.279 ms 66.086 ms 69.025 ms
5 218.100.48.142 (218.100.48.142) 71.112 ms 72.939 ms 76.107 ms
6 192.5.5.241 (192.5.5.241) 79.131 ms 44.760 ms 46.550 ms
 
Here we can see hop 4 is core network of BSNL while hop 5 is NIXI Chennai and hop 6 is ultimately F root server.
Initially I got feeling that this is because of broken IGP implementation for BSNL network since their border gateway routers are holding different routing tables and they are not syncing them properly but one strange thing here РHop 4 in this traceroute i.e 218.248.250.142 is same as last hop in 1st traceroute (done from BSNL Haryana). How come same router has no route when looked from Haryana while it works for Chennai (and nearby) users!
This gives clue that forward path is there but return path has issues. For some reason ISC router in NIXI Chennai is not able to get return path to reply for non-Chennai region users.
I raised this concern with Network Operations Center of ISC yesterday along with mailing lists like APNIC & RIPE Atlas. One of my friend who is expert into these issues pointed us to right direction which is NIXI routing policy.
 
Reading out policy:
”¬†An ISP at any NIXI node must at a minimum announce all its regional routes to the NIXI router at that NIXI location. All ISPs connecting to that NIXI node are entitled to receive these routes using a single BGP session with the NIXI router. This will guarantee the exchange of regional traffic within a NIXI node. This is referred to as forced regional multi-lateral peering under the policy”
 
So this is what is happening:
Each operator is providing ISC with limited regional routes around Chennai area and not to it’s entire network. This was later confirmed by ISC NOC reply. This is sort of awful situation in terms of policy which is breaking India backbones badly and ISPs not able to even reach root servers instances hosted within country. Worst, I have been told that there is 10 day time for fix of this problem else ISC is¬†prepared¬†to pull off plug from Chennai F root node. Since it’s anycasting, once there won’t be any instance inside India injecting routes, traffic will simply start flowing towards other F root server instances in nearby countries. If this happens, it will be surely a sad thing for India since we will loose one out of three four¬†root nameservers. ūüôĀ
 
With hope that next update on this issue will be positive, time to end this post for now. Feel free to share your comments below.
(Incase your are ISP or datacenter in India – I would be happy to discuss this issue with you. You can contact me directly from here)
 
 

***Updates***

 
ISC Network Operations Center acted very efficiently on this issue.  Special thanks to Mr Leo Bicknell from ISC who is taking care of it. In initial phase full routing table was made available to F root server via STPI Internet transit. This fixed the problem (on test basis) and after a week of testing,  F root instance in Chennai has been switched off until ISC & NIXI get into a contract over additional bandwidth and costs.
For now Indian traffic is being routed to other anycasting instance of F root. Earlier it was California, US instance and now it is Hong Kong instance.
traceroute to f.root-servers.net. (192.5.5.241), 30 hops max, 60 byte packets
1 router.local (192.168.1.1) [AS8151/AS28513] 2.536 ms 2.799 ms 3.624 ms
2 117.200.48.1 (117.200.48.1) [AS9829] 29.778 ms 32.488 ms 34.677 ms
3 218.248.173.38 (218.248.173.38) [AS9829] 36.966 ms 43.283 ms 44.119 ms
4 59.163.207.81.static.chennai.vsnl.net.in (59.163.207.81) [AS4755] 189.973 ms 192.398 ms 195.118 ms
5 ix-4-2.tcore1.CXR-Chennai.as6453.net (180.87.36.9) [*] 199.965 ms 202.296 ms 203.396 ms
6 if-5-2.tcore1.SVW-Singapore.as6453.net (180.87.12.53) [*] 206.915 ms if-3-3.tcore2.CXR-Chennai.as6453.net (180.87.36.6) [*] 166.267 ms if-5-2.tcore1.SVW-Singapore.as6453.net (180.87.12.53) [*] 160.231 ms
7 if-2-2.tcore2.SVW-Singapore.as6453.net (180.87.12.2) [*] 161.044 ms if-5-2.tcore2.SVW-Singapore.as6453.net (180.87.15.69) [*] 164.876 ms 167.178 ms
8 Vlan1870.icore1.HK2-HongKong.as6453.net (180.87.15.61) [*] 217.994 ms Vlan1779.icore1.HK2-HongKong.as6453.net (180.87.15.38) [*] 216.922 ms Vlan1870.icore1.HK2-HongKong.as6453.net (180.87.15.61) [*] 218.664 ms
9 isc2-FE.hkix.net (202.40.161.200) [AS2687/AS4862/AS9498/AS10026/AS1221] 206.692 ms 208.920 ms isc1-FE.hkix.net (202.40.161.202) [AS2687/AS4862/AS9498/AS10026/AS1221] 211.561 ms
10 f.root-servers.net (192.5.5.241) [AS55440/AS3557/AS23708/AS8167] 247.908 ms 252.645 ms 253.544 ms
 
 
I am looking forward towards permanent fix and will update here once I get updates or find any further changes in the routing tables.

10 Apr

BSNL routing tables screw up

It has been super boring evening considering my sessional tests tomorrow. Test time is dull as always. I have been precisely measnuring latency on BSNL link from BSNL Haryana to Singapore based servers. The fluctuation in latency is pretty much common now. Someones we get 120ms latency to Singapore (an expected number based on distance) while other time it goes off as high as 310ms. Latency with openDNS nodes in Singapore makes it pretty much poor to use openDNS here.
 
Based on my collected data and BGPlay’s routing records, here’s what’s happening. My IP is coming /20 BGP annoucement from BSNL Autonomous System 9829 –¬†117.207.48.0/20. Looking at BGP table records for¬†that block from BGPlay’s routing data¬†archive¬†source.
 
On Sunday Morning 00:00am UTC, BSNL was found to be announcing¬†117.207.48.0/20 from AS9829 which was carried over via it’s upstream ISPs – Tata Communications (AS6453) and Reliance Globalcom (AS18101). From Tata’s AS6453 most of other Tier 1 backbones and many other small ISPs were getting routes. Similarly in case of Reliance, AS18101 was announcing blocks to it’s other network FLAG Telecom AS15412 which was further passing routes across many ISPs in the world. ISPs like Tinet, AT&T, Savvis, Seabone, Sprint etc were getting announcement via AS6453 while Level3, NTT, Hurricane Electric, Swisscom & many others were getting annoucements via Reliance-FLAG backbone.

At this instance – connectivity to network via Reliance route looks pretty good. All traffic comes via direct path – entering Reliance’s network from nearest point & next reaching BSNL. While at the same time, unfortunately in case of other path via Tata Communications – it seems like no matter where traffic orignates from, it is always routed via US. That is even if traffic is orignating from Singapore, it is being routed to India via US.
 

Quick check on Tata Communications AS6453 PoPs at this instant:

Router: gin-svq-core1
Site: SG, Singapore – SVQ, EQUNIX
Command: show ip bgp 117.207.48.0/20
BGP routing table entry for 117.207.48.0/20
Bestpath Modifiers: deterministic-med
Paths: (3 available, best #2)
Multipath: eBGP
9829
nyy-mcore4. (metric 3777) from tv2-core1. (tv2-core1.)
Origin IGP, valid, internal
Community:
Originator: nyy-mcore4.
9829
nyy-mcore4. (metric 3777) from hk2-core3. (hk2-core3.)
Origin IGP, valid, internal, best
Community:
Originator: nyy-mcore4.
9829
nyy-mcore4. (metric 3777) from s9r-core1. (s9r-core1.)
Origin IGP, valid, internal
Community:
Originator: nyy-mcore4.
 
Router: gin-lhx-core1
Site: GB, London – LHX, TATA COMM. HARBOR EXCHANGE
Command: show ip bgp 117.207.48.0/20
BGP routing table entry for 117.207.48.0/20
Bestpath Modifiers: deterministic-med
Paths: (2 available, best #1)
Multipath: eBGP
9829
nyy-mcore4. (metric 3036) from ldn-mcore3. (ldn-mcore3.)
Origin IGP, valid, internal, best
Community:
Originator: nyy-mcore4.
9829
nyy-mcore4. (metric 3036) from l78-tcore1. (66.110.10.237)
Origin IGP, valid, internal
Community:
Originator: nyy-mcore4.
 
 
Thus we can see in both cases – Tata’s router is getting updates from¬†nyy-mcore4 which is their router in New York city. This forces packets all way down to New York before they are routed back in Asia to BSNL.
 
Next, same thing goes for next few hours. On Sunday noon time 12:55pm, we can see a path change from multiple providers like Savvis AS3561. We can see change in routing table and now between Tata AS6453 and BSNL AS9829, a new network comes in. It is AS4755 which is Tata Communications other network VSNL. Tata still uses AS4755 in India and AS6453 everywhere else.

Within next few seconds, we can see similar changes from AS293 – ESnet,
Path Change from 3561 6453 9829
to 3561 6453 4755 9829
Path Change from 293 6453 9829
to 293 6453 4755 9829
Path Change from 3303 15412 18101 9829
to 3303 6453 4755 9829
Path Change from  812 6453 9829
to 812 6453 4755 9829
Path Change from 3257 6453 9829
to 3257 6453 4755 9829
Path Change from 3130 1239 6453 9829
to 3130 1239 6453 4755 9829 
Path Change from 7018 6453 9829
to 7018 6453 4755 9829 
Path Change from 1299 6453 9829
to 1299 6453 4755 9829 
Within a min, we can see half of routes are going via AS4755 rather then AS6453 directly.
Next, we see a route withdrawal – 701 6453 9829 followed by route re-annoucement 701 6453 4755 9829 and with this we see whole block being announced via AS4755 (which exists only in India as per as I know). Next on 12:58pm, we see another series of path changes all reversing back to AS6453 – AS9829 skipping AS4755 in between. On 12:50pm we can see half of routes being back with AS6453 link directly and half still with AS6453. Very soon all routes get back on direct link AS4755 goes out of picture again.

Summary of what’s happening:

  1. BSNL’s network is having high latency on various routes including Singapore and Europe.
  2. For specific block in testing, we can see BSNL is announcing them via Tata Communications & Reliance Globalcom.
  3. Tata Communications uses two autonomous systems – AS4755 (VSNL) for Indian operations & AS6453 for everywhere else.
  4. We can see routing table changes almost daily which bring AS4755 between AS9829 and AS6453 on random basis.
  5. AS4755 is believed to be operated only in India and I can see whenever AS4755 is coming in picture, packets to BSNL-AS9829 are handed off directly and latency is pretty good.
  6. For other times when we have AS6453 > AS9829 routing, we can see block is being annouced from AS6453 in New York.
  7. Very likely problem is NOT from Tata Communications end, but rather from BSNL’s end. BSNL is constantly switching annoucing peers – AS4755 at one end in India while AS6453 router on other end in New York. Though it is hard to confirm this speculation.

 
 
 
Glad Airlines don’t route flights in the manner in which wrong routing goes! ūüôā