BSNL routing tables screw up

It has been super boring evening considering my sessional tests tomorrow. Test time is dull as always. I have been precisely measnuring latency on BSNL link from BSNL Haryana to Singapore based servers. The fluctuation in latency is pretty much common now. Someones we get 120ms latency to Singapore (an expected number based on distance) while other time it goes off as high as 310ms. Latency with openDNS nodes in Singapore makes it pretty much poor to use openDNS here. 

Based on my collected data and BGPlay’s routing records, here’s what’s happening. My IP is coming /20 BGP annoucement from BSNL Autonomous System 9829 - 117.207.48.0/20. Looking at BGP table records for that block from BGPlay’s routing data archive source.

On Sunday Morning 00:00am UTC, BSNL was found to be announcing 117.207.48.0/20 from AS9829 which was carried over via it’s upstream ISPs – Tata Communications (AS6453) and Reliance Globalcom (AS18101). From Tata’s AS6453 most of other Tier 1 backbones and many other small ISPs were getting routes. Similarly in case of Reliance, AS18101 was announcing blocks to it’s other network FLAG Telecom AS15412 which was further passing routes across many ISPs in the world. ISPs like Tinet, AT&T, Savvis, Seabone, Sprint etc were getting announcement via AS6453 while Level3, NTT, Hurricane Electric, Swisscom & many others were getting annoucements via Reliance-FLAG backbone. 

At this instance – connectivity to network via Reliance route looks pretty good. All traffic comes via direct path – entering Reliance’s network from nearest point & next reaching BSNL. While at the same time, unfortunately in case of other path via Tata Communications – it seems like no matter where traffic orignates from, it is always routed via US. That is even if traffic is orignating from Singapore, it is being routed to India via US.

Quick check on Tata Communications AS6453 PoPs at this instant:

Router: gin-svq-core1
Site: SG, Singapore – SVQ, EQUNIX
Command: show ip bgp 117.207.48.0/20

BGP routing table entry for 117.207.48.0/20
Bestpath Modifiers: deterministic-med
Paths: (3 available, best #2)
Multipath: eBGP
9829
nyy-mcore4. (metric 3777) from tv2-core1. (tv2-core1.)
Origin IGP, valid, internal
Community:
Originator: nyy-mcore4.

9829
nyy-mcore4. (metric 3777) from hk2-core3. (hk2-core3.)
Origin IGP, valid, internal, best
Community:
Originator: nyy-mcore4.

9829
nyy-mcore4. (metric 3777) from s9r-core1. (s9r-core1.)
Origin IGP, valid, internal
Community:
Originator: nyy-mcore4.

Router: gin-lhx-core1
Site: GB, London – LHX, TATA COMM. HARBOR EXCHANGE
Command: show ip bgp 117.207.48.0/20

BGP routing table entry for 117.207.48.0/20
Bestpath Modifiers: deterministic-med
Paths: (2 available, best #1)
Multipath: eBGP
9829
nyy-mcore4. (metric 3036) from ldn-mcore3. (ldn-mcore3.)
Origin IGP, valid, internal, best
Community:
Originator: nyy-mcore4.

9829
nyy-mcore4. (metric 3036) from l78-tcore1. (66.110.10.237)
Origin IGP, valid, internal
Community:
Originator: nyy-mcore4.

Thus we can see in both cases – Tata’s router is getting updates from nyy-mcore4 which is their router in New York city. This forces packets all way down to New York before they are routed back in Asia to BSNL. 

Next, same thing goes for next few hours. On Sunday noon time 12:55pm, we can see a path change from multiple providers like Savvis AS3561. We can see change in routing table and now between Tata AS6453 and BSNL AS9829, a new network comes in. It is AS4755 which is Tata Communications other network VSNL. Tata still uses AS4755 in India and AS6453 everywhere else.

Within next few seconds, we can see similar changes from AS293 – ESnet, 
 

Path Change from 3561 6453 9829
to 3561 6453 4755 9829

Path Change from 293 6453 9829
to 293 6453 4755 9829

Path Change from 3303 15412 18101 9829
to 3303 6453 4755 9829

Path Change from  812 6453 9829
to 812 6453 4755 9829

Path Change from 3257 6453 9829
to 3257 6453 4755 9829

Path Change from 3130 1239 6453 9829
to 3130 1239 6453 4755 9829 

Path Change from 7018 6453 9829
to 7018 6453 4755 9829 

Path Change from 1299 6453 9829
to 1299 6453 4755 9829 

Within a min, we can see half of routes are going via AS4755 rather then AS6453 directly.

Next, we see a route withdrawal – 701 6453 9829 followed by route re-annoucement 701 6453 4755 9829 and with this we see whole block being announced via AS4755 (which exists only in India as per as I know). Next on 12:58pm, we see another series of path changes all reversing back to AS6453 – AS9829 skipping AS4755 in between. On 12:50pm we can see half of routes being back with AS6453 link directly and half still with AS6453. Very soon all routes get back on direct link AS4755 goes out of picture again.

Summary of what’s happening:

1. BSNL’s network is having high latency on various routes including Singapore and Europe.
2. For specific block in testing, we can see BSNL is announcing them via Tata Communications & Reliance Globalcom.
3. Tata Communications uses two autonomous systems – AS4755 (VSNL) for Indian operations & AS6453 for everywhere else.
4. We can see routing table changes almost daily which bring AS4755 between AS9829 and AS6453 on random basis.
5. AS4755 is believed to be operated only in India and I can see whenever AS4755 is coming in picture, packets to BSNL-AS9829 are handed off directly and latency is pretty good.
6. For other times when we have AS6453 > AS9829 routing, we can see block is being annouced from AS6453 in New York.
7. Very likely problem is NOT from Tata Communications end, but rather from BSNL’s end. BSNL is constantly switching annoucing peers – AS4755 at one end in India while AS6453 router on other end in New York. Though it is hard to confirm this speculation.

Glad Airlines don’t route flights in the manner in which wrong routing goes!