24 Sep

BSNL-Level3 bad routing case

Quick analysis of BSNL-Level3 bad routing issue

 

I can see BSNL having pretty high latency again with most of Europe again. It seems like they are using Level3 Communications AS 3356 along with Tata-VSNL for upstream. With Level3 transit BSNL has badly screwed up reverse path causing very high latency and awful bandwidth.

 

Expected latency values here should be around 150ms. A packet should not take more then 150ms round trip between Radaur, Haryana to Munich located server.

 

Quick view at traceroute:

Clearly hop 3 is New Delhi (30ms latency), hop 4 is Mumbai (again as per latency values). Hop 5 is London Level3. Seems like BSNL used Europe-India gateway link here (a submarine cable from Mumbai to London owned by multiple providers including BSNL and Bharti Airtel along with Global Crossing which is now owned by Level3). Also, as far as I know Level3 does not has a ISP license in India (doT’s list here) and thus they cannot sell bandwidth at Mumbai. Likely BSNL is using its own ILD license in this case and thus BSNL is responsible for purchase of bandwidth in London.

Thus, as per that traceroute and fact that BSNL is one who is purchasing transit from Level3 in London, BSNL should be having BGP session in London and should be exchanging it’s routing table in turn for global routing table provided by transit. While latency jumps as soon as we hit London as per that traceroute. Clearing BSNL > Level3 path seems OK while return path on Level3 > BSNL is faulty. 

 

Using Level3’s looking glass, we can have a quick check on traceroute to my IP:

 

Hop3 – Level3, hop4 is Gblx (which is now owned by Level3), hop 5 is Gblx New York and hop 6 is BSNL router in New York. The target BSNL ip is coming from 117.207.48.0/20. Now interesting thing here is BSNL uses Level3 + Gblx both for transit. So return path via Gblx is not an issue but the path London > New York > India is surely an issue.

 

Looking for prefix 117.207.48.0/20 in Level3 London router:

Only two paths that too via Gblx. No direct return path. Again, it is not big issue since Gblx could have a return path right within London (or somewhere else in Europe).

 

Let’s check Gblx Europe router for entry for 117.207.48.0/20

 

 Just one path. Doing a traceroute to see the actual path (since I don’t know where that next hop is located!) 🙂

 

 

Clearly here’s the issue. BSNL again is doing selective BGP announcement of prefixes at New York only and that is why Europe to India traffic is being routed via New York. BSNL is allowing entry path into it’s network from outside India only at New York and few other selected locations which causes serious damage to latency.

 

Time for me to get back on work of routing packets! Thanks for reading. 🙂

20 Jan

Broken Indian Backbone Networks

Just came across another case of high latency. This one was special so decided to share via blog post.

 

Here we start!

Testing routing to European networks from Tata Communications Mumbai nodes.
Test IP of my Germany based server: 93.104.209.174 

Checking routing from both Autonomous Systems of Tata Communications – AS4755 (VSNL) and AS6453 (TeleGlobe).

 

[DDET Click to expand ping result]

Router: TATA COMM MUMBAI(AS4755) 
Site: TATA COMM MUMBAI(AS4755) 
Command: ping 93.104.209.174

Sending 5, 100-byte ICMP Echos to 93.104.209.174, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 264/326/444 ms

[/DDET]

326ms latency is way too high for route between India and Europe.

 

Next, checking from other Autonomous System AS6453:

[DDET Click to expand ping result]

Router: gin-mlv-core1
Site: IN, Mumbai – MLV, VSNL
Command: ping 93.104.209.174

Sending 5, 100-byte ICMP Echos to 93.104.209.174, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 132/132/136 ms

[/DDET]

132ms – pretty good something what we can expect for this route.

So what’s really wrong? Why VSNL-AS4755 has high latency while AS6453 low – even when AS4755 routes (most of) international traffic via AS6453 which is very popular system (and one of tier 1 networks). 

 

Let’s observe traceroute from both Mumbai PoPs of each system

[DDET Click to expand traceroute from AS4755] 

Router: TATA COMM MUMBAI(AS4755)
Site: TATA COMM MUMBAI(AS4755)
Command: traceroute 93.104.209.174

Tracing the route to 93.104.209.174

1 59.163.16.14 [MPLS: Label 8237 Exp 0] 200 msec
59.163.55.150 [MPLS: Label 2188 Exp 0] 200 msec
121.240.12.110 [MPLS: Label 2188 Exp 0] 200 msec
2 172.31.33.125 200 msec
172.31.33.201 400 msec 4 msec
3 180.87.39.25 200 msec 204 msec 0 msec
4 80.231.130.5 [AS 6453] 404 msec 260 msec 408 msec
5 80.231.130.10 [AS 6453] 396 msec * *
6 195.219.83.102 [AS 6453] 264 msec 252 msec 264 msec
7 4.69.139.120 [AS 3356] 256 msec 256 msec 416 msec
8 4.69.153.141 [AS 3356] 420 msec 404 msec 404 msec
9 4.69.148.186 [AS 3356] 408 msec
4.69.148.194 [AS 3356] 400 msec 420 msec
10 4.69.140.26 [AS 3356] 408 msec
4.69.140.30 [AS 3356] 408 msec 412 msec
11 4.69.140.9 [AS 3356] 376 msec
4.69.140.13 [AS 3356] 268 msec
4.69.140.5 [AS 3356] 268 msec
12 4.69.134.1 [AS 3356] 320 msec 424 msec 432 msec
13 62.140.24.50 [AS 3356] 280 msec 272 msec 272 msec
14 212.18.7.157 [AS 8767] 276 msec 272 msec 272 msec
15 93.104.204.34 [AS 8767] 276 msec 276 msec 276 msec
16 93.104.209.174 [AS 8767] 264 msec 268 msec 272 msec

[/DDET]

 

Comparing with results from other ASN

[DDET Click to expand traceroute from AS6453]

Router: gin-mlv-core1
Site: IN, Mumbai – MLV, VSNL
Command: traceroute ip 93.104.209.174

Tracing the route to server7.anuragbhatia.com (93.104.209.174)

1 if-13-0-0.core2.MLV-Mumbai.as6453.net (209.58.105.42) [MPLS: Label 512 Exp 0] 204 msec
if-3-0-0.core2.MLV-Mumbai.as6453.net (209.58.105.82) [MPLS: Label 512 Exp 0] 308 msec
if-2-1-0-0.tcore1.MLV-Mumbai.as6453.net (180.87.38.21) [MPLS: Label 664353 Exp 0] 200 msec
2 if-2-2.tcore2.MLV-Mumbai.as6453.net (180.87.38.2) [MPLS: Label 686484 Exp 0] 308 msec 312 msec
if-10-3-1-0.tcore2.MLV-Mumbai.as6453.net (180.87.39.61) [MPLS: Label 686484 Exp 0] 200 msec
3 if-6-2.tcore1.L78-London.as6453.net (80.231.130.5) 204 msec 200 msec 200 msec
4 Vlan704.icore1.LDN-London.as6453.net (80.231.130.10) 200 msec 148 msec *
5 Vlan533.icore1.LDN-London.as6453.net (195.219.83.102) 120 msec 124 msec 124 msec
6 ae-52-52.csw2.London1.Level3.net (4.69.139.120) [AS 3356] 124 msec 124 msec 128 msec
7 ae-57-222.ebr2.London1.Level3.net (4.69.153.133) [AS 3356] 128 msec
ae-58-223.ebr2.London1.Level3.net (4.69.153.137) [AS 3356] 132 msec
ae-59-224.ebr2.London1.Level3.net (4.69.153.141) [AS 3356] 124 msec
8 ae-24-24.ebr2.Frankfurt1.Level3.net (4.69.148.198) [AS 3356] 132 msec
ae-21-21.ebr2.Frankfurt1.Level3.net (4.69.148.186) [AS 3356] 124 msec
ae-24-24.ebr2.Frankfurt1.Level3.net (4.69.148.198) [AS 3356] 128 msec
9 ae-92-92.csw4.Frankfurt1.Level3.net (4.69.140.30) [AS 3356] 136 msec
ae-62-62.csw1.Frankfurt1.Level3.net (4.69.140.18) [AS 3356] 196 msec 132 msec
10 ae-91-91.ebr1.Frankfurt1.Level3.net (4.69.140.13) [AS 3356] 132 msec
ae-81-81.ebr1.Frankfurt1.Level3.net (4.69.140.9) [AS 3356] 124 msec
ae-71-71.ebr1.Frankfurt1.Level3.net (4.69.140.5) [AS 3356] 124 msec
11 ae-4-4.car1.Munich1.Level3.net (4.69.134.1) [AS 3356] 416 msec 208 msec 200 msec
12 62.140.24.50 [AS 3356] 200 msec 148 msec 132 msec
13 ten0-1-0-0.r5.muc2.m-online.net (212.18.7.157) [AS 8767] 140 msec 132 msec 136 msec
14 gw01.giga-hosting.biz (93.104.204.34) [AS 8767] 132 msec 132 msec 132 msec
15 server7.anuragbhatia.com (93.104.209.174) [AS 8767] 140 msec 136 msec 132 msec

[/DDET] 

We can see beyond hop 4 in AS4755 traceroute, path is pretty much same for AS6453. So something is wrong within 1st 4 hops of AS4755.

 

 

Looking at initial part again:

1 59.163.16.14 [MPLS: Label 8237 Exp 0] 200 msec
59.163.55.150 [MPLS: Label 2188 Exp 0] 200 msec
121.240.12.110 [MPLS: Label 2188 Exp 0] 200 msec
2 172.31.33.125 200 msec
172.31.33.201 400 msec 4 msec
3 180.87.39.25 200 msec 204 msec 0 msec
4 80.231.130.5 [AS 6453] 404 msec 260 msec 408 msec

 

Taking lowest (i.e best) possible values on hop 3 = 0 ms, hop 4 = 260ms. Something is wrong here. Since we can clearly see it’s a jump between two Autonomous Systems (though belonging to same operator), looking at reverse path from AS6453 to AS4755 test IP – 59.163.16.14 from 59.163.16.0/22 announcement of AS4755 in Mumbai.

 

Tracing return paths from Mumbai & London PoP’s of AS6453:

[DDET Click to expand traceroute]

Router: gin-mlv-core1
Site: IN, Mumbai – MLV, VSNL
Command: traceroute ip 59.163.16.14

Tracing the route to 59.163.16.14.static.vsnl.net.in (59.163.16.14)

1 if-8-1-0-0.tcore1.MLV-Mumbai.as6453.net (180.87.38.17) [MPLS: Label 302880 Exp 0] 0 msec
if-13-0-0.core2.MLV-Mumbai.as6453.net (209.58.105.42) [MPLS: Label 68 Exp 0] 0 msec
if-3-0-0.core2.MLV-Mumbai.as6453.net (209.58.105.82) [MPLS: Label 68 Exp 0] 0 msec
2 if-2-2.tcore2.MLV-Mumbai.as6453.net (180.87.38.2) 4 msec 0 msec 0 msec
3 180.87.39.26 20 msec 0 msec 0 msec
4 172.31.33.126 4 msec
172.29.250.18 4 msec *

 [/DDET]

Pretty good – Mumbai AS6453 to Mumbai AS4755 – 4ms latency – fine.

Next, looking at route from London AS6453 to Mumbai AS4755 router:

[DDET Click to expand traceroute]

Router: gin-lhx-core1
Site: GB, London – LHX, TATA COMM. HARBOR EXCHANGE
Command: traceroute ip 59.163.16.14

Tracing the route to 59.163.16.14.static.vsnl.net.in (59.163.16.14)

1 if-10-0.core2.LHX-London.as6453.net (195.219.5.34) [MPLS: Label 2318 Exp 0] 4 msec
if-5-0.core2.LHX-London.as6453.net (195.219.15.218) [MPLS: Label 2318 Exp 0] 4 msec
if-10-0.core2.LHX-London.as6453.net (195.219.5.34) [MPLS: Label 2318 Exp 0] 0 msec
2 if-3-0.mcore3.LDN-London.as6453.net (195.219.195.1) [MPLS: Label 1217 Exp 0] 0 msec 4 msec 0 msec
3 if-1-1-1-0.tcore1.L78-London.as6453.net (195.219.195.10) [MPLS: Label 678531 Exp 0] 144 msec 140 msec 156 msec
4 if-4-2.tcore1.NJY-Newark.as6453.net (80.231.130.34) [MPLS: Label 401680 Exp 0] 176 msec 148 msec 144 msec
5 if-2-2.tcore2.NJY-Newark.as6453.net (66.198.70.2) [MPLS: Label 388752 Exp 0] 148 msec 140 msec 144 msec
6 if-3-2.tcore2.AEQ-Ashburn.as6453.net (216.6.87.9) [MPLS: Label 304096 Exp 0] 144 msec 144 msec 144 msec
7 if-8-1508.tcore2.LVW-LosAngeles.as6453.net (64.86.252.73) [MPLS: Label 305424 Exp 0] 140 msec 144 msec 140 msec
8 if-2-2.tcore1.LVW-LosAngeles.as6453.net (66.110.59.1) 144 msec 140 msec 140 msec
9 180.87.39.81 260 msec 264 msec 260 msec
10 172.31.33.182 268 msec * 264 msec

[/DDET]

And here we go – London to Mumbai trip via New York-Los Angles!

 

Summary

Forward path from Mumbai to Germany based server is like:

Mumbai (AS4755) – Mumbai (AS6453) – London (AS6453) – further to Level3 (AS3356) which carries it to destination network

 

Return path is like:

Server in Germany – routed to London via Level3 – handed over to Tata (AS6453) in London – New Yark (AS6453) – Los Angles (AS6453) – Mumbai (AS4755). 

 

This broken reverse path is screwing up connectivity of VSNL backbone from Europe since packets are returning from US with very high latency.

 

What exactly is causing this issue?

Well, problem seems because of London PoP’s (infact many others in Europe) of Tata AS6453 which are not having proper entry in BGP routing table for blocks holded by VSNL-AS4755. 

 

E.g looking at BGP table for VSNL test IP 59.163.16.14 from London Telehouse Tata router:

[DDET Click  to expand BGP output]

Router: gin-lhx-core1
Site: GB, London – LHX, TATA COMM. HARBOR EXCHANGE
Command: show ip bgp 59.163.16.14

BGP routing table entry for 59.163.16.0/22
Bestpath Modifiers: deterministic-med
Paths: (1 available, best #1)
Multipath: eBGP
4755 4755
lvw-tcore1. (metric 3133) from l78-tcore1. (66.110.10.237)
Origin IGP, valid, internal, best
Community:
Originator: 66.110.10.247

 [/DDET]

Clearly – only route is to Los Angles.

 

Quick check on Los Angles BGP table:

 [DDET Click to expand BGP output]

Router: gin-laa-mcore3
Site: US, Los Angeles – LAA, TATA COMM. TGBUS
Command: show ip bgp 59.163.16.14

BGP routing table entry for 59.163.16.0/22
Bestpath Modifiers: deterministic-med
Paths: (8 available, best #3)
12 13 15 16 17 19
4755 4755
lvw-tcore1. (metric 1) from pdi-mcore4. (pdi-mcore4.)
Origin IGP, valid, internal
Community:
Originator: 66.110.10.247
4755 4755
lvw-tcore1. (metric 1) from dtx-core1. (dtx-core1.)
Origin IGP, valid, internal
Community:
Originator: 66.110.10.247
4755 4755
lvw-tcore1. (metric 1) from lvw-tcore1. (66.110.10.247)
Origin IGP, valid, internal, best
Community:
4755
mlv-tcore2. (metric 6569) from mlv-tcore2. (66.110.10.215)
Origin IGP, valid, internal
Community:
4755
svw-tcore1. (metric 3397) from hk2-core3. (hk2-core3.)
Origin IGP, valid, internal
Community:
Originator: 66.110.10.113
4755
svw-tcore2. (metric 3396) from s9r-core1. (s9r-core1.)
Origin IGP, valid, internal
Community:
Originator: 66.110.10.113
4755
cxr-tcore1. (metric 6544) from cxr-tcore1. (66.110.10.113)
Origin IGP, valid, internal
Community:
4755
tv2-tcore2. (metric 3046) from tv2-core1. (tv2-core1.)
Origin IGP, valid, internal
Community:
Originator: 66.110.10.113

 [/DDET]

 

Here we can see entry for route to Mumbai. 

 

So is Tata Communications AS6453 has issues on routing tables within own network?

Well, not exactly. If we check route from London AS6453 to Mumbai AS6453:

[DDET Click to expand traceroute result]

Router: gin-ldn-mcore3
Site: GB, London – LDN, TATA COMM. TELEHOUSE
Command: traceroute ip 209.58.105.42

Tracing the route to if-13-0-0.core2.MLV-Mumbai.as6453.net (209.58.105.42)

1 if-1-1-1-0.tcore1.L78-London.as6453.net (195.219.195.10) 116 msec
if-1-2-2-0.tcore1.L78-London.as6453.net (195.219.195.30) 4 msec
if-1-1-1-0.tcore1.L78-London.as6453.net (195.219.195.10) 0 msec
2 if-6-2.tcore2.MLV-Mumbai.as6453.net (80.231.130.6) 208 msec 120 msec 136 msec
3 if-5-0-0.core2.MLV-Mumbai.as6453.net (180.87.39.73) 132 msec 120 msec 212 msec

 [/DDET]

Pretty much good. So whole problem is between routing tables for AS4755 which used to be VSNL – Govt, owned monopoly in India and AS6453 which used to TeleGlobe – both of which now belongs to Tata Communications.

 

Both of these Autonomous Systems play an important role since AS6453 is one of big tier 1 backbone network, while AS4755 has significant presence in India and most of domestic ISP’s purchase bandwidth from Tata Communications via AS4755 backbone. The extent of brokeness can be seen from a end user BSNL connection – e.g a traceroute to Mumbai based PoP of AS6453 is goes like:

[DDET Click to expand traceroute]

traceroute to 209.58.105.42 (209.58.105.42), 30 hops max, 60 byte packets
1 router.local (192.168.1.1) [AS8151/AS28513] 4.402 ms 4.994 ms 5.791 ms
2 117.207.48.1 (117.207.48.1) [AS9829] 31.590 ms 33.010 ms 35.378 ms
3 218.248.173.46 (218.248.173.46) [AS9829] 39.866 ms 41.187 ms 43.588 ms
4 if-14-1.mcore4.NYY-NewYork.as6453.net (64.86.71.5) [AS23520] 390.739 ms 391.655 ms 392.353 ms
5 if-8-0-2-28.tcore1.NYY-NewYork.as6453.net (209.58.60.34) [AS18895] 280.781 ms 283.087 ms 285.724 ms
6 if-6-10.tcore1.PYE-Paris.as6453.net (216.6.90.10) [AS7633] 389.172 ms 355.571 ms 356.519 ms
7 if-8-1600.tcore1.WYN-Marseille.as6453.net (80.231.217.5) [AS6453] 370.170 ms 373.257 ms 375.285 ms
8 if-9-5.tcore1.MLV-Mumbai.as6453.net (80.231.217.18) [AS6453] 457.250 ms 483.489 ms 461.521 ms
9 if-2-1-0.core1.MLV-Mumbai.as6453.net (180.87.38.9) [*] 491.050 ms if-9-0-0.core1.MLV-Mumbai.as6453.net (180.87.38.18) [*] 473.610 ms if-2-1-0.core1.MLV-Mumbai.as6453.net (180.87.38.9) [*] 474.672 ms
10 if-13-0-0.core2.MLV-Mumbai.as6453.net (209.58.105.42) [AS6453] 469.137 ms 439.925 ms 442.429 ms

[/DDET]

With hope that Tata Communications will eventually fix these broken backbones, I get back to my Java exam preparation! 🙂

 

 

Disclaimer: Analysis done in this post is purely based on facts found from publically available information. I have no intention to harm Tata Communications from this. If anyone finds anything incorrect in this post, please feel free to contact me, and I would be happy to fix it.