20 Jan

Broken Indian Backbone Networks

Just came across another case of high latency. This one was special so decided to share via blog post.

 

Here we start!

Testing routing to European networks from Tata Communications Mumbai nodes.
Test IP of my Germany based server: 93.104.209.174 

Checking routing from both Autonomous Systems of Tata Communications – AS4755 (VSNL) and AS6453 (TeleGlobe).

 

[DDET Click to expand ping result]

Router: TATA COMM MUMBAI(AS4755) 
Site: TATA COMM MUMBAI(AS4755) 
Command: ping 93.104.209.174

Sending 5, 100-byte ICMP Echos to 93.104.209.174, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 264/326/444 ms

[/DDET]

326ms latency is way too high for route between India and Europe.

 

Next, checking from other Autonomous System AS6453:

[DDET Click to expand ping result]

Router: gin-mlv-core1
Site: IN, Mumbai – MLV, VSNL
Command: ping 93.104.209.174

Sending 5, 100-byte ICMP Echos to 93.104.209.174, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 132/132/136 ms

[/DDET]

132ms – pretty good something what we can expect for this route.

So what’s really wrong? Why VSNL-AS4755 has high latency while AS6453 low – even when AS4755 routes (most of) international traffic via AS6453 which is very popular system (and one of tier 1 networks). 

 

Let’s observe traceroute from both Mumbai PoPs of each system

[DDET Click to expand traceroute from AS4755] 

Router: TATA COMM MUMBAI(AS4755)
Site: TATA COMM MUMBAI(AS4755)
Command: traceroute 93.104.209.174

Tracing the route to 93.104.209.174

1 59.163.16.14 [MPLS: Label 8237 Exp 0] 200 msec
59.163.55.150 [MPLS: Label 2188 Exp 0] 200 msec
121.240.12.110 [MPLS: Label 2188 Exp 0] 200 msec
2 172.31.33.125 200 msec
172.31.33.201 400 msec 4 msec
3 180.87.39.25 200 msec 204 msec 0 msec
4 80.231.130.5 [AS 6453] 404 msec 260 msec 408 msec
5 80.231.130.10 [AS 6453] 396 msec * *
6 195.219.83.102 [AS 6453] 264 msec 252 msec 264 msec
7 4.69.139.120 [AS 3356] 256 msec 256 msec 416 msec
8 4.69.153.141 [AS 3356] 420 msec 404 msec 404 msec
9 4.69.148.186 [AS 3356] 408 msec
4.69.148.194 [AS 3356] 400 msec 420 msec
10 4.69.140.26 [AS 3356] 408 msec
4.69.140.30 [AS 3356] 408 msec 412 msec
11 4.69.140.9 [AS 3356] 376 msec
4.69.140.13 [AS 3356] 268 msec
4.69.140.5 [AS 3356] 268 msec
12 4.69.134.1 [AS 3356] 320 msec 424 msec 432 msec
13 62.140.24.50 [AS 3356] 280 msec 272 msec 272 msec
14 212.18.7.157 [AS 8767] 276 msec 272 msec 272 msec
15 93.104.204.34 [AS 8767] 276 msec 276 msec 276 msec
16 93.104.209.174 [AS 8767] 264 msec 268 msec 272 msec

[/DDET]

 

Comparing with results from other ASN

[DDET Click to expand traceroute from AS6453]

Router: gin-mlv-core1
Site: IN, Mumbai – MLV, VSNL
Command: traceroute ip 93.104.209.174

Tracing the route to server7.anuragbhatia.com (93.104.209.174)

1 if-13-0-0.core2.MLV-Mumbai.as6453.net (209.58.105.42) [MPLS: Label 512 Exp 0] 204 msec
if-3-0-0.core2.MLV-Mumbai.as6453.net (209.58.105.82) [MPLS: Label 512 Exp 0] 308 msec
if-2-1-0-0.tcore1.MLV-Mumbai.as6453.net (180.87.38.21) [MPLS: Label 664353 Exp 0] 200 msec
2 if-2-2.tcore2.MLV-Mumbai.as6453.net (180.87.38.2) [MPLS: Label 686484 Exp 0] 308 msec 312 msec
if-10-3-1-0.tcore2.MLV-Mumbai.as6453.net (180.87.39.61) [MPLS: Label 686484 Exp 0] 200 msec
3 if-6-2.tcore1.L78-London.as6453.net (80.231.130.5) 204 msec 200 msec 200 msec
4 Vlan704.icore1.LDN-London.as6453.net (80.231.130.10) 200 msec 148 msec *
5 Vlan533.icore1.LDN-London.as6453.net (195.219.83.102) 120 msec 124 msec 124 msec
6 ae-52-52.csw2.London1.Level3.net (4.69.139.120) [AS 3356] 124 msec 124 msec 128 msec
7 ae-57-222.ebr2.London1.Level3.net (4.69.153.133) [AS 3356] 128 msec
ae-58-223.ebr2.London1.Level3.net (4.69.153.137) [AS 3356] 132 msec
ae-59-224.ebr2.London1.Level3.net (4.69.153.141) [AS 3356] 124 msec
8 ae-24-24.ebr2.Frankfurt1.Level3.net (4.69.148.198) [AS 3356] 132 msec
ae-21-21.ebr2.Frankfurt1.Level3.net (4.69.148.186) [AS 3356] 124 msec
ae-24-24.ebr2.Frankfurt1.Level3.net (4.69.148.198) [AS 3356] 128 msec
9 ae-92-92.csw4.Frankfurt1.Level3.net (4.69.140.30) [AS 3356] 136 msec
ae-62-62.csw1.Frankfurt1.Level3.net (4.69.140.18) [AS 3356] 196 msec 132 msec
10 ae-91-91.ebr1.Frankfurt1.Level3.net (4.69.140.13) [AS 3356] 132 msec
ae-81-81.ebr1.Frankfurt1.Level3.net (4.69.140.9) [AS 3356] 124 msec
ae-71-71.ebr1.Frankfurt1.Level3.net (4.69.140.5) [AS 3356] 124 msec
11 ae-4-4.car1.Munich1.Level3.net (4.69.134.1) [AS 3356] 416 msec 208 msec 200 msec
12 62.140.24.50 [AS 3356] 200 msec 148 msec 132 msec
13 ten0-1-0-0.r5.muc2.m-online.net (212.18.7.157) [AS 8767] 140 msec 132 msec 136 msec
14 gw01.giga-hosting.biz (93.104.204.34) [AS 8767] 132 msec 132 msec 132 msec
15 server7.anuragbhatia.com (93.104.209.174) [AS 8767] 140 msec 136 msec 132 msec

[/DDET] 

We can see beyond hop 4 in AS4755 traceroute, path is pretty much same for AS6453. So something is wrong within 1st 4 hops of AS4755.

 

 

Looking at initial part again:

1 59.163.16.14 [MPLS: Label 8237 Exp 0] 200 msec
59.163.55.150 [MPLS: Label 2188 Exp 0] 200 msec
121.240.12.110 [MPLS: Label 2188 Exp 0] 200 msec
2 172.31.33.125 200 msec
172.31.33.201 400 msec 4 msec
3 180.87.39.25 200 msec 204 msec 0 msec
4 80.231.130.5 [AS 6453] 404 msec 260 msec 408 msec

 

Taking lowest (i.e best) possible values on hop 3 = 0 ms, hop 4 = 260ms. Something is wrong here. Since we can clearly see it’s a jump between two Autonomous Systems (though belonging to same operator), looking at reverse path from AS6453 to AS4755 test IP – 59.163.16.14 from 59.163.16.0/22 announcement of AS4755 in Mumbai.

 

Tracing return paths from Mumbai & London PoP’s of AS6453:

[DDET Click to expand traceroute]

Router: gin-mlv-core1
Site: IN, Mumbai – MLV, VSNL
Command: traceroute ip 59.163.16.14

Tracing the route to 59.163.16.14.static.vsnl.net.in (59.163.16.14)

1 if-8-1-0-0.tcore1.MLV-Mumbai.as6453.net (180.87.38.17) [MPLS: Label 302880 Exp 0] 0 msec
if-13-0-0.core2.MLV-Mumbai.as6453.net (209.58.105.42) [MPLS: Label 68 Exp 0] 0 msec
if-3-0-0.core2.MLV-Mumbai.as6453.net (209.58.105.82) [MPLS: Label 68 Exp 0] 0 msec
2 if-2-2.tcore2.MLV-Mumbai.as6453.net (180.87.38.2) 4 msec 0 msec 0 msec
3 180.87.39.26 20 msec 0 msec 0 msec
4 172.31.33.126 4 msec
172.29.250.18 4 msec *

 [/DDET]

Pretty good – Mumbai AS6453 to Mumbai AS4755 – 4ms latency – fine.

Next, looking at route from London AS6453 to Mumbai AS4755 router:

[DDET Click to expand traceroute]

Router: gin-lhx-core1
Site: GB, London – LHX, TATA COMM. HARBOR EXCHANGE
Command: traceroute ip 59.163.16.14

Tracing the route to 59.163.16.14.static.vsnl.net.in (59.163.16.14)

1 if-10-0.core2.LHX-London.as6453.net (195.219.5.34) [MPLS: Label 2318 Exp 0] 4 msec
if-5-0.core2.LHX-London.as6453.net (195.219.15.218) [MPLS: Label 2318 Exp 0] 4 msec
if-10-0.core2.LHX-London.as6453.net (195.219.5.34) [MPLS: Label 2318 Exp 0] 0 msec
2 if-3-0.mcore3.LDN-London.as6453.net (195.219.195.1) [MPLS: Label 1217 Exp 0] 0 msec 4 msec 0 msec
3 if-1-1-1-0.tcore1.L78-London.as6453.net (195.219.195.10) [MPLS: Label 678531 Exp 0] 144 msec 140 msec 156 msec
4 if-4-2.tcore1.NJY-Newark.as6453.net (80.231.130.34) [MPLS: Label 401680 Exp 0] 176 msec 148 msec 144 msec
5 if-2-2.tcore2.NJY-Newark.as6453.net (66.198.70.2) [MPLS: Label 388752 Exp 0] 148 msec 140 msec 144 msec
6 if-3-2.tcore2.AEQ-Ashburn.as6453.net (216.6.87.9) [MPLS: Label 304096 Exp 0] 144 msec 144 msec 144 msec
7 if-8-1508.tcore2.LVW-LosAngeles.as6453.net (64.86.252.73) [MPLS: Label 305424 Exp 0] 140 msec 144 msec 140 msec
8 if-2-2.tcore1.LVW-LosAngeles.as6453.net (66.110.59.1) 144 msec 140 msec 140 msec
9 180.87.39.81 260 msec 264 msec 260 msec
10 172.31.33.182 268 msec * 264 msec

[/DDET]

And here we go – London to Mumbai trip via New York-Los Angles!

 

Summary

Forward path from Mumbai to Germany based server is like:

Mumbai (AS4755) – Mumbai (AS6453) – London (AS6453) – further to Level3 (AS3356) which carries it to destination network

 

Return path is like:

Server in Germany – routed to London via Level3 – handed over to Tata (AS6453) in London – New Yark (AS6453) – Los Angles (AS6453) – Mumbai (AS4755). 

 

This broken reverse path is screwing up connectivity of VSNL backbone from Europe since packets are returning from US with very high latency.

 

What exactly is causing this issue?

Well, problem seems because of London PoP’s (infact many others in Europe) of Tata AS6453 which are not having proper entry in BGP routing table for blocks holded by VSNL-AS4755. 

 

E.g looking at BGP table for VSNL test IP 59.163.16.14 from London Telehouse Tata router:

[DDET Click  to expand BGP output]

Router: gin-lhx-core1
Site: GB, London – LHX, TATA COMM. HARBOR EXCHANGE
Command: show ip bgp 59.163.16.14

BGP routing table entry for 59.163.16.0/22
Bestpath Modifiers: deterministic-med
Paths: (1 available, best #1)
Multipath: eBGP
4755 4755
lvw-tcore1. (metric 3133) from l78-tcore1. (66.110.10.237)
Origin IGP, valid, internal, best
Community:
Originator: 66.110.10.247

 [/DDET]

Clearly – only route is to Los Angles.

 

Quick check on Los Angles BGP table:

 [DDET Click to expand BGP output]

Router: gin-laa-mcore3
Site: US, Los Angeles – LAA, TATA COMM. TGBUS
Command: show ip bgp 59.163.16.14

BGP routing table entry for 59.163.16.0/22
Bestpath Modifiers: deterministic-med
Paths: (8 available, best #3)
12 13 15 16 17 19
4755 4755
lvw-tcore1. (metric 1) from pdi-mcore4. (pdi-mcore4.)
Origin IGP, valid, internal
Community:
Originator: 66.110.10.247
4755 4755
lvw-tcore1. (metric 1) from dtx-core1. (dtx-core1.)
Origin IGP, valid, internal
Community:
Originator: 66.110.10.247
4755 4755
lvw-tcore1. (metric 1) from lvw-tcore1. (66.110.10.247)
Origin IGP, valid, internal, best
Community:
4755
mlv-tcore2. (metric 6569) from mlv-tcore2. (66.110.10.215)
Origin IGP, valid, internal
Community:
4755
svw-tcore1. (metric 3397) from hk2-core3. (hk2-core3.)
Origin IGP, valid, internal
Community:
Originator: 66.110.10.113
4755
svw-tcore2. (metric 3396) from s9r-core1. (s9r-core1.)
Origin IGP, valid, internal
Community:
Originator: 66.110.10.113
4755
cxr-tcore1. (metric 6544) from cxr-tcore1. (66.110.10.113)
Origin IGP, valid, internal
Community:
4755
tv2-tcore2. (metric 3046) from tv2-core1. (tv2-core1.)
Origin IGP, valid, internal
Community:
Originator: 66.110.10.113

 [/DDET]

 

Here we can see entry for route to Mumbai. 

 

So is Tata Communications AS6453 has issues on routing tables within own network?

Well, not exactly. If we check route from London AS6453 to Mumbai AS6453:

[DDET Click to expand traceroute result]

Router: gin-ldn-mcore3
Site: GB, London – LDN, TATA COMM. TELEHOUSE
Command: traceroute ip 209.58.105.42

Tracing the route to if-13-0-0.core2.MLV-Mumbai.as6453.net (209.58.105.42)

1 if-1-1-1-0.tcore1.L78-London.as6453.net (195.219.195.10) 116 msec
if-1-2-2-0.tcore1.L78-London.as6453.net (195.219.195.30) 4 msec
if-1-1-1-0.tcore1.L78-London.as6453.net (195.219.195.10) 0 msec
2 if-6-2.tcore2.MLV-Mumbai.as6453.net (80.231.130.6) 208 msec 120 msec 136 msec
3 if-5-0-0.core2.MLV-Mumbai.as6453.net (180.87.39.73) 132 msec 120 msec 212 msec

 [/DDET]

Pretty much good. So whole problem is between routing tables for AS4755 which used to be VSNL – Govt, owned monopoly in India and AS6453 which used to TeleGlobe – both of which now belongs to Tata Communications.

 

Both of these Autonomous Systems play an important role since AS6453 is one of big tier 1 backbone network, while AS4755 has significant presence in India and most of domestic ISP’s purchase bandwidth from Tata Communications via AS4755 backbone. The extent of brokeness can be seen from a end user BSNL connection – e.g a traceroute to Mumbai based PoP of AS6453 is goes like:

[DDET Click to expand traceroute]

traceroute to 209.58.105.42 (209.58.105.42), 30 hops max, 60 byte packets
1 router.local (192.168.1.1) [AS8151/AS28513] 4.402 ms 4.994 ms 5.791 ms
2 117.207.48.1 (117.207.48.1) [AS9829] 31.590 ms 33.010 ms 35.378 ms
3 218.248.173.46 (218.248.173.46) [AS9829] 39.866 ms 41.187 ms 43.588 ms
4 if-14-1.mcore4.NYY-NewYork.as6453.net (64.86.71.5) [AS23520] 390.739 ms 391.655 ms 392.353 ms
5 if-8-0-2-28.tcore1.NYY-NewYork.as6453.net (209.58.60.34) [AS18895] 280.781 ms 283.087 ms 285.724 ms
6 if-6-10.tcore1.PYE-Paris.as6453.net (216.6.90.10) [AS7633] 389.172 ms 355.571 ms 356.519 ms
7 if-8-1600.tcore1.WYN-Marseille.as6453.net (80.231.217.5) [AS6453] 370.170 ms 373.257 ms 375.285 ms
8 if-9-5.tcore1.MLV-Mumbai.as6453.net (80.231.217.18) [AS6453] 457.250 ms 483.489 ms 461.521 ms
9 if-2-1-0.core1.MLV-Mumbai.as6453.net (180.87.38.9) [*] 491.050 ms if-9-0-0.core1.MLV-Mumbai.as6453.net (180.87.38.18) [*] 473.610 ms if-2-1-0.core1.MLV-Mumbai.as6453.net (180.87.38.9) [*] 474.672 ms
10 if-13-0-0.core2.MLV-Mumbai.as6453.net (209.58.105.42) [AS6453] 469.137 ms 439.925 ms 442.429 ms

[/DDET]

With hope that Tata Communications will eventually fix these broken backbones, I get back to my Java exam preparation! 🙂

 

 

Disclaimer: Analysis done in this post is purely based on facts found from publically available information. I have no intention to harm Tata Communications from this. If anyone finds anything incorrect in this post, please feel free to contact me, and I would be happy to fix it.

19 Jan

Tata Communications – NTT routing issue for Akamai

Interestingly routing issues didn’t spare one of top CDN provider – Akamai!

 

So what’s wrong?

(from my BSNL connection):

PING akamai.com (61.213.189.49) 56(84) bytes of data.
64 bytes from 61.213.189.49: icmp_req=1 ttl=52 time=492 ms
64 bytes from 61.213.189.49: icmp_req=2 ttl=52 time=492 ms
64 bytes from 61.213.189.49: icmp_req=3 ttl=52 time=474 ms
64 bytes from 61.213.189.49: icmp_req=4 ttl=51 time=492 ms
64 bytes from 61.213.189.49: icmp_req=5 ttl=51 time=489 ms

— akamai.com ping statistics —
5 packets transmitted, 5 received, 0% packet loss, time 22236ms
rtt min/avg/max/mdev = 474.296/488.469/492.837/7.183 ms

 

~ 500ms is way too high. Even US is at like 300ms latency.

 

Looking at traceroute: 

[DDET traceroute from BSNL – Click to expand ]

traceroute to akamai.com (61.213.189.49), 30 hops max, 60 byte packets
1 router.local (192.168.1.1) [AS8151/AS28513] 4.223 ms 4.979 ms 5.879 ms
2 117.200.48.1 (117.200.48.1) [AS9829] 45.241 ms 46.384 ms 52.839 ms
3 218.248.173.46 (218.248.173.46) [AS9829] 87.089 ms * *
4 115.114.57.165.static-Mumbai.vsnl.net.in (115.114.57.165) [AS4755] 74.675 ms 76.970 ms 80.856 ms
5 if-0-100.tcore2.MLV-Mumbai.as6453.net (180.87.39.25) [*] 83.234 ms 84.403 ms 87.742 ms
6 if-6-2.tcore1.L78-London.as6453.net (80.231.130.5) [AS6453] 230.777 ms 185.553 ms 194.288 ms
7 * Vlan704.icore1.LDN-London.as6453.net (80.231.130.10) [AS6453] 203.104 ms *
8 Vlan522.icore1.LDN-London.as6453.net (195.219.83.22) [AS6453] 308.973 ms 310.324 ms 311.038 ms
9 ae-4.r23.londen03.uk.bb.gin.ntt.net (129.250.5.40) [AS2914] 311.799 ms 333.841 ms 313.348 ms
10 as-0.r22.osakjp01.jp.bb.gin.ntt.net (129.250.5.35) [AS2914] 499.075 ms 501.158 ms 512.657 ms
11 ae-5.r24.tokyjp01.jp.bb.gin.ntt.net (129.250.3.221) [AS2914] 484.258 ms 485.401 ms 499.039 ms
12 * * *
13 xe-2-3.a17.tokyjp01.jp.ra.gin.ntt.net (61.213.169.214) [AS2914] 488.807 ms 489.543 ms 495.396 ms
14 61.213.189.49 (61.213.189.49) [AS2914] 506.170 ms 501.504 ms 507.296 ms

 [/DDET]

So route is like Mumbai (India) – London (UK) – Tokyo (Japan). 

 

It’s not really issue with BSNL at this point but with Tata Communications AS6453. Looking at traceroute from Tata Comm’s Mumbai node using their looking glass.

[DDET Click to expand traceroute from Tata ]

Router: gin-mlv-core1
Site: IN, Mumbai – MLV, VSNL
Command: traceroute ip 61.213.189.49

Tracing the route to 61.213.189.49

1 if-3-0-0.core2.MLV-Mumbai.as6453.net (209.58.105.82) [MPLS: Label 512 Exp 0] 0 msec
if-2-1-0-0.tcore1.MLV-Mumbai.as6453.net (180.87.38.21) [MPLS: Label 664353 Exp 0] 132 msec
if-3-1-0-0.tcore1.MLV-Mumbai.as6453.net (180.87.38.10) [MPLS: Label 664353 Exp 0] 140 msec
2 if-2-2.tcore2.MLV-Mumbai.as6453.net (180.87.38.2) [MPLS: Label 686484 Exp 0] 128 msec
if-10-3-1-0.tcore2.MLV-Mumbai.as6453.net (180.87.39.61) [MPLS: Label 686484 Exp 0] 136 msec 136 msec
3 if-6-2.tcore1.L78-London.as6453.net (80.231.130.5) 116 msec 128 msec 116 msec
4 *
Vlan704.icore1.LDN-London.as6453.net (80.231.130.10) 204 msec *
5 Vlan522.icore1.LDN-London.as6453.net (195.219.83.22) 132 msec 136 msec 136 msec
6 ae-4.r23.londen03.uk.bb.gin.ntt.net (129.250.5.40) [AS 2914] [MPLS: Label 302128 Exp 0] 128 msec 128 msec 136 msec
7 as-0.r22.osakjp01.jp.bb.gin.ntt.net (129.250.5.35) [AS 2914] [MPLS: Label 643206 Exp 0] 364 msec 364 msec 408 msec
8 ae-5.r24.tokyjp01.jp.bb.gin.ntt.net (129.250.3.221) [AS 2914] 376 msec 372 msec 368 msec
9 xe-3-1.a17.tokyjp01.jp.ra.gin.ntt.net (61.213.169.150) [AS 2914] 372 msec 380 msec 388 msec
10 xe-2-3.a17.tokyjp01.jp.ra.gin.ntt.net (61.213.169.214) [AS 2914] 480 msec 516 msec 400 msec
11 61.213.189.49 [AS 2914] 404 msec 388 msec 384 msec

 [/DDET]

Clearly identical traceroute and poor routing between Tata AS6453 and NTT AS2914.

 

Looking at route from Reliance Flagtel Mumbai node:

[DDET Click to expand traceroute from Reliance]

You requested a Traceroute from Mumbai (62.216.135.130) to 61.213.189.49
1 ge-0-2-0.0.pjr02.mmb004.flagtel.com (85.95.25.70) 131.460 ms 131.710 ms 131.509 ms
MPLS Label=422848 CoS=0 TTL=1 S=1
2 so-1-3-0.0.pjr02.hkg005.flagtel.com (85.95.25.118) 131.409 ms 131.465 ms 131.679 ms
MPLS Label=373792 CoS=0 TTL=1 S=1
3 so-0-2-0.0.pjr02.wad001.flagtel.com (85.95.25.189) 133.951 ms 129.679 ms 129.712 ms
MPLS Label=346448 CoS=0 TTL=1 S=1
4 so-7-1-0.0.cjr04.tok002.flagtel.com (85.95.25.202) 131.280 ms 131.285 ms 131.210 ms
5 80.81.64.74 (80.81.64.74) 132.028 ms 131.745 ms 131.768 ms
6 a60-254-153-245.deploy.akamaitechnologies.com (60.254.153.245) [AS 20940] 132.706 ms 133.643 ms 134.960 ms
7 61.213.189.49 (61.213.189.49) [AS 20940] 132.558 ms 132.508 ms 132.577 ms

[/DDET]

Good – direct route. No issues on Reliance network.

 

Unfortunately I can’t test with Airtel since their looking glass seems not working at all right now and I am already busy with exams to wait & try again. (if someone from Airtel reading this post, please please & please setup a good looking glass for AS9498).

 

Quick check on what’s wrong with Tata.
I tried looking at route from their Tokyo node, and here’s output:

[DDET Click to expand traceroute]

Router: gin-tv2-core1
Site: JP, Tokyo – TV2, EQUINIX
Command: traceroute ip 61.213.189.49

Tracing the route to 61.213.189.49

1 if-0-2-1-40.tcore1.TV2-Tokyo.as6453.net (180.87.180.41) [MPLS: Label 316726 Exp 0] 108 msec
if-8-2-1-38.tcore1.TV2-Tokyo.as6453.net (209.58.61.118) [MPLS: Label 316726 Exp 0] 108 msec
if-0-2-1-40.tcore1.TV2-Tokyo.as6453.net (180.87.180.41) [MPLS: Label 316726 Exp 0] 112 msec
2 if-9-2.tcore2.PDI-PaloAlto.as6453.net (180.87.180.17) 104 msec 108 msec 104 msec
3 Vlan3254.icore1.SQN-SanJose.as6453.net (66.198.144.6) 116 msec 108 msec 108 msec
4 Vlan507.icore1.SQN-SanJose.as6453.net (209.58.116.22) 108 msec 108 msec 108 msec
5 ae-7.r21.snjsca04.us.bb.gin.ntt.net (129.250.5.54) [AS 2914] [MPLS: Label 310512 Exp 0] 108 msec 104 msec
ae-6.r20.snjsca04.us.bb.gin.ntt.net (129.250.5.12) [AS 2914] [MPLS: Label 301136 Exp 0] 108 msec
6 as-2.r20.tokyjp01.jp.bb.gin.ntt.net (129.250.2.35) [AS 2914] [MPLS: Label 574335 Exp 0] 240 msec
as-0.r21.tokyjp01.jp.bb.gin.ntt.net (129.250.4.45) [AS 2914] [MPLS: Label 470118 Exp 0] 216 msec
as-2.r20.tokyjp01.jp.bb.gin.ntt.net (129.250.2.35) [AS 2914] [MPLS: Label 574335 Exp 0] 220 msec
7 ae-1.r25.tokyjp01.jp.bb.gin.ntt.net (129.250.2.73) [AS 2914] 228 msec
ae-1.r24.tokyjp01.jp.bb.gin.ntt.net (129.250.2.21) [AS 2914] 308 msec
ae-1.r25.tokyjp01.jp.bb.gin.ntt.net (129.250.2.73) [AS 2914] 296 msec
8 xe-3-1.a17.tokyjp01.jp.ra.gin.ntt.net (61.213.169.150) [AS 2914] 228 msec
xe-4-1.a17.tokyjp01.jp.ra.gin.ntt.net (61.213.169.154) [AS 2914] 212 msec
xe-3-1.a17.tokyjp01.jp.ra.gin.ntt.net (61.213.169.150) [AS 2914] 228 msec
9 xe-2-3.a17.tokyjp01.jp.ra.gin.ntt.net (61.213.169.214) [AS 2914] 204 msec 256 msec 212 msec
10 61.213.189.49 [AS 2914] 220 msec 212 msec 216 msec

[/DDET]

So route is like Tokyo (Japan) – San Jose (US) – Packet handover to NTT – (back to) Tokyo (Japan). It seems Tata AS6453 is exchanging traffic with NTT AS2914 in London and California but not in Japan itself.

 

Looking at BGP table for Akamai IP from Tata AS6453 Mumbai node:

[DDET Click to expand BGP output]

Router: gin-mlv-core1
Site: IN, Mumbai – MLV, VSNL
Command: show ip bgp 61.213.189.49

BGP routing table entry for 61.213.160.0/19
Bestpath Modifiers: deterministic-med
Paths: (3 available, best #1)
Multipath: eBGP
2914
ldn-icore1. (metric 2991) from cxr-tcore1. (66.110.10.113)
Origin IGP, valid, internal, best
Community:
Originator: ldn-icore1.
2914
ldn-icore1. (metric 2991) from mlv-tcore2. (66.110.10.215)
Origin IGP, valid, internal
Community:
Originator: ldn-icore1.
2914
ldn-icore1. (metric 2991) from mlv-tcore1. (66.110.10.202)
Origin IGP, valid, internal
Community:
Originator: ldn-icore1.

[/DDET]

Clearly – all routes to London. No route to Singapore or Japan directly.

 

Next, check on Japan node BGP table: 

[DDET Click to expand BGP table output]

Router: gin-tv2-core1
Site: JP, Tokyo – TV2, EQUINIX
Command: show ip bgp 61.213.189.49

BGP routing table entry for 61.213.160.0/19
Bestpath Modifiers: deterministic-med
Paths: (9 available, best #9)
2914
lvw-tcore2. (metric 3329) from svq-core1. (svq-core1.)
Origin IGP, valid, internal
Community:
Originator: 66.110.10.248
2914
av2-tcore2. (metric 6417) from ad1-core1. (ad1-core1.)
Origin IGP, valid, internal
Community:
Originator: 66.110.10.88
2914
lvw-tcore2. (metric 3329) from laa-mcore3. (laa-mcore3.)
Origin IGP, valid, internal
Community:
Originator: 66.110.10.248
2914
ldn-icore1. (metric 6408) from l78-mcore3. (Loopback5.mcore3.L78-London.)
Origin IGP, valid, internal
Community:
Originator: ldn-icore1.
2914
pvu-tcore1. (metric 6396) from pye-core1. (pye-core1.)
Origin IGP, valid, internal
Community:
Originator: 66.110.10.90
2914
nto-icore1. (metric 3405) from nyy-mcore4. (nyy-mcore4.)
Origin IGP, valid, internal
Community:
Originator: nto-icore1.
2914
dt8-tcore2. (metric 3346) from dtx-core1. (dtx-core1.)
Origin IGP, valid, internal
Community:
Originator: Loopback5.bb1.LAU-Laurentides.as6453.net.
2914
ct8-tcore1. (metric 3364) from ct8-core1. (ct8-core1.)
Origin IGP, valid, internal
Community:
Originator: 66.110.11.16
2914
sqn-icore1. (metric 3318) from pdi-mcore4. (pdi-mcore4.)
Origin IGP, valid, internal, best
Community:
Originator: sqn-icore1.

 [/DDET]

9 routes – London, New York, California but no direct route. Tata’s nodes are not listening to any announcement for that block in Tokyo directly.
It’s not that just Tata is not handling packets to NTT in Japan but reverse is also true. Looking at route from NTT Japan node to Tata’s Japan node:

 

NTT Tokyo to Tata Tokyo node: 

[DDET Click to expand traceroute]

Query Results:
Router: Tokyo – JP
Command: traceroute 209.58.61.118

Disclaimer: Traceroute is a useful tool for determining the route a packet takes, but it should not be used as an accurate measure of network performance. For more information please view the Traceroute Disclaimer.

traceroute to 209.58.61.118 (209.58.61.118), 30 hops max, 40 byte packets
1 ae-0.r21.tokyjp01.jp.bb.gin.ntt.net (129.250.2.72) 35.957 ms 13.354 ms 30.162 ms
MPLS Label=433510 CoS=0 TTL=1 S=0
MPLS Label=422560 CoS=0 TTL=1 S=1
2 as-0.r21.lsanca03.us.bb.gin.ntt.net (129.250.3.145) 133.142 ms ae-7.r21.osakjp01.jp.bb.gin.ntt.net (129.250.2.70) 35.097 ms 9.023 ms
MPLS Label=631431 CoS=0 TTL=1 S=0
MPLS Label=303728 CoS=0 TTL=2 S=1
3 ae-0.r20.lsanca03.us.bb.gin.ntt.net (129.250.3.33) 139.183 ms 121.634 ms 140.311 ms
MPLS Label=306720 CoS=0 TTL=1 S=1
4 ix-11-2-1-0.tcore2.LVW-LosAngeles.as6453.net (64.86.252.65) 109.416 ms ix-9-1-2-0.tcore2.LVW-LosAngeles.as6453.net (216.6.84.65) 147.333 ms ae-1.r05.lsanca03.us.bb.gin.ntt.net (129.250.2.230) 126.940 ms
5 Vlan507.icore1.SQN-SanJose.as6453.net (209.58.116.21) 152.854 ms 132.124 ms 146.938 ms
6 if-2-2.tcore1.LVW-LosAngeles.as6453.net (66.110.59.1) 172.250 ms if-4-2.tcore1.PDI-PaloAlto.as6453.net (66.198.127.9) 121.056 ms 112.809 ms
7 if-8-2-1-38.tcore1.TV2-Tokyo.as6453.net (209.58.61.118) 223.186 ms if-4-2.tcore1.PDI-PaloAlto.as6453.net (66.198.127.9) 108.956 ms if-8-2-1-38.tcore1.TV2-Tokyo.as6453.net (209.58.61.118) 244.013 ms

{master}

 [/DDET]

Clearly same  – a round trip to US!

Overall this is bit surprising for me since both Tata AS6453 and NTT AS2914 are considered as Tier 1 backbone players and they usually exchange traffic in settlement free peering. In this case  Tata & NTT are peering but not in Japan (why?). No clear ideas on why. Will try finding out after my exams! 🙂

 

With hope that you are reaching my blog without having a round trip across globe 😉 time for me to get back on cramming for college exams!