19 Dec

Google Public DNS and Akamai issues in India

A quick blog post on a interesting issue coming up due to combined problem of CDN failure on Google Public DNS and bad Akamai performance due to Tata-NTT peering issue.

 I was trying Zembra mail since there’s no more free Google Apps edition and one of my friend asked me to basic email on his domain up. It was more or less a straight task by installing Zembra with decent GUI.

 

I downloaded it on my Europe based server and during installation realized it was for 64 bit and thus I turned my head to my other server in India.
I started download again and it was slow. DEAD SLOW!

 

Something like this:

root@server2:~# wget http://files2.zimbra.com/downloads/8.0.1_GA/zcs-8.0.1_GA_5438.UBUNTU12_64.20121105164409.tgz
–2012-12-18 14:02:59– http://files2.zimbra.com/downloads/8.0.1_GA/zcs-8.0.1_GA_5438.UBUNTU12_64.20121105164409.tgz
Resolving files2.zimbra.com (files2.zimbra.com)… 23.32.241.26, 125.56.200.51
Connecting to files2.zimbra.com (files2.zimbra.com)|23.32.241.26|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 701053545 (669M) [binary/octet-stream]
Saving to: zcs-8.0.1_GA_5438.UBUNTU12_64.20121105164409.tgz.1'

0% [ ] 5,545,378 67.5K/s eta 2h 28m ^C
root@server2:~#

 

Would have taken 2hrs + on 512Kbps speed while server is on 100Mbps connection and I usually get 20Mbps or so for US/Europe based sources. Since I downloaded same 700MB file on Europe based server and it was quite fast 40Mbps+ while here just 512kbps.

 

I looked at route from Indian server and route was:

traceroute to 23.32.241.26 (23.32.241.26), 30 hops max, 60 byte packets
1 103.6.87.1 (103.6.87.1) [AS36236] 1.310 ms 1.426 ms 1.716 ms
2 180.179.33.245 (180.179.33.245) [AS17439/AS9584] 0.843 ms 0.951 ms 0.958 ms
3 180.179.37.93 (180.179.37.93) [AS17439] 0.761 ms 0.762 ms 0.750 ms
4 * * *
5 180.179.37.137 (180.179.37.137) [AS17439] 0.840 ms 0.938 ms 1.091 ms
6 59.163.105.170.static-chennai.vsnl.net.in (59.163.105.170) [AS4755] 4.441 ms 3.891 ms 3.848 ms
7 * * *
8 ix-0-100.tcore2.MLV-Mumbai.as6453.net (180.87.39.25) [*] 27.175 ms 27.178 ms 27.927 ms
9 if-6-2.tcore1.L78-London.as6453.net (80.231.130.5) [AS6453] 143.917 ms 145.977 ms if-2-2.tcore1.MLV-Mumbai.as6453.net (180.87.38.1) [*] 135.972 ms
10 if-9-5.tcore1.WYN-Marseille.as6453.net (80.231.217.17) [AS6453] 132.133 ms 133.902 ms 133.286 ms
11 if-8-1600.tcore1.PYE-Paris.as6453.net (80.231.217.6) [AS6453] 134.763 ms 133.217 ms 136.691 ms
12 if-2-2.tcore1.PVU-Paris.as6453.net (80.231.154.17) [AS6453] 134.674 ms 137.558 ms *
13 * * *
14 ae-1.r21.parsfr01.fr.bb.gin.ntt.net (129.250.2.224) [AS2914] 153.657 ms 155.550 ms 152.412 ms
15 as-4.r22.amstnl02.nl.bb.gin.ntt.net (129.250.3.84) [AS2914] 155.136 ms 151.697 ms as-0.r25.tokyjp01.jp.bb.gin.ntt.net (129.250.3.79) [AS2914] 383.668 ms
16 * * *
17 * xe-3-2.a16.tokyjp01.jp.ra.gin.ntt.net (203.105.72.78) [AS2914] 271.296 ms 270.202 ms
18 a23-32-241-26.deploy.akamaitechnologies.com (23.32.241.26) [AS20940] 280.674 ms 282.407 ms xe-3-2.a16.tokyjp01.jp.ra.gin.ntt.net (203.105.72.78) [AS2914] 263.254 ms

 

Akami CDN node in Japan and route via Europe!!

 

This poor performance case is result of multiple issues:

  1. Datacenter is not running own DNS server but instead replying on Google Public DNS 8.8.8.8 and 8.8.4.4 which has upstream/full table connectivity outside in East Asia and not really in India. Thus Google DNS resolvers always pass IP of nodes outside India in Japan or sometimes in Malaysia. I blogged about it in detail sometimes back here.
  2. Datacenter is picking Tata-VSNL AS4755 for upstream for this route (not Airtel or Reliance), and interesting enough Tata does NOT peers with NTT Communications in Japan. They peer everywhere else except home market of NTT which I guess will be case with other ISPs as well. Thus nearest point for Tata to handle traffic to NTT is Europe which we can see in traceroute. 
  3. Since IP belongs to Akamai Japan, it brings traffic back to Asia over NTT Asia's network and eventually passes it to Akamai. Strange that Akamai has not fixed this problem from long time. They must be knowing this since I posted this problem in NANOG mailing list last year and also blogged about it here.

 

OK - the fix!

I can surely do better then waiting for 2 hours to download that package! 
I quickly installed BIND and since BIND runs as "recursive resolver" by default, I simply pointed /etc/resolv.conf to 127.0.0.1

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# nameserver 8.8.8.8
# nameserver 8.8.4.4

nameserver 127.0.0.1

 

OK - now running download again, let's see how it works:

 

root@server2:~/tmp2# wget http://files2.zimbra.com/downloads/8.0.1_GA/zcs-8.0.1_GA_5438.UBUNTU12_64.20121105164409.tgz
--2012-12-18 14:31:55-- http://files2.zimbra.com/downloads/8.0.1_GA/zcs-8.0.1_GA_5438.UBUNTU12_64.20121105164409.tgz
Resolving files2.zimbra.com (files2.zimbra.com)... 125.252.226.97, 125.252.226.106
Connecting to files2.zimbra.com (files2.zimbra.com)|125.252.226.97|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 701053545 (669M) [binary/octet-stream]
Saving to:
zcs-8.0.1_GA_5438.UBUNTU12_64.20121105164409.tgz’

100%[============================================================================================================>] 701,053,545 3.07M/s in 3m 45s

2012-12-18 14:35:43 (2.98 MB/s) – `zcs-8.0.1_GA_5438.UBUNTU12_64.20121105164409.tgz’ saved [701053545/701053545]

root@server2:~/tmp2# 

 

Fast? Yeah a lot!

 

How?

Simply doing a trace to destination this time takes me to: 

traceroute to 125.252.226.97 (125.252.226.97), 30 hops max, 60 byte packets
1 103.6.87.1 (103.6.87.1) [AS36236] 2.089 ms 2.060 ms 2.047 ms
2 180.179.33.245 (180.179.33.245) [AS17439/AS9584] 2.012 ms 2.002 ms 1.997 ms
3 180.179.37.89 (180.179.37.89) [AS17439] 1.940 ms 1.949 ms 1.937 ms
4 180.179.37.38 (180.179.37.38) [AS17439] 2.258 ms 2.668 ms 3.144 ms
5 218.100.48.143 (218.100.48.143) [*] 2.199 ms 2.181 ms 2.174 ms
6 * 182.79.220.182 (182.79.220.182) [*] 1.683 ms 1.802 ms
7 a125-252-226-97.deploy.akamaitechnologies.com (125.252.226.97) [AS9498] 1.821 ms 1.775 ms 1.947 ms

An interesting case here is that NTT owns majority stake in Netmagic datacenter where this server is located! But likely they can’t do much since they need a license in India to offer their own network in Netmagic or simply peer more? 🙂

 

This is how I increased my download speed from 512Kbps to 24Mbps! 😉 

19 Jan

Tata Communications – NTT routing issue for Akamai

Interestingly routing issues didn’t spare one of top CDN provider – Akamai!

 

So what’s wrong?

(from my BSNL connection):

PING akamai.com (61.213.189.49) 56(84) bytes of data.
64 bytes from 61.213.189.49: icmp_req=1 ttl=52 time=492 ms
64 bytes from 61.213.189.49: icmp_req=2 ttl=52 time=492 ms
64 bytes from 61.213.189.49: icmp_req=3 ttl=52 time=474 ms
64 bytes from 61.213.189.49: icmp_req=4 ttl=51 time=492 ms
64 bytes from 61.213.189.49: icmp_req=5 ttl=51 time=489 ms

— akamai.com ping statistics —
5 packets transmitted, 5 received, 0% packet loss, time 22236ms
rtt min/avg/max/mdev = 474.296/488.469/492.837/7.183 ms

 

~ 500ms is way too high. Even US is at like 300ms latency.

 

Looking at traceroute: 

[DDET traceroute from BSNL – Click to expand ]

traceroute to akamai.com (61.213.189.49), 30 hops max, 60 byte packets
1 router.local (192.168.1.1) [AS8151/AS28513] 4.223 ms 4.979 ms 5.879 ms
2 117.200.48.1 (117.200.48.1) [AS9829] 45.241 ms 46.384 ms 52.839 ms
3 218.248.173.46 (218.248.173.46) [AS9829] 87.089 ms * *
4 115.114.57.165.static-Mumbai.vsnl.net.in (115.114.57.165) [AS4755] 74.675 ms 76.970 ms 80.856 ms
5 if-0-100.tcore2.MLV-Mumbai.as6453.net (180.87.39.25) [*] 83.234 ms 84.403 ms 87.742 ms
6 if-6-2.tcore1.L78-London.as6453.net (80.231.130.5) [AS6453] 230.777 ms 185.553 ms 194.288 ms
7 * Vlan704.icore1.LDN-London.as6453.net (80.231.130.10) [AS6453] 203.104 ms *
8 Vlan522.icore1.LDN-London.as6453.net (195.219.83.22) [AS6453] 308.973 ms 310.324 ms 311.038 ms
9 ae-4.r23.londen03.uk.bb.gin.ntt.net (129.250.5.40) [AS2914] 311.799 ms 333.841 ms 313.348 ms
10 as-0.r22.osakjp01.jp.bb.gin.ntt.net (129.250.5.35) [AS2914] 499.075 ms 501.158 ms 512.657 ms
11 ae-5.r24.tokyjp01.jp.bb.gin.ntt.net (129.250.3.221) [AS2914] 484.258 ms 485.401 ms 499.039 ms
12 * * *
13 xe-2-3.a17.tokyjp01.jp.ra.gin.ntt.net (61.213.169.214) [AS2914] 488.807 ms 489.543 ms 495.396 ms
14 61.213.189.49 (61.213.189.49) [AS2914] 506.170 ms 501.504 ms 507.296 ms

 [/DDET]

So route is like Mumbai (India) – London (UK) – Tokyo (Japan). 

 

It’s not really issue with BSNL at this point but with Tata Communications AS6453. Looking at traceroute from Tata Comm’s Mumbai node using their looking glass.

[DDET Click to expand traceroute from Tata ]

Router: gin-mlv-core1
Site: IN, Mumbai – MLV, VSNL
Command: traceroute ip 61.213.189.49

Tracing the route to 61.213.189.49

1 if-3-0-0.core2.MLV-Mumbai.as6453.net (209.58.105.82) [MPLS: Label 512 Exp 0] 0 msec
if-2-1-0-0.tcore1.MLV-Mumbai.as6453.net (180.87.38.21) [MPLS: Label 664353 Exp 0] 132 msec
if-3-1-0-0.tcore1.MLV-Mumbai.as6453.net (180.87.38.10) [MPLS: Label 664353 Exp 0] 140 msec
2 if-2-2.tcore2.MLV-Mumbai.as6453.net (180.87.38.2) [MPLS: Label 686484 Exp 0] 128 msec
if-10-3-1-0.tcore2.MLV-Mumbai.as6453.net (180.87.39.61) [MPLS: Label 686484 Exp 0] 136 msec 136 msec
3 if-6-2.tcore1.L78-London.as6453.net (80.231.130.5) 116 msec 128 msec 116 msec
4 *
Vlan704.icore1.LDN-London.as6453.net (80.231.130.10) 204 msec *
5 Vlan522.icore1.LDN-London.as6453.net (195.219.83.22) 132 msec 136 msec 136 msec
6 ae-4.r23.londen03.uk.bb.gin.ntt.net (129.250.5.40) [AS 2914] [MPLS: Label 302128 Exp 0] 128 msec 128 msec 136 msec
7 as-0.r22.osakjp01.jp.bb.gin.ntt.net (129.250.5.35) [AS 2914] [MPLS: Label 643206 Exp 0] 364 msec 364 msec 408 msec
8 ae-5.r24.tokyjp01.jp.bb.gin.ntt.net (129.250.3.221) [AS 2914] 376 msec 372 msec 368 msec
9 xe-3-1.a17.tokyjp01.jp.ra.gin.ntt.net (61.213.169.150) [AS 2914] 372 msec 380 msec 388 msec
10 xe-2-3.a17.tokyjp01.jp.ra.gin.ntt.net (61.213.169.214) [AS 2914] 480 msec 516 msec 400 msec
11 61.213.189.49 [AS 2914] 404 msec 388 msec 384 msec

 [/DDET]

Clearly identical traceroute and poor routing between Tata AS6453 and NTT AS2914.

 

Looking at route from Reliance Flagtel Mumbai node:

[DDET Click to expand traceroute from Reliance]

You requested a Traceroute from Mumbai (62.216.135.130) to 61.213.189.49
1 ge-0-2-0.0.pjr02.mmb004.flagtel.com (85.95.25.70) 131.460 ms 131.710 ms 131.509 ms
MPLS Label=422848 CoS=0 TTL=1 S=1
2 so-1-3-0.0.pjr02.hkg005.flagtel.com (85.95.25.118) 131.409 ms 131.465 ms 131.679 ms
MPLS Label=373792 CoS=0 TTL=1 S=1
3 so-0-2-0.0.pjr02.wad001.flagtel.com (85.95.25.189) 133.951 ms 129.679 ms 129.712 ms
MPLS Label=346448 CoS=0 TTL=1 S=1
4 so-7-1-0.0.cjr04.tok002.flagtel.com (85.95.25.202) 131.280 ms 131.285 ms 131.210 ms
5 80.81.64.74 (80.81.64.74) 132.028 ms 131.745 ms 131.768 ms
6 a60-254-153-245.deploy.akamaitechnologies.com (60.254.153.245) [AS 20940] 132.706 ms 133.643 ms 134.960 ms
7 61.213.189.49 (61.213.189.49) [AS 20940] 132.558 ms 132.508 ms 132.577 ms

[/DDET]

Good – direct route. No issues on Reliance network.

 

Unfortunately I can’t test with Airtel since their looking glass seems not working at all right now and I am already busy with exams to wait & try again. (if someone from Airtel reading this post, please please & please setup a good looking glass for AS9498).

 

Quick check on what’s wrong with Tata.
I tried looking at route from their Tokyo node, and here’s output:

[DDET Click to expand traceroute]

Router: gin-tv2-core1
Site: JP, Tokyo – TV2, EQUINIX
Command: traceroute ip 61.213.189.49

Tracing the route to 61.213.189.49

1 if-0-2-1-40.tcore1.TV2-Tokyo.as6453.net (180.87.180.41) [MPLS: Label 316726 Exp 0] 108 msec
if-8-2-1-38.tcore1.TV2-Tokyo.as6453.net (209.58.61.118) [MPLS: Label 316726 Exp 0] 108 msec
if-0-2-1-40.tcore1.TV2-Tokyo.as6453.net (180.87.180.41) [MPLS: Label 316726 Exp 0] 112 msec
2 if-9-2.tcore2.PDI-PaloAlto.as6453.net (180.87.180.17) 104 msec 108 msec 104 msec
3 Vlan3254.icore1.SQN-SanJose.as6453.net (66.198.144.6) 116 msec 108 msec 108 msec
4 Vlan507.icore1.SQN-SanJose.as6453.net (209.58.116.22) 108 msec 108 msec 108 msec
5 ae-7.r21.snjsca04.us.bb.gin.ntt.net (129.250.5.54) [AS 2914] [MPLS: Label 310512 Exp 0] 108 msec 104 msec
ae-6.r20.snjsca04.us.bb.gin.ntt.net (129.250.5.12) [AS 2914] [MPLS: Label 301136 Exp 0] 108 msec
6 as-2.r20.tokyjp01.jp.bb.gin.ntt.net (129.250.2.35) [AS 2914] [MPLS: Label 574335 Exp 0] 240 msec
as-0.r21.tokyjp01.jp.bb.gin.ntt.net (129.250.4.45) [AS 2914] [MPLS: Label 470118 Exp 0] 216 msec
as-2.r20.tokyjp01.jp.bb.gin.ntt.net (129.250.2.35) [AS 2914] [MPLS: Label 574335 Exp 0] 220 msec
7 ae-1.r25.tokyjp01.jp.bb.gin.ntt.net (129.250.2.73) [AS 2914] 228 msec
ae-1.r24.tokyjp01.jp.bb.gin.ntt.net (129.250.2.21) [AS 2914] 308 msec
ae-1.r25.tokyjp01.jp.bb.gin.ntt.net (129.250.2.73) [AS 2914] 296 msec
8 xe-3-1.a17.tokyjp01.jp.ra.gin.ntt.net (61.213.169.150) [AS 2914] 228 msec
xe-4-1.a17.tokyjp01.jp.ra.gin.ntt.net (61.213.169.154) [AS 2914] 212 msec
xe-3-1.a17.tokyjp01.jp.ra.gin.ntt.net (61.213.169.150) [AS 2914] 228 msec
9 xe-2-3.a17.tokyjp01.jp.ra.gin.ntt.net (61.213.169.214) [AS 2914] 204 msec 256 msec 212 msec
10 61.213.189.49 [AS 2914] 220 msec 212 msec 216 msec

[/DDET]

So route is like Tokyo (Japan) – San Jose (US) – Packet handover to NTT – (back to) Tokyo (Japan). It seems Tata AS6453 is exchanging traffic with NTT AS2914 in London and California but not in Japan itself.

 

Looking at BGP table for Akamai IP from Tata AS6453 Mumbai node:

[DDET Click to expand BGP output]

Router: gin-mlv-core1
Site: IN, Mumbai – MLV, VSNL
Command: show ip bgp 61.213.189.49

BGP routing table entry for 61.213.160.0/19
Bestpath Modifiers: deterministic-med
Paths: (3 available, best #1)
Multipath: eBGP
2914
ldn-icore1. (metric 2991) from cxr-tcore1. (66.110.10.113)
Origin IGP, valid, internal, best
Community:
Originator: ldn-icore1.
2914
ldn-icore1. (metric 2991) from mlv-tcore2. (66.110.10.215)
Origin IGP, valid, internal
Community:
Originator: ldn-icore1.
2914
ldn-icore1. (metric 2991) from mlv-tcore1. (66.110.10.202)
Origin IGP, valid, internal
Community:
Originator: ldn-icore1.

[/DDET]

Clearly – all routes to London. No route to Singapore or Japan directly.

 

Next, check on Japan node BGP table: 

[DDET Click to expand BGP table output]

Router: gin-tv2-core1
Site: JP, Tokyo – TV2, EQUINIX
Command: show ip bgp 61.213.189.49

BGP routing table entry for 61.213.160.0/19
Bestpath Modifiers: deterministic-med
Paths: (9 available, best #9)
2914
lvw-tcore2. (metric 3329) from svq-core1. (svq-core1.)
Origin IGP, valid, internal
Community:
Originator: 66.110.10.248
2914
av2-tcore2. (metric 6417) from ad1-core1. (ad1-core1.)
Origin IGP, valid, internal
Community:
Originator: 66.110.10.88
2914
lvw-tcore2. (metric 3329) from laa-mcore3. (laa-mcore3.)
Origin IGP, valid, internal
Community:
Originator: 66.110.10.248
2914
ldn-icore1. (metric 6408) from l78-mcore3. (Loopback5.mcore3.L78-London.)
Origin IGP, valid, internal
Community:
Originator: ldn-icore1.
2914
pvu-tcore1. (metric 6396) from pye-core1. (pye-core1.)
Origin IGP, valid, internal
Community:
Originator: 66.110.10.90
2914
nto-icore1. (metric 3405) from nyy-mcore4. (nyy-mcore4.)
Origin IGP, valid, internal
Community:
Originator: nto-icore1.
2914
dt8-tcore2. (metric 3346) from dtx-core1. (dtx-core1.)
Origin IGP, valid, internal
Community:
Originator: Loopback5.bb1.LAU-Laurentides.as6453.net.
2914
ct8-tcore1. (metric 3364) from ct8-core1. (ct8-core1.)
Origin IGP, valid, internal
Community:
Originator: 66.110.11.16
2914
sqn-icore1. (metric 3318) from pdi-mcore4. (pdi-mcore4.)
Origin IGP, valid, internal, best
Community:
Originator: sqn-icore1.

 [/DDET]

9 routes – London, New York, California but no direct route. Tata’s nodes are not listening to any announcement for that block in Tokyo directly.
It’s not that just Tata is not handling packets to NTT in Japan but reverse is also true. Looking at route from NTT Japan node to Tata’s Japan node:

 

NTT Tokyo to Tata Tokyo node: 

[DDET Click to expand traceroute]

Query Results:
Router: Tokyo – JP
Command: traceroute 209.58.61.118

Disclaimer: Traceroute is a useful tool for determining the route a packet takes, but it should not be used as an accurate measure of network performance. For more information please view the Traceroute Disclaimer.

traceroute to 209.58.61.118 (209.58.61.118), 30 hops max, 40 byte packets
1 ae-0.r21.tokyjp01.jp.bb.gin.ntt.net (129.250.2.72) 35.957 ms 13.354 ms 30.162 ms
MPLS Label=433510 CoS=0 TTL=1 S=0
MPLS Label=422560 CoS=0 TTL=1 S=1
2 as-0.r21.lsanca03.us.bb.gin.ntt.net (129.250.3.145) 133.142 ms ae-7.r21.osakjp01.jp.bb.gin.ntt.net (129.250.2.70) 35.097 ms 9.023 ms
MPLS Label=631431 CoS=0 TTL=1 S=0
MPLS Label=303728 CoS=0 TTL=2 S=1
3 ae-0.r20.lsanca03.us.bb.gin.ntt.net (129.250.3.33) 139.183 ms 121.634 ms 140.311 ms
MPLS Label=306720 CoS=0 TTL=1 S=1
4 ix-11-2-1-0.tcore2.LVW-LosAngeles.as6453.net (64.86.252.65) 109.416 ms ix-9-1-2-0.tcore2.LVW-LosAngeles.as6453.net (216.6.84.65) 147.333 ms ae-1.r05.lsanca03.us.bb.gin.ntt.net (129.250.2.230) 126.940 ms
5 Vlan507.icore1.SQN-SanJose.as6453.net (209.58.116.21) 152.854 ms 132.124 ms 146.938 ms
6 if-2-2.tcore1.LVW-LosAngeles.as6453.net (66.110.59.1) 172.250 ms if-4-2.tcore1.PDI-PaloAlto.as6453.net (66.198.127.9) 121.056 ms 112.809 ms
7 if-8-2-1-38.tcore1.TV2-Tokyo.as6453.net (209.58.61.118) 223.186 ms if-4-2.tcore1.PDI-PaloAlto.as6453.net (66.198.127.9) 108.956 ms if-8-2-1-38.tcore1.TV2-Tokyo.as6453.net (209.58.61.118) 244.013 ms

{master}

 [/DDET]

Clearly same  – a round trip to US!

Overall this is bit surprising for me since both Tata AS6453 and NTT AS2914 are considered as Tier 1 backbone players and they usually exchange traffic in settlement free peering. In this case  Tata & NTT are peering but not in Japan (why?). No clear ideas on why. Will try finding out after my exams! 🙂

 

With hope that you are reaching my blog without having a round trip across globe 😉 time for me to get back on cramming for college exams!