30 Jul

Welcome Amazon AWS AS16509 to India!

Today I spotted some routes from Amazon AWS Cloud services –  AS16509 in Indian tables. AS16509 was originating prefixes while sitting in downstream of Tata-VSNL AS4755 and Reliance AS18101. I almost missed Amazon AWS’s announcement on their blog about Indian PoPs for their DNS service – Route53 and CDN service – Cloudfront.

 

New PoP’s of Amazon in India are at Mumbai and Chennai and I see pretty much consistent BGP announcements to Tata and Reliance from these locations. Prefixes I have seen so far:

 

Unicast prefixes originated in India (for Cloufront CDN):

54.230.172.0/22
54.230.188.0/22
54.239.160.0/22
54.239.188.0/22

 

Anycast prefixes (for anycasted DNS route 53)

205.251.192.0/23
205.251.198.0/23

 

Note: I pulled these prefixes by looking at upstream peers in India (which is Tata and Reliance) and running simply sh ip bgp regexp 4755 16509sh ip bgp regexp 18101 16509 on Oregon routeviews & few other major data collection points of global IPv4 table. 

I can’t see any upstream from Airtel AS9498 or any other major Indian telco. Also at NIXI prefixes are available partially. I see prefiex at NIXI Mumbai carried by Tata VSNL. At NIXI Chennai prefixes are present with one degree prepend (AS4755 AS4755 twice) making route less preferable. While at NIXI Delhi there seems no route at all for Amazon’s prefixes (Tata follows regional route policy at NIXI). 

 

So now big question here – which datacenter is that? 

I doubt it would be Tata or Reliance since they are core competitiors and run datacenters pretty much on their own networks with almost zero carrier neutral options (few exceptions are there). My strong guess is that it’s Netmagic’s datacenter in Mumbai and Chennai with direct upstream links (bypassing Netmagic’s network). Just my guess. Cannot verify it from record of AS16509 on peeringdb.net – http://www.peeringdb.com/view.php?asn=16509

 

With that being said here’s a trace to cdn.anuragbhatia.com (which I use via Amazon Cloudfront):

Anurags-MacBook-Pro:~ anurag$ traceroute -a cdn.anuragbhatia.com
traceroute: Warning: cdn.anuragbhatia.com has multiple addresses; using 54.230.189.204
traceroute to ddlfp4nmkhyfr.cloudfront.net (54.230.189.204), 64 hops max, 52 byte packets
1 [AS1] 10.0.0.1 (10.0.0.1) 1.152 ms 0.765 ms 0.627 ms
2 [AS10223] 192.168.1.1 (192.168.1.1) 1.460 ms 2.906 ms 1.569 ms
3 [AS9829] 117.218.197.1 (117.218.197.1) 16.339 ms 17.905 ms 15.704 ms
4 [AS9829] 218.248.169.118 (218.248.169.118) 94.835 ms 29.628 ms 118.135 ms
5 [AS4755] 115.114.89.21.static-mumbai.vsnl.net.in (115.114.89.21) 60.472 ms 61.304 ms 59.103 ms
6 [AS0] 172.31.19.245 (172.31.19.245) 84.706 ms 87.201 ms 85.640 ms
7 [AS4755] 115.114.130.126.static-chennai.vsnl.net.in (115.114.130.126) 82.327 ms 83.276 ms 81.583 ms
8 [AS16509] server-54-230-189-204.maa3.r.cloudfront.net (54.230.189.204) 85.261 ms 85.185 ms 84.269 ms
Anurags-MacBook-Pro:~ anurag$

 

Always nice to maa in all these nodes at Chennai. Basically most of companies (including Google) use 3 digit airport code in name of node (in rDNS PTR record of router’s WAN IP). For Chennai (which used to be known as Madras) airport code is still MAA and this is why you will see maa in Chennai nodes and BOM on Mumbai based nodes. 🙂

 

Time to get back to work. Have a good week ahead! 🙂

10 Apr

BSNL routing tables screw up

It has been super boring evening considering my sessional tests tomorrow. Test time is dull as always. I have been precisely measnuring latency on BSNL link from BSNL Haryana to Singapore based servers. The fluctuation in latency is pretty much common now. Someones we get 120ms latency to Singapore (an expected number based on distance) while other time it goes off as high as 310ms. Latency with openDNS nodes in Singapore makes it pretty much poor to use openDNS here.
 
Based on my collected data and BGPlay’s routing records, here’s what’s happening. My IP is coming /20 BGP annoucement from BSNL Autonomous System 9829 – 117.207.48.0/20. Looking at BGP table records for that block from BGPlay’s routing data archive source.
 
On Sunday Morning 00:00am UTC, BSNL was found to be announcing 117.207.48.0/20 from AS9829 which was carried over via it’s upstream ISPs – Tata Communications (AS6453) and Reliance Globalcom (AS18101). From Tata’s AS6453 most of other Tier 1 backbones and many other small ISPs were getting routes. Similarly in case of Reliance, AS18101 was announcing blocks to it’s other network FLAG Telecom AS15412 which was further passing routes across many ISPs in the world. ISPs like Tinet, AT&T, Savvis, Seabone, Sprint etc were getting announcement via AS6453 while Level3, NTT, Hurricane Electric, Swisscom & many others were getting annoucements via Reliance-FLAG backbone.

At this instance – connectivity to network via Reliance route looks pretty good. All traffic comes via direct path – entering Reliance’s network from nearest point & next reaching BSNL. While at the same time, unfortunately in case of other path via Tata Communications – it seems like no matter where traffic orignates from, it is always routed via US. That is even if traffic is orignating from Singapore, it is being routed to India via US.
 

Quick check on Tata Communications AS6453 PoPs at this instant:

Router: gin-svq-core1
Site: SG, Singapore – SVQ, EQUNIX
Command: show ip bgp 117.207.48.0/20
BGP routing table entry for 117.207.48.0/20
Bestpath Modifiers: deterministic-med
Paths: (3 available, best #2)
Multipath: eBGP
9829
nyy-mcore4. (metric 3777) from tv2-core1. (tv2-core1.)
Origin IGP, valid, internal
Community:
Originator: nyy-mcore4.
9829
nyy-mcore4. (metric 3777) from hk2-core3. (hk2-core3.)
Origin IGP, valid, internal, best
Community:
Originator: nyy-mcore4.
9829
nyy-mcore4. (metric 3777) from s9r-core1. (s9r-core1.)
Origin IGP, valid, internal
Community:
Originator: nyy-mcore4.
 
Router: gin-lhx-core1
Site: GB, London – LHX, TATA COMM. HARBOR EXCHANGE
Command: show ip bgp 117.207.48.0/20
BGP routing table entry for 117.207.48.0/20
Bestpath Modifiers: deterministic-med
Paths: (2 available, best #1)
Multipath: eBGP
9829
nyy-mcore4. (metric 3036) from ldn-mcore3. (ldn-mcore3.)
Origin IGP, valid, internal, best
Community:
Originator: nyy-mcore4.
9829
nyy-mcore4. (metric 3036) from l78-tcore1. (66.110.10.237)
Origin IGP, valid, internal
Community:
Originator: nyy-mcore4.
 
 
Thus we can see in both cases – Tata’s router is getting updates from nyy-mcore4 which is their router in New York city. This forces packets all way down to New York before they are routed back in Asia to BSNL.
 
Next, same thing goes for next few hours. On Sunday noon time 12:55pm, we can see a path change from multiple providers like Savvis AS3561. We can see change in routing table and now between Tata AS6453 and BSNL AS9829, a new network comes in. It is AS4755 which is Tata Communications other network VSNL. Tata still uses AS4755 in India and AS6453 everywhere else.

Within next few seconds, we can see similar changes from AS293 – ESnet,
Path Change from 3561 6453 9829
to 3561 6453 4755 9829
Path Change from 293 6453 9829
to 293 6453 4755 9829
Path Change from 3303 15412 18101 9829
to 3303 6453 4755 9829
Path Change from  812 6453 9829
to 812 6453 4755 9829
Path Change from 3257 6453 9829
to 3257 6453 4755 9829
Path Change from 3130 1239 6453 9829
to 3130 1239 6453 4755 9829 
Path Change from 7018 6453 9829
to 7018 6453 4755 9829 
Path Change from 1299 6453 9829
to 1299 6453 4755 9829 
Within a min, we can see half of routes are going via AS4755 rather then AS6453 directly.
Next, we see a route withdrawal – 701 6453 9829 followed by route re-annoucement 701 6453 4755 9829 and with this we see whole block being announced via AS4755 (which exists only in India as per as I know). Next on 12:58pm, we see another series of path changes all reversing back to AS6453 – AS9829 skipping AS4755 in between. On 12:50pm we can see half of routes being back with AS6453 link directly and half still with AS6453. Very soon all routes get back on direct link AS4755 goes out of picture again.

Summary of what’s happening:

  1. BSNL’s network is having high latency on various routes including Singapore and Europe.
  2. For specific block in testing, we can see BSNL is announcing them via Tata Communications & Reliance Globalcom.
  3. Tata Communications uses two autonomous systems – AS4755 (VSNL) for Indian operations & AS6453 for everywhere else.
  4. We can see routing table changes almost daily which bring AS4755 between AS9829 and AS6453 on random basis.
  5. AS4755 is believed to be operated only in India and I can see whenever AS4755 is coming in picture, packets to BSNL-AS9829 are handed off directly and latency is pretty good.
  6. For other times when we have AS6453 > AS9829 routing, we can see block is being annouced from AS6453 in New York.
  7. Very likely problem is NOT from Tata Communications end, but rather from BSNL’s end. BSNL is constantly switching annoucing peers – AS4755 at one end in India while AS6453 router on other end in New York. Though it is hard to confirm this speculation.

 
 
 
Glad Airlines don’t route flights in the manner in which wrong routing goes! 🙂