Tag Archives: Netmagic

25 Nov

Peering with content networks in India

peering

One of frequent email and contact form message I get my blog is about available content networks in India and where one can peer. There are certain content networks in India and of course most of the content networks have open peering policy and are usually happy with direct inter-connection (we call as “peering“) with the ISP networks (often referred to as “eyeball networks”). Some of these networks have a backbone which connects back to their key datacenter locations on their own circuits via Singapore/Europe, some other have simply placed their caching server where cache fill happens over IP transit.

 

Based on publically known information across community and of course peeringdb, following content players are available in India and known to be open for peering:

  1. Google
  2. Microsoft
  3. Amazon
  4. Limelight

 

A quick list of these with datacenter names and locations as taken from Peeringdb record of these networks.

Organisation ASN City Datacenter Location
Amazon 16509 Mumbai GPX Mumbai Unit A-001, Boomerang Chandivali Farm Road, Near Chandivali Studio, Andheri East Mumbai, Mumbai, 400 051
Amazon 16509 Noida Sify Greenfort – Noida B7, Block A, Sector 132, Noida Expressway, Noida , UP 201304
Amazon 16509 Mumbai Tata Mumbai IDC LVSB, Opposite Kirti College
6th floor, Prabahdevi
Mumbai, MH, 400 028
Google 15169 Chennai Bharti Airtel Santhome Bharti Towers, 101 Santhome High Road, Chennai, 600 028
Google 15169 Mumbai GPX Mumbai Unit A-001, Boomerang Chandivali Farm Road, Near Chandivali Studio, Andheri East Mumbai, Mumbai, 400 051
Google 15169 Noida Sify Greenfort – Noida B7, Block A, Sector 132, Noida Expressway, Noida , UP 201304
Google 15169 Chennai TATA Communications Ltd 14th floor, 2nd block
4, Swami Sivanand Salai, Chennai, TN 600 002
Google 15169 Delhi Tata Delhi VSB, Bangla Sahib Road, New Delhi 110001
Google 15169 Mumbai Tata Mumbai IDC LVSB, Opposite Kirti College
6th floor, Prabhadevi
Mumbai, MH, 400 028
Limelight 55439 / 22822 Chennai Bharti Airtel Santhome Bharti Towers, 101 Santhome High Road, Chennai, 600 028
Limelight 55439 / 22822 Mumbai Netmagic Vikhroli Mehra Industrial Estate
LBS Marg, Vikhroli
Mumbai, 400 079
Microsoft 8075 Mumbai Bharti Airtel Mumbai Plot No, TPS-2, 14/3, 2nd floor
Dattatray Road, Linking Road Extension
Mumbai, 400054
Microsoft 8075 Chennai Bharti Airtel Santhome Bharti Towers, 101 Santhome High Road, Chennai, 600 028
Microsoft 8075 Chennai TATA Communications Ltd 14th floor, 2nd block
4, Swami Sivanand Salai, Chennai, TN 600 002
Microsoft 8075 Delhi Tata Communications Ltd – GK1 Greater Kailash-1
New Delhi, 110048
Microsoft 8075 Mumbai Tata Mumbai IDC LVSB, Opposite Kirti College
6th floor, Prabhadevi
Mumbai, MH, 400 028

 

Besides these Google also has an option of GGC, Akamai has an option of Akamai Caching server, Facebook has the option for caching server which is hosted inside ISP’s network and Netflix has an option for OCAs. Besides these networks there are known nodes of Verizon’s Edgecast in Delhi, Mumbai & Chennai (as per this map), Cloudflare has nodes in Delhi, Mumbai & Chennai (as per this map), PCH & K-root server have a node with Web Werks available on MCH peering fabric and Dyn has a node in Mumbai (as per this map).

Go ahead and peer as after all it all starts with a handshake. 🙂

01 Apr

Why NIXI AS24029 appears to be transit ASN?

And my post on 1st April. Don’t take it as April fool post 😉

 

Multiple times NIXI’s AS24029 has been reported as acting like transit ASN for multiple networks. I have analysed it in past and this is very much because of route leaks by few specific networks. I have explained difference in peering Vs transit routes and their handling previously on my blog.

In short: A network is supposed to re-announce it’s peering and transit routes only to customer and not to any other peer or upstream. Whenever NIXI’s ASN appears in global routing table, its always the case where one or more networks are re-announcing routes learnt via NIXI to their upstream transits. 

 

Looking at Hurricane Electric’s bgp.he.net for NIXI’s AS24029, we get:

 

 

Now according to this – many ASNs are peers of NIXI and visible to HE. The problem with HE’s data is that it doesn’t shows who is downstream and who is upstream (but is pretty fast!). Looking at stat.ripe.net data for AS24029, we get:

peers_of_AS24029

 

This is very interesting data as left side are the ones which are actually announcing these routes to their upstreams. Finding  upstream is tricky since these are filtered out at global level are don’t stay in the global routing table. It would be overall hard to find ones whose path count is low but for ones with large path count, we can likely see those routes in RIPE RIS collected data. 

Using bgpdump on RIPE RIS data, I get:

anurag@server7:~/test$ grep "24029" latest.txt 
TABLE_DUMP_V2|03/31/14 16:00:05|A|68.67.63.245|22652|103.27.140.0/22|22652 3356 55410 {18101,24029,45804,132777}|IGP
TABLE_DUMP_V2|03/31/14 16:00:05|A|46.149.194.2|15469|103.27.140.0/22|15469 6762 3356 55410 {18101,24029,45804,132777}|IGP
TABLE_DUMP_V2|03/31/14 16:00:05|A|193.160.39.1|57821|103.27.140.0/22|57821 48039 3320 3356 55410 {18101,24029,45804,132777}|IGP
TABLE_DUMP_V2|03/31/14 16:00:05|A|79.143.241.12|29608|103.27.140.0/22|29608 3356 55410 {18101,24029,45804,132777}|IGP
TABLE_DUMP_V2|03/31/14 16:00:05|A|203.119.76.5|4608|103.27.140.0/22|4608 1221 4637 1273 55410 {18101,24029,45804,132777}|IGP
TABLE_DUMP_V2|03/31/14 16:00:05|A|46.149.194.1|15469|103.27.140.0/22|15469 6762 3356 55410 {18101,24029,45804,132777}|IGP
TABLE_DUMP_V2|03/31/14 16:00:05|A|193.150.22.1|57381|103.27.140.0/22|57381 42708 3356 55410 {18101,24029,45804,132777}|IGP
TABLE_DUMP_V2|03/31/14 16:00:05|A|202.12.28.1|4777|103.27.140.0/22|4777 2500 2500 2500 7660 4635 1273 55410 {18101,24029,45804,132777}|IGP
TABLE_DUMP_V2|03/31/14 16:00:05|A|208.51.134.248|3549|103.27.140.0/22|3549 3356 55410 {18101,24029,45804,132777}|IGP
TABLE_DUMP_V2|03/31/14 16:00:05|A|203.123.48.6|37989|103.27.140.0/22|37989 4844 2914 3491 55410 {18101,24029,45804,132777}|IGP
TABLE_DUMP_V2|03/31/14 16:00:05|A|213.200.87.254|3257|103.27.140.0/22|3257 3356 55410 {18101,24029,45804,132777}|IGP
TABLE_DUMP_V2|03/31/14 16:00:05|A|218.189.6.2|9304|103.27.140.0/22|9304 4635 1273 55410 {18101,24029,45804,132777}|IGP
TABLE_DUMP_V2|03/31/14 16:00:05|A|12.0.1.63|7018|103.27.140.0/22|7018 3356 55410 {18101,24029,45804,132777}|IGP
TABLE_DUMP_V2|03/31/14 16:00:05|A|212.25.27.44|8758|103.27.140.0/22|8758 3356 55410 {18101,24029,45804,132777}|IGP
TABLE_DUMP_V2|03/31/14 16:00:05|A|176.12.110.8|50300|103.27.140.0/22|50300 3356 55410 {18101,24029,45804,132777}|IGP
TABLE_DUMP_V2|03/31/14 16:00:05|A|193.0.0.56|3333|103.27.140.0/22|3333 3356 55410 {18101,24029,45804,132777}|IGP
TABLE_DUMP_V2|03/31/14 16:00:05|A|195.47.235.100|6881|103.27.140.0/22|6881 15685 6939 1273 55410 {18101,24029,45804,132777}|IGP
TABLE_DUMP_V2|03/31/14 16:00:05|A|208.51.134.248|3549|103.245.200.0/24|3549 6453 4755 17439 24029 9498 132542|IGP
TABLE_DUMP_V2|03/31/14 16:00:05|A|12.0.1.63|7018|103.245.200.0/24|7018 6453 4755 17439 24029 9498 132542|IGP
TABLE_DUMP_V2|03/31/14 16:00:06|A|46.149.194.2|15469|116.214.114.0/24|15469 6762 9498 17447 24029 4755 24063|IGP
TABLE_DUMP_V2|03/31/14 16:00:06|A|46.149.194.1|15469|116.214.114.0/24|15469 6762 9498 17447 24029 4755 24063|IGP
TABLE_DUMP_V2|03/31/14 16:00:06|A|193.150.22.1|57381|116.214.114.0/24|57381 42708 9498 17447 24029 4755 24063|IGP
TABLE_DUMP_V2|03/31/14 16:00:06|A|176.12.110.8|50300|116.214.114.0/24|50300 9498 17447 24029 4755 24063|IGP
TABLE_DUMP_V2|03/31/14 16:00:06|A|203.123.48.6|37989|116.214.114.0/24|37989 4844 9498 17447 24029 4755 24063|IGP
TABLE_DUMP_V2|03/31/14 16:00:06|A|213.200.87.254|3257|116.214.114.0/24|3257 9498 17447 24029 4755 24063|IGP
TABLE_DUMP_V2|03/31/14 16:00:06|A|195.47.235.100|6881|116.214.114.0/24|6881 3257 9498 17447 24029 4755 24063|IGP
TABLE_DUMP_V2|03/31/14 16:00:06|A|208.51.134.248|3549|116.214.114.0/24|3549 7473 9498 17447 24029 4755 24063|IGP
TABLE_DUMP_V2|03/31/14 16:00:06|A|46.149.194.2|15469|123.136.159.0/24|15469 6762 9498 45528 45528 45528 45528 24029 18101 18101 9396|IGP
TABLE_DUMP_V2|03/31/14 16:00:06|A|46.149.194.1|15469|123.136.159.0/24|15469 6762 9498 45528 45528 45528 45528 24029 18101 18101 9396|IGP
TABLE_DUMP_V2|03/31/14 16:00:06|A|193.150.22.1|57381|123.136.159.0/24|57381 42708 9498 45528 45528 45528 45528 24029 18101 18101 9396|IGP
TABLE_DUMP_V2|03/31/14 16:00:06|A|218.189.6.2|9304|123.136.159.0/24|9304 4635 9498 45528 45528 45528 45528 24029 18101 18101 9396|IGP
TABLE_DUMP_V2|03/31/14 16:00:06|A|203.123.48.6|37989|123.136.159.0/24|37989 4844 7473 9498 45528 45528 45528 45528 24029 18101 18101 9396|IGP
TABLE_DUMP_V2|03/31/14 16:00:06|A|208.51.134.248|3549|123.136.159.0/24|3549 9498 45528 45528 45528 45528 24029 18101 18101 9396|IGP
TABLE_DUMP_V2|03/31/14 16:00:06|A|202.12.28.1|4777|123.136.159.0/24|4777 2500 2500 2500 7660 4635 9498 45528 45528 45528 45528 24029 18101 18101 9396|IGP
TABLE_DUMP_V2|03/31/14 16:00:09|A|46.149.194.2|15469|182.48.209.0/24|15469 6762 9498 17443 4755 24029 9583 45769|IGP
TABLE_DUMP_V2|03/31/14 16:00:09|A|46.149.194.1|15469|182.48.209.0/24|15469 6762 9498 17443 4755 24029 9583 45769|IGP
TABLE_DUMP_V2|03/31/14 16:00:09|A|195.47.235.100|6881|182.48.209.0/24|6881 3257 9498 17443 4755 24029 9583 45769|IGP
TABLE_DUMP_V2|03/31/14 16:00:09|A|208.51.134.248|3549|182.48.209.0/24|3549 10026 9498 17443 4755 24029 9583 45769|IGP
TABLE_DUMP_V2|03/31/14 16:00:09|A|203.123.48.6|37989|182.48.209.0/24|37989 4844 7473 9498 17443 4755 24029 9583 45769|IGP
TABLE_DUMP_V2|03/31/14 16:00:09|A|202.12.28.1|4777|182.48.209.0/24|4777 2516 10026 9498 17443 4755 24029 9583 45769|IGP
TABLE_DUMP_V2|03/31/14 16:00:09|A|213.200.87.254|3257|182.48.209.0/24|3257 9498 17443 4755 24029 9583 45769|IGP
TABLE_DUMP_V2|03/31/14 16:00:09|A|202.12.28.1|4777|183.87.210.0/24|4777 2516 6453 4755 18229 24029 9583 45841|IGP
TABLE_DUMP_V2|03/31/14 16:00:09|A|202.12.28.1|4777|183.87.244.0/24|4777 2516 6453 4755 18229 24029 9583 45841|IGP
TABLE_DUMP_V2|03/31/14 16:00:09|A|202.12.28.1|4777|183.87.250.0/24|4777 2516 6453 4755 18229 24029 9583 45841|IGP
TABLE_DUMP_V2|03/31/14 16:00:12|A|193.150.22.1|57381|202.3.15.0/24|57381 42708 9583 24186 24029 4755 45927|IGP
TABLE_DUMP_V2|03/31/14 16:00:12|A|203.119.76.5|4608|202.140.137.0/24|4608 1221 4637 6453 4755 10201 10201 10201 10201 10201 10201 10201 24029 9583|IGP
TABLE_DUMP_V2|03/31/14 16:00:12|A|202.12.28.1|4777|202.140.137.0/24|4777 2516 6453 4755 10201 10201 10201 10201 10201 10201 10201 24029 9583|IGP
TABLE_DUMP_V2|03/31/14 16:00:12|A|208.51.134.248|3549|202.140.137.0/24|3549 6453 4755 10201 10201 10201 10201 10201 10201 10201 24029 9583|IGP
TABLE_DUMP_V2|03/31/14 16:00:13|A|46.149.194.2|15469|203.119.50.0/24|15469 6762 9498 24186 24029 18101 18101|IGP
TABLE_DUMP_V2|03/31/14 16:00:13|A|46.149.194.1|15469|203.119.50.0/24|15469 6762 9498 24186 24029 18101 18101|IGP
TABLE_DUMP_V2|03/31/14 16:00:13|A|193.150.22.1|57381|203.119.50.0/24|57381 42708 9498 24186 24029 18101 18101|IGP
TABLE_DUMP_V2|03/31/14 16:00:13|A|203.123.48.6|37989|203.119.50.0/24|37989 4844 9498 24186 24029 18101 18101|IGP
TABLE_DUMP_V2|03/31/14 16:00:13|A|208.51.134.248|3549|203.119.50.0/24|3549 9498 24186 24029 18101 18101|IGP
anurag@server7:~/test$

 

Refinding more of AS path part, we get:

anurag@server7:~/test$ grep "24029" latest.txt |awk -F '|' '{print $7}' |awk -F '24029' '{print $1}'|awk -F ' ' '{print $NF}' |sort -u
10201 - Aircel
17439 - Netmagic
17447 - Net4India
18229 - CtrlS Datacenter
24186 - RailTel India
45528 - Tikona Networks
4755 - Tata Comm/VSNL
{18101, - Reliance Comm
anurag@server7:~/test$

Here we get the culprit ASNs. 🙂

So why does this happens?

Mostly it happens due the way filters are controlled in routers. Most of networks open their filters with upstreams to announce their customer routes. Now if customer routes are received via NIXI, they are re-announced as well. So in many of these cases these networks have/had the origination ASN as customer. 

These are the prefixes which are causing this:

anurag@server7:~/test$ grep "24029" latest.txt |awk -F '|' '{print $6}'|sort -u
103.245.200.0/24
103.27.140.0/22
116.214.114.0/24
123.136.159.0/24
182.48.209.0/24
183.87.210.0/24
183.87.244.0/24
183.87.250.0/24
202.140.137.0/24
202.3.15.0/24
203.119.50.0/24
anurag@server7:~/test$

 

 So that’s all about NIXI route leaks. Wish NIXI becomes a International hub for traffic exchange between Europe/Middle East and East Asia and as per current policy it’s no where around promoting domestic traffic exchange let alone international one! 

 

Disclaimer: I work for an Indian ISP and all comments here are completely personal. In no way it reflects my employers view. 

19 Dec

Google Public DNS and Akamai issues in India

A quick blog post on a interesting issue coming up due to combined problem of CDN failure on Google Public DNS and bad Akamai performance due to Tata-NTT peering issue.

 I was trying Zembra mail since there’s no more free Google Apps edition and one of my friend asked me to basic email on his domain up. It was more or less a straight task by installing Zembra with decent GUI.

 

I downloaded it on my Europe based server and during installation realized it was for 64 bit and thus I turned my head to my other server in India.
I started download again and it was slow. DEAD SLOW!

 

Something like this:

root@server2:~# wget http://files2.zimbra.com/downloads/8.0.1_GA/zcs-8.0.1_GA_5438.UBUNTU12_64.20121105164409.tgz
–2012-12-18 14:02:59– http://files2.zimbra.com/downloads/8.0.1_GA/zcs-8.0.1_GA_5438.UBUNTU12_64.20121105164409.tgz
Resolving files2.zimbra.com (files2.zimbra.com)… 23.32.241.26, 125.56.200.51
Connecting to files2.zimbra.com (files2.zimbra.com)|23.32.241.26|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 701053545 (669M) [binary/octet-stream]
Saving to: `zcs-8.0.1_GA_5438.UBUNTU12_64.20121105164409.tgz.1′

0% [ ] 5,545,378 67.5K/s eta 2h 28m ^C
root@server2:~#

 

Would have taken 2hrs + on 512Kbps speed while server is on 100Mbps connection and I usually get 20Mbps or so for US/Europe based sources. Since I downloaded same 700MB file on Europe based server and it was quite fast 40Mbps+ while here just 512kbps.

 

I looked at route from Indian server and route was:

traceroute to 23.32.241.26 (23.32.241.26), 30 hops max, 60 byte packets
1 103.6.87.1 (103.6.87.1) [AS36236] 1.310 ms 1.426 ms 1.716 ms
2 180.179.33.245 (180.179.33.245) [AS17439/AS9584] 0.843 ms 0.951 ms 0.958 ms
3 180.179.37.93 (180.179.37.93) [AS17439] 0.761 ms 0.762 ms 0.750 ms
4 * * *
5 180.179.37.137 (180.179.37.137) [AS17439] 0.840 ms 0.938 ms 1.091 ms
6 59.163.105.170.static-chennai.vsnl.net.in (59.163.105.170) [AS4755] 4.441 ms 3.891 ms 3.848 ms
7 * * *
8 ix-0-100.tcore2.MLV-Mumbai.as6453.net (180.87.39.25) [*] 27.175 ms 27.178 ms 27.927 ms
9 if-6-2.tcore1.L78-London.as6453.net (80.231.130.5) [AS6453] 143.917 ms 145.977 ms if-2-2.tcore1.MLV-Mumbai.as6453.net (180.87.38.1) [*] 135.972 ms
10 if-9-5.tcore1.WYN-Marseille.as6453.net (80.231.217.17) [AS6453] 132.133 ms 133.902 ms 133.286 ms
11 if-8-1600.tcore1.PYE-Paris.as6453.net (80.231.217.6) [AS6453] 134.763 ms 133.217 ms 136.691 ms
12 if-2-2.tcore1.PVU-Paris.as6453.net (80.231.154.17) [AS6453] 134.674 ms 137.558 ms *
13 * * *
14 ae-1.r21.parsfr01.fr.bb.gin.ntt.net (129.250.2.224) [AS2914] 153.657 ms 155.550 ms 152.412 ms
15 as-4.r22.amstnl02.nl.bb.gin.ntt.net (129.250.3.84) [AS2914] 155.136 ms 151.697 ms as-0.r25.tokyjp01.jp.bb.gin.ntt.net (129.250.3.79) [AS2914] 383.668 ms
16 * * *
17 * xe-3-2.a16.tokyjp01.jp.ra.gin.ntt.net (203.105.72.78) [AS2914] 271.296 ms 270.202 ms
18 a23-32-241-26.deploy.akamaitechnologies.com (23.32.241.26) [AS20940] 280.674 ms 282.407 ms xe-3-2.a16.tokyjp01.jp.ra.gin.ntt.net (203.105.72.78) [AS2914] 263.254 ms

 

Akami CDN node in Japan and route via Europe!!

 

This poor performance case is result of multiple issues:

  1. Datacenter is not running own DNS server but instead replying on Google Public DNS 8.8.8.8 and 8.8.4.4 which has upstream/full table connectivity outside in East Asia and not really in India. Thus Google DNS resolvers always pass IP of nodes outside India in Japan or sometimes in Malaysia. I blogged about it in detail sometimes back here.
  2. Datacenter is picking Tata-VSNL AS4755 for upstream for this route (not Airtel or Reliance), and interesting enough Tata does NOT peers with NTT Communications in Japan. They peer everywhere else except home market of NTT which I guess will be case with other ISPs as well. Thus nearest point for Tata to handle traffic to NTT is Europe which we can see in traceroute. 
  3. Since IP belongs to Akamai Japan, it brings traffic back to Asia over NTT Asia’s network and eventually passes it to Akamai. Strange that Akamai has not fixed this problem from long time. They must be knowing this since I posted this problem in NANOG mailing list last year and also blogged about it here.

 

OK – the fix!

I can surely do better then waiting for 2 hours to download that package! 
I quickly installed BIND and since BIND runs as “recursive resolver” by default, I simply pointed /etc/resolv.conf to 127.0.0.1

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND — YOUR CHANGES WILL BE OVERWRITTEN
# nameserver 8.8.8.8
# nameserver 8.8.4.4

nameserver 127.0.0.1

 

OK – now running download again, let’s see how it works:

 

root@server2:~/tmp2# wget http://files2.zimbra.com/downloads/8.0.1_GA/zcs-8.0.1_GA_5438.UBUNTU12_64.20121105164409.tgz
–2012-12-18 14:31:55– http://files2.zimbra.com/downloads/8.0.1_GA/zcs-8.0.1_GA_5438.UBUNTU12_64.20121105164409.tgz
Resolving files2.zimbra.com (files2.zimbra.com)… 125.252.226.97, 125.252.226.106
Connecting to files2.zimbra.com (files2.zimbra.com)|125.252.226.97|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 701053545 (669M) [binary/octet-stream]
Saving to: `zcs-8.0.1_GA_5438.UBUNTU12_64.20121105164409.tgz’

100%[============================================================================================================>] 701,053,545 3.07M/s in 3m 45s

2012-12-18 14:35:43 (2.98 MB/s) – `zcs-8.0.1_GA_5438.UBUNTU12_64.20121105164409.tgz’ saved [701053545/701053545]

root@server2:~/tmp2# 

 

Fast? Yeah a lot!

 

How?

Simply doing a trace to destination this time takes me to: 

traceroute to 125.252.226.97 (125.252.226.97), 30 hops max, 60 byte packets
1 103.6.87.1 (103.6.87.1) [AS36236] 2.089 ms 2.060 ms 2.047 ms
2 180.179.33.245 (180.179.33.245) [AS17439/AS9584] 2.012 ms 2.002 ms 1.997 ms
3 180.179.37.89 (180.179.37.89) [AS17439] 1.940 ms 1.949 ms 1.937 ms
4 180.179.37.38 (180.179.37.38) [AS17439] 2.258 ms 2.668 ms 3.144 ms
5 218.100.48.143 (218.100.48.143) [*] 2.199 ms 2.181 ms 2.174 ms
6 * 182.79.220.182 (182.79.220.182) [*] 1.683 ms 1.802 ms
7 a125-252-226-97.deploy.akamaitechnologies.com (125.252.226.97) [AS9498] 1.821 ms 1.775 ms 1.947 ms

An interesting case here is that NTT owns majority stake in Netmagic datacenter where this server is located! But likely they can’t do much since they need a license in India to offer their own network in Netmagic or simply peer more? 🙂

 

This is how I increased my download speed from 512Kbps to 24Mbps! 😉