13 Dec

Issues with Google’s network in India due to Vardah

Today Google (AS15169) seem to be facing issues in their Indian network due to tropical cyclone Vardah. Their traffic at PoPs in Mumbai dipped considerably for a number of ISPs. My guess is that it’s likely because of outage in a large Govt. operator’s network who has overhead fibres along with utility lines.

It’s very important to note that “Google PoP” faced the issue and it’s no where close to saying that Google services went down. Google has a large network across the globe where they peer with networks. If one segment of this network goes down, traffic is re-routed via other parts and as per design even if the network goes down completely in say Mumbai or Chennai, services should stay live. While in real practice considerable degradation occurs because most of the Indian networks get a very large amount of traffic from Google and usually do not have that much extra capacity on their IP transit links, resulting in choking of transits during issues on their PNI with Google.

 

This shows how traffic of an ISP connected to Google in Mumbai dipped during peak time around at 4pm on Monday 12th Dec (IST) and went to zero little before midnight. I triggered a trace to aspmx.l.google.com. which is outside India from RIPE atlas probes in India and in general routing to that goes via Google’s backbone.

Cluster with hostname aspmx.l.google.com (and few others) carry the Gmail/Google Apps traffic and it’s published by Google Apps users in their domain’s MX records. Measurement results here – https://atlas.ripe.net/measurements/6959474/#!probes

 

Cases, where latency is less than 200ms, are actually failures. Some examples of failed traces:

From AS4755: 

 

 

 

 

 

 

from AS9430:

 

 

 

 

 

from AS9829:

 

 

 

 

 

 

 

 

Clearly, a lot of re-routing was happening and many times it was routers which took a while to re-route traffic across a down path due to BGP convergence.

 

With the hope that your YouTube streams run fine while you share them on Gmail, time for me to get back to work! 🙂

25 Nov

Peering with content networks in India

peering

One of frequent email and contact form message I get my blog is about available content networks in India and where one can peer. There are certain content networks in India and of course most of the content networks have open peering policy and are usually happy with direct inter-connection (we call as “peering“) with the ISP networks (often referred to as “eyeball networks”). Some of these networks have a backbone which connects back to their key datacenter locations on their own circuits via Singapore/Europe, some other have simply placed their caching server where cache fill happens over IP transit.

 

Based on publically known information across community and of course peeringdb, following content players are available in India and known to be open for peering:

  1. Google
  2. Microsoft
  3. Amazon
  4. Limelight

 

A quick list of these with datacenter names and locations as taken from Peeringdb record of these networks.

Organisation ASN City Datacenter Location
Amazon 16509 Mumbai GPX Mumbai Unit A-001, Boomerang Chandivali Farm Road, Near Chandivali Studio, Andheri East Mumbai, Mumbai, 400 051
Amazon 16509 Noida Sify Greenfort – Noida B7, Block A, Sector 132, Noida Expressway, Noida , UP 201304
Amazon 16509 Mumbai Tata Mumbai IDC LVSB, Opposite Kirti College
6th floor, Prabahdevi
Mumbai, MH, 400 028
Google 15169 Chennai Bharti Airtel Santhome Bharti Towers, 101 Santhome High Road, Chennai, 600 028
Google 15169 Mumbai GPX Mumbai Unit A-001, Boomerang Chandivali Farm Road, Near Chandivali Studio, Andheri East Mumbai, Mumbai, 400 051
Google 15169 Noida Sify Greenfort – Noida B7, Block A, Sector 132, Noida Expressway, Noida , UP 201304
Google 15169 Chennai TATA Communications Ltd 14th floor, 2nd block
4, Swami Sivanand Salai, Chennai, TN 600 002
Google 15169 Delhi Tata Delhi VSB, Bangla Sahib Road, New Delhi 110001
Google 15169 Mumbai Tata Mumbai IDC LVSB, Opposite Kirti College
6th floor, Prabhadevi
Mumbai, MH, 400 028
Limelight 55439 / 22822 Chennai Bharti Airtel Santhome Bharti Towers, 101 Santhome High Road, Chennai, 600 028
Limelight 55439 / 22822 Mumbai Netmagic Vikhroli Mehra Industrial Estate
LBS Marg, Vikhroli
Mumbai, 400 079
Microsoft 8075 Mumbai Bharti Airtel Mumbai Plot No, TPS-2, 14/3, 2nd floor
Dattatray Road, Linking Road Extension
Mumbai, 400054
Microsoft 8075 Chennai Bharti Airtel Santhome Bharti Towers, 101 Santhome High Road, Chennai, 600 028
Microsoft 8075 Chennai TATA Communications Ltd 14th floor, 2nd block
4, Swami Sivanand Salai, Chennai, TN 600 002
Microsoft 8075 Delhi Tata Communications Ltd – GK1 Greater Kailash-1
New Delhi, 110048
Microsoft 8075 Mumbai Tata Mumbai IDC LVSB, Opposite Kirti College
6th floor, Prabhadevi
Mumbai, MH, 400 028

 

Besides these Google also has an option of GGC, Akamai has an option of Akamai Caching server, Facebook has the option for caching server which is hosted inside ISP’s network and Netflix has an option for OCAs. Besides these networks there are known nodes of Verizon’s Edgecast in Delhi, Mumbai & Chennai (as per this map), Cloudflare has nodes in Delhi, Mumbai & Chennai (as per this map), PCH & K-root server have a node with Web Werks available on MCH peering fabric and Dyn has a node in Mumbai (as per this map).

Go ahead and peer as after all it all starts with a handshake. 🙂