Issues with Google's network in India due to Vardah

Today Google (AS15169) seem to be facing issues in their Indian network due to tropical cyclone Vardah. Their traffic at PoPs in Mumbai dipped considerably for a number of ISPs. My guess is that it’s likely because of outage in a large Govt. operator’s network who has overhead fibres along with utility lines.

It’s very important to note that “Google PoP” faced the issue and it’s no where close to saying that Google services went down. Google has a large network across the globe where they peer with networks. If one segment of this network goes down, traffic is re-routed via other parts and as per design even if the network goes down completely in say Mumbai or Chennai, services should stay live. While in real practice considerable degradation occurs because most of the Indian networks get a very large amount of traffic from Google and usually do not have that much extra capacity on their IP transit links, resulting in choking of transits during issues on their PNI with Google.   This shows how traffic of an ISP connected to Google in Mumbai dipped during peak time around at 4pm on Monday 12th Dec (IST) and went to zero little before midnight. I triggered a trace to aspmx.l.google.com. which is outside India from RIPE atlas probes in India and in general routing to that goes via Google’s backbone. Cluster with hostname aspmx.l.google.com (and few others) carry the Gmail/Google Apps traffic and it’s published by Google Apps users in their domain’s MX records.

Measurement results here - https://atlas.ripe.net/measurements/6959474/#!probes

Cases, where latency is less than 200ms, are actually failures. Some examples of failed traces:

from AS4755:

from AS9430:

from AS9829:

Clearly, a lot of re-routing was happening and many times it was routers which took a while to re-route traffic across a down path due to BGP convergence.   With the hope that your YouTube streams run fine while you share them on Gmail, time for me to get back to work! 😄