Why Indian internet traffic routes from outside of India?

Isp-Column

After my last post about home networking, I am jumping back into global routing. More specifically how Indian traffic is hitting the globe when it does not need to. This is an old discussion across senior management folks in telcos, policymakers, and more. It’s about “Does Indian internet traffic routes from outside of India?” and if the answer is yes then “Why?” and “How much?”

It became a hot topic, especially after the Snowden leaks. There was even an advisory back in 2018 from Deputy National Security Advisor to ensure Indian internet traffic stays local (news here). Over time this has come up a few dozen times in my discussion with senior members from the Indian ISP community, individuals, and even latency-sensitive gamers. So I am going to document some of that part here. I am going to put whatever can be verified publically and going to avoid putting any private discussions I had with friends in these respective networks. The data specially traceroutes will have measurement IDs from RIPE Atlas so they can be independently verified by other network engineers.

 

Does Indian internet traffic routes from outside of India?

Yes, a part of it does that.

 

How do we know that’s happening?

Well, a simple traceroute is good enough to reveal the forward path. When triggering traces from large endpoints (on projects like RIPE Atlas) to another set of large endpoints, one can find that. I am going to put some traces showing that behavior as I proceed in this blog post.

 

Why does that happen?

There can be broadly four reasons on why that happens in the Indian context:

  1. Simple misconfiguration where pools which are supposed to be advertised locally are missed OR a more specific pool is announced to International transits & peers compared to domestic peers/transit. More specific = always wins in the routing.

  2. Heterogeneous route filtering - As networks roll out route filtering (based on IRR and RPKI), they typically start with their peers and at a later stage start filtering their customers. So e.g Tata Comm AS4755 filters routes based on IRR route objects from its peers (like Airtel AS9498) but Airtel AS9498 is also a customer of Tata Comm AS6453 outside of India. There are visible cases where Tata Comm AS4755 would drop peering routes from AS9498 but would accept them Tata Comm global network AS6453 from outside of India. While both AS6453 & AS4755 belong to Tata Comm, at the technical layer AS6453 is upstream of AS4755. Remember AS9498 is a peer of Tata Comm AS4755 in India but a downstream customer of Tata Comm global backbone AS6453 outside of India.

  3. Paths on which actual traffic flow is just not expected. This is true, especially for CDN IPs. CDNs often DNS magic to map end-users to the nearest node. They typically do not expect users on ISP 1 to be connecting to caching nodes on ISP 2. But if we trace from ISP 1 towards caching IPs on ISP 2 - it might go from outside of India. Ideally, this should not happen because of domestic peering within networks (more on this later in this post) but practically has a very low/negligible impact on actual traffic flow. This is also often for loopback IPs large operators where aggregate is announced somewhere far away & more specific /32 IPv4 or /128 IPv6 stays local within the network. Again these do not have actual traffic flow.

  4. Lack of peering/interconnection within India

Out of these 4th is the largest cause and has a serious impact on end-users. Before jumping on lack of peering/interconnection let’s understand how interconnection is supposed to be.

 

A reminder of transit free network concept

There are around 16 networks in the world that are known as transit-free / tier 1 networks. These networks typically hold a very large set of the global routing table as its direct or indirect customers. All networks in the world are either directly purchasing IP transit from them or from their customer or from their customer’s customer and so on. These operators peer with each other and by virtue of having to peer with each other they can reach any network in the world. Any given network is either their direct/indirect customer or their peer’s direct indirect customer. The routing table these players collectively hold is often called a Default Free Zone (DFZ). Thus to ensure full internet connectivity a network must ensure that its route reach at least one transit-free network as a customer (as a customer so that it announces them to the rest of its peers). Once routes reach at least one transit-free network, they are visible in the default-free zone and hence comes the guarantee of reachability. This is just about the concept of reaching every network in the world. The majority of traffic by volume actually does not hit transit-free / tier 1 networks as it’s from content players to eyeball networks and is often routes via various public peering ports over internet exchanges and through private peering PNI ports. If you want to learn more about peering, read Mr. Bill Norton’s Dr Peering Playbook.

So how do Indian networks ensure global reachability? They buy IP transit from various transit-free networks. Now any lack of domestic peering won’t lead to a blackholing/total drop of traffic but instead of spillover to upstreams of these networks outside of the country. Thus to analyze where Indian traffic routes from outside, we should focus on networks that connect to networks outside India specially the networks in the default-free zone.

 

Which Indian networks connects to outside of India?

As of now, there are 3105 ASNs allocated to Indian networks out of which 2172 ASNs are active and visible in the global routing table when looking via points like RIPE RIS collectors. By looking at the raw routing table & with a test one can find which are the ASNs out of 2172 connect outside of India.

Indian ASNs which connect to outside world on layer 3 routing:

  1. Bharti Airtel AS9498
  2. Tata Communications AS4755/AS6453
  3. Jio AS55836/AS64049
  4. Vodafone IDEA AS55410 AS55644
  5. Sify AS9583
  6. BSNL AS9829
  7. Telstra AS4637
  8. Reliance Communications FLAG AS18101/AS15412

(Note: This is purely from the view of layer 3 routing. There can be cases like Powergrid connecting to Nepal/Bangladesh but on layer 1 circuits and hence not visible when looking at the BGP data)

Now this list of networks is important as they appear to be transit for the entire Indian table with 39000+ routes. As long as routing within these networks happens within India, our traffic would be local. Else it would “leak” from outside of India. One cannot ensure 2172 networks connect to each other but as long as these specific backbones are connected (in peering or transit relation) within India, traffic should be local. Of course, more dense peering by their downstream locally is even better. Folks at DE-CIX India and Extreme IX are doing a good job to promote that.

Notable cases where Indian traffic is being routed from outside of India:

  1. Traffic between Telstra Reach AS4637 PoPs in India and Airtel AS9498 routes from outside of India (RIPE atlas measurements here and raw traces here).

  2. Some traffic between Telstra Reach AS4637 PoPs in India and Tata Comm AS4755 routes from outside of India (RIPE atlas measurements here and raw traces here). This feels like a config issue within AS4755 because some traces also show direct paths like this one from their looking glass. Ideally iBGP within AS4755 should have ensured direct local paths but seems like Delhi PoP has while Hyderabad is missing.

  3. Traffic between Sify AS9583 and Airtel AS9498 is routing from outside of India (RIPE atlas measurements here and raw traces here).

  4. Traffic from Airtel AS9498 towards BSNL AS9829 South Indian pools is routing from outside of India (RIPE atlas measurements here and raw traces here). Here from data from looking glass, it seems clear that Airtel AS9498 is rejecting AS9829 routes at NIXI Chennai. All other major operators including Tata Comm, Jio, etc are picking those routes as visible from traces.

  5. Traffic destination to Vodafone IDEA (VI) for many pools is routing from outside of India. This includes cases where traffic is from Airtel, Tata Comm, Jio (RIPE atlas measurements here and raw traces here)

In all these cases all these networks are at multiple NIXI exchanges and traditional knowledge says that they should be getting routes of each other in session via the NIXI route server (forced multi-lateral peering policy). But that’s clearly not happening. In the case of Telstra, routes are clearly visible at NIXI Noida (lookup result here) and thus it seems to be an issue of Airtel & Tata Comm. In the case of Sify, their routes are visible across key NIXI PoPs - Noida, Mumbai, and Chennai (lookup here). In case of the BSNL issue - their South Indian routes are visible at NIXI Chennai (lookup here). In the case of Vodafone IDEA (VI) their routes are not visible at any of NIXI PoPs and hence only way for networks to send them traffic is from outside of India.

 

Is there a real world impact of this sort of awful routing?

Yes, there is. Take e.g trace from Airtel fixed line AS24560 towards India’s largest state-owned bank SBI portal (www.onlinesbi.com - 103.68.221.190) which is on Sify. (RIPE Atlas measurements here)

2021-06-20 22:50 UTC

Traceroute to 103.68.221.190 (103.68.221.190), 48 byte packets

1 192.168.1.1 0.231ms 0.168ms 0.27ms
2 192.168.2.1 0.844ms 1.047ms 0.765ms
3 122.179.4.1 abts-kk-dynamic-001.4.179.122.airtelbroadband.in AS24560 73.924ms 69.936ms 56.354ms
4 122.185.118.173 nsg-corporate-173.118.185.122.airtel.in AS9498 67.181ms 59.309ms 73.925ms
5 182.79.154.0 198.111ms 199.051ms 205.77ms
6 149.14.224.161 be6391.rcr21.b015591-1.lon13.atlas.cogentco.com AS174 202.371ms 208.665ms 201.43ms
7 130.117.2.65 be2053.ccr41.lon13.atlas.cogentco.com AS174 197.046ms 197.73ms 198.251ms
8 154.54.58.174 be2870.ccr22.lon01.atlas.cogentco.com AS174 199.739ms 200.94ms 201.756ms
9 149.6.149.114 AS174 199.097ms 198.14ms 199.624ms
10 100.66.12.5 207.597ms 223.398ms 209.189ms
11 100.66.12.5 206.488ms 206.549ms 208.43ms
12 223.31.161.154 223-30-0-0.lan.sify.net AS9583 207.665ms 206.16ms 207.075ms
13 * * *
14 * * *
15 * * *

Thus requests from Airtel users to SBI portal route via London. There we throw the concerns of National security or localized routing out of the window.

Thus sad part here is that instead of acting as a major interconnection hub for Asia, we are having poor routing in networks within India. I hope this broken peering ecosystem is fixed eventually.

Disclaimer: This is my personal blog and I am posting here as an individual. Post has nothing to do with my employer.