21 Jun

Why Indian internet traffic routes from outside of India?

After my last post about home networking, I am jumping back into global routing. More specifically how Indian traffic is hitting the globe when it does not need to. This is an old discussion across senior management folks in telcos, policymakers, and more. It’s about “Does Indian internet traffic routes from outside of India?” and if the answer is yes then “Why?” and “How much?”

It became a hot topic, especially after the Snowden leaks. There was even an advisory back in 2018 from Deputy National Security Advisor to ensure Indian internet traffic stays local (news here). Over time this has come up a few dozen times in my discussion with senior members from the Indian ISP community, individuals, and even latency-sensitive gamers. So I am going to document some of that part here. I am going to put whatever can be verified publically and going to avoid putting any private discussions I had with friends in these respective networks. The data specially traceroutes will have measurement IDs from RIPE Atlas so they can be independently verified by other network engineers.

Does Indian internet traffic routes from outside of India?

Yes, a part of it does that.

How do we know that’s happening?

Well, a simple traceroute is good enough to reveal the forward path. When triggering traces from large endpoints (on projects like RIPE Atlas) to another set of large endpoints, one can find that. I am going to put some traces showing that behavior as I proceed in this blog post.

Why does that happen?

There can be broadly four reasons on why that happens in the Indian context:

  1. Simple misconfiguration where pools which are supposed to be advertised locally are missed OR a more specific pool is announced to International transits & peers compared to domestic peers/transit. More specific = always wins in the routing.

  2. Heterogeneous route filtering – As networks roll out route filtering (based on IRR and RPKI), they typically start with their peers and at a later stage start filtering their customers. So e.g Tata Comm AS4755 filters routes based on IRR route objects from its peers (like Airtel AS9498) but Airtel AS9498 is also a customer of Tata Comm AS6453 outside of India. There are visible cases where Tata Comm AS4755 would drop peering routes from AS9498 but would accept them Tata Comm global network AS6453 from outside of India. While both AS6453 & AS4755 belong to Tata Comm, at the technical layer AS6453 is upstream of AS4755. Remember AS9498 is a peer of Tata Comm AS4755 in India but a downstream customer of Tata Comm global backbone AS6453 outside of India.

  3. Paths on which actual traffic flow is just not expected. This is true, especially for CDN IPs. CDNs often DNS magic to map end-users to the nearest node. They typically do not expect users on ISP 1 to be connecting to caching nodes on ISP 2. But if we trace from ISP 1 towards caching IPs on ISP 2 – it might go from outside of India. Ideally, this should not happen because of domestic peering within networks (more on this later in this post) but practically has a very low/negligible impact on actual traffic flow. This is also often for loopback IPs large operators where aggregate is announced somewhere far away & more specific /32 IPv4 or /128 IPv6 stays local within the network. Again these do not have actual traffic flow.

  4. Lack of peering/interconnection within India

Out of these 4th is the largest cause and has a serious impact on end-users. Before jumping on lack of peering/interconnection let’s understand how interconnection is supposed to be.

A reminder of transit free network concept

There are around 16 networks in the world that are known as transit-free / tier 1 networks. These networks typically hold a very large set of the global routing table as its direct or indirect customers. All networks in the world are either directly purchasing IP transit from them or from their customer or from their customer’s customer and so on. These operators peer with each other and by virtue of having to peer with each other they can reach any network in the world. Any given network is either their direct/indirect customer or their peer’s direct indirect customer. The routing table these players collectively hold is often called a Default Free Zone (DFZ). Thus to ensure full internet connectivity a network must ensure that its route reach at least one transit-free network as a customer (as a customer so that it announces them to the rest of its peers). Once routes reach at least one transit-free network, they are visible in the default-free zone and hence comes the guarantee of reachability. This is just about the concept of reaching every network in the world. The majority of traffic by volume actually does not hit transit-free / tier 1 networks as it’s from content players to eyeball networks and is often routes via various public peering ports over internet exchanges and through private peering PNI ports. If you want to learn more about peering, read Mr. Bill Norton’s Dr Peering Playbook.

So how do Indian networks ensure global reachability? They buy IP transit from various transit-free networks. Now any lack of domestic peering won’t lead to a blackholing/total drop of traffic but instead of spillover to upstreams of these networks outside of the country. Thus to analyze where Indian traffic routes from outside, we should focus on networks that connect to networks outside India specially the networks in the default-free zone.

Which Indian networks connects to outside of India?

As of now, there are 3105 ASNs allocated to Indian networks out of which 2172 ASNs are active and visible in the global routing table when looking via points like RIPE RIS collectors. By looking at the raw routing table & with a test one can find which are the ASNs out of 2172 connect outside of India.

Indian ASNs which connect to outside world on layer 3 routing:

  1. Bharti Airtel AS9498
  2. Tata Communications AS4755/AS6453
  3. Jio AS55836/AS64049
  4. Vodafone IDEA AS55410 AS55644
  5. Sify AS9583
  6. BSNL AS9829
  7. Telstra AS4637
  8. Reliance Communications FLAG AS18101/AS15412

(Note: This is purely from the view of layer 3 routing. There can be cases like Powergrid connecting to Nepal/Bangladesh but on layer 1 circuits and hence not visible when looking at the BGP data)

Now this list of networks is important as they appear to be transit for the entire Indian table with 39000+ routes. As long as routing within these networks happens within India, our traffic would be local. Else it would “leak” from outside of India. One cannot ensure 2172 networks connect to each other but as long as these specific backbones are connected (in peering or transit relation) within India, traffic should be local. Of course, more dense peering by their downstream locally is even better. Folks at DE-CIX India and Extreme IX are doing a good job to promote that.

Notable cases where Indian traffic is being routed from outside of India:

  1. Traffic between Telstra Reach AS4637 PoPs in India and Airtel AS9498 routes from outside of India (RIPE atlas measurements here and raw traces here).

  2. Some traffic between Telstra Reach AS4637 PoPs in India and Tata Comm AS4755 routes from outside of India (RIPE atlas measurements here and raw traces here). This feels like a config issue within AS4755 because some traces also show direct paths like this one from their looking glass. Ideally iBGP within AS4755 should have ensured direct local paths but seems like Delhi PoP has while Hyderabad is missing.

  3. Traffic between Sify AS9583 and Airtel AS9498 is routing from outside of India (RIPE atlas measurements here and raw traces here).

  4. Traffic from Airtel AS9498 towards BSNL AS9829 South Indian pools is routing from outside of India (RIPE atlas measurements here and raw traces here). Here from data from looking glass, it seems clear that Airtel AS9498 is rejecting AS9829 routes at NIXI Chennai. All other major operators including Tata Comm, Jio, etc are picking those routes as visible from traces.

  5. Traffic destination to Vodafone IDEA (VI) for many pools is routing from outside of India. This includes cases where traffic is from Airtel, Tata Comm, Jio (RIPE atlas measurements here and raw traces here)

In all these cases all these networks are at multiple NIXI exchanges and traditional knowledge says that they should be getting routes of each other in session via the NIXI route server (forced multi-lateral peering policy). But that’s clearly not happening. In the case of Telstra, routes are clearly visible at NIXI Noida (lookup result here) and thus it seems to be an issue of Airtel & Tata Comm. In the case of Sify, their routes are visible across key NIXI PoPs – Noida, Mumbai, and Chennai (lookup here). In case of the BSNL issue – their South Indian routes are visible at NIXI Chennai (lookup here). In the case of Vodafone IDEA (VI) their routes are not visible at any of NIXI PoPs and hence only way for networks to send them traffic is from outside of India.

Is there a real world impact of this sort of awful routing?

Yes, there is. Take e.g trace from Airtel fixed line AS24560 towards India’s largest state-owned bank SBI portal (www.onlinesbi.com – 103.68.221.190) which is on Sify. (RIPE Atlas measurements here)

2021-06-20 22:50 UTC

Traceroute to 103.68.221.190 (103.68.221.190), 48 byte packets

1 192.168.1.1 0.231ms 0.168ms 0.27ms
2 192.168.2.1 0.844ms 1.047ms 0.765ms
3 122.179.4.1 abts-kk-dynamic-001.4.179.122.airtelbroadband.in AS24560 73.924ms 69.936ms 56.354ms
4 122.185.118.173 nsg-corporate-173.118.185.122.airtel.in AS9498 67.181ms 59.309ms 73.925ms
5 182.79.154.0 198.111ms 199.051ms 205.77ms
6 149.14.224.161 be6391.rcr21.b015591-1.lon13.atlas.cogentco.com AS174 202.371ms 208.665ms 201.43ms
7 130.117.2.65 be2053.ccr41.lon13.atlas.cogentco.com AS174 197.046ms 197.73ms 198.251ms
8 154.54.58.174 be2870.ccr22.lon01.atlas.cogentco.com AS174 199.739ms 200.94ms 201.756ms
9 149.6.149.114 AS174 199.097ms 198.14ms 199.624ms
10 100.66.12.5 207.597ms 223.398ms 209.189ms
11 100.66.12.5 206.488ms 206.549ms 208.43ms
12 223.31.161.154 223-30-0-0.lan.sify.net AS9583 207.665ms 206.16ms 207.075ms
13 * * *
14 * * *
15 * * *

Thus requests from Airtel users to SBI portal route via London. There we throw the concerns of National security or localized routing out of the window.

Thus sad part here is that instead of acting as a major interconnection hub for Asia, we are having poor routing in networks within India. I hope this broken peering ecosystem is fixed eventually.

Disclaimer: This is my personal blog and I am posting here as an individual. Post has nothing to do with my employer.

13 Feb

Indian IPv6 deployment

I had calls with a couple of friends over this week and somehow discussion IPv6 deployment came up. “How much has been IPv6 deployment in India now in 2020” is a very interesting question. It’s often added with – “how much of my traffic will flow over IPv6 once it is enabled“?

Game of numbers

There is a drastic difference in IPv6 deployment depending on which statistic we are looking at here in India. There can be a bunch of factors based on which we can try to judge IPv6 deployment:

  1. How many operators are offering IPv6 to end users?
  2. How many end-users are on IPv6?
  3. How’s the content available on IPv6 in terms of a number of IPv6 enabled websites?
  4. How’s the content available on IPv6 in terms of traffic volume over IPv6?

First two and last two points are related and point towards from vertically opposite ends. (Call it good or bad) the fact of high centralisation. There has been an ongoing centralisation of mobile operators and in-country like ours they connect a very large number of end-users. The number of fixed-line networks has increased considerably but at the same time in proportion to a number of mobile users they user base growth has been much lower.

On the content side like everywhere else in the world, there’s a lot more centralisation of content. Many of my Indian ISP friends tell me that Google + Akamai + Microsoft + Netflix + Facebook + Cloudflare is way over 75% of their traffic. Think about it, that’s just 6 AS number out of 68000+ odd networks in the world as per BGP routing table. Thus by traffic profile, we are looking at 0.0014% networks serving 75% of content traffic. The reason for such centralisation is actually beyond network and more around the success of products of these organisations followed by factors like the winner (or top 3) gets it all in most of these domains.

For eyeball traffic APNIC IPv6 stats for India (source here) as well as Hurricane Electric’s IPv6 progress report (source here) give us some numbers:

  1. Very few fixed-line operators are offering IPv6 but on the mobile side – a large number of mobile networks are offering IPv6. Jio was on IPv6 since launch and as traffic increased, Airtel as well as Vodafone + IDEA also significantly increased their deployment. On fixed line, it’s just Jio + ACT broadband with any sizable IPv6 footprint. BSNL + Airtel have virtually no deployment. There might be some other network in the list but it’s off the radar.
  2. There are a lot more end users on IPv6 than despite the small number of networks offering it because of the large mobile user base. On Jio 80%+ user base, on Airtel, it’s 45%, on Vodafone & IDEA (merged company but still separate ASNs) it’s close to 50%. That’s the number of users who are connecting over IPv6 when given option of IPv4 and IPv6 as tested by APNIC. That number is huge!
  3. Around 98.5% of TLDs (top-level domain names like .com, .net etc) are IPv6 enabled. For .com & .net domains, only 7% of the domains have an AAAA record (compared to ones having an A record). Most of the numbers are much lower if we look at the content side of IPv6 (ignoring the traffic volume).
  4. If we look at the content players with IPv6 and include the traffic volume numbers then it’s way higher. It is estimated that globally somewhere between 20-30 top ASNs carry 90%+ traffic and almost all of those top 20 are IPv6 enabled.

What do all these numbers actually mean?

  1. If you are an eyeball network in India and you deploy IPv6, you can expect way over 70-80% traffic (by volume) on IPv6.
  2. If you are a content network/datacenter in India and application is targetting to home fixed-line / enterprise network, expect a rather low amount of IPv6 traffic but would be rapidly increasing as more fixed-line networks deploy.
  3. If you are a content network/datacenter in India and content hosted at your end attracts mobile traffic, you can expect way over 50%-60% of that mobile traffic over IPv6.

Some additional reasons to consider deploying IPv6

  1. In India, ISPs need to maintain carrier-grade NAT logs of the translations. If one is doing dual-stack, a large part of traffic will flow over IPv6 saving on those logging requirements.
  2. For ISPs, it will save you from significant strain on CGNAT device.
  3. For content network/webmaster/datacenter – IPv6 will help in delivering your traffic outside of (often) congested CGNAT paths.

Happy IPv6ing! 🙂

04 Jan

VoWifi experience on Jio

Since last week of Nov 2019, I am having serious issues with Airtel at my home. Somehow 4G signal SNR is very poor and most of the calls just fail on that. Airtel support just mentioned that they are putting a new site in my area in Jan 2020 but fail to explain why suddenly it went so bad. I can imagine that support team staff does not have visibility to network in real-time and likely it would be an issue with the 4G antenna on one or more towers. 2G signal was good but latency was extremely high to connect call plus calls still failed regardless (maybe due to high strain on 2G).

So by the end of December, I just gave up and gave the request for porting to Jio. Yesterday my number started on Jio and around 24 hours later I got access to VoWifi which is quite nice. Both Airtel & Jio are rolling our VoWifi but for Airtel, many users have reported it for being locked on Airtel fixed line only.

What is VoWifi?

Well, it’s just native offloading of voice calls over the fixed-line network and is extremely useful for both network operators as well as end-users. Native IP equipment is cheap and it’s way easy to extend cell coverage at home via Wifi instead of proprietary inbuilding solutions. Plus landline was good for quality but form factor and usability was pain. Now VoWifi kind of merges the fixed-line and cell phones and we get best of both worlds.

I think many people confused BSNL Wings with VoWifi. VoWifi is seamless offloading of calls natively while BSNL Wings offered a separate number, an app to run and a separate billing account to maintain with the operator. BSNL Wings was a bad product idea and really awful execution.

Coming to Jio VoWifi, some misc observations:

  1. It’s working for me on the non-Jio fixed-line connection in Haryana circle. I have IAXN (GEPON FTTH) and Siti broadband (DOCSIS 3.0) at home running in active/backup configuration for high availability.
  2. The device has an IPsec tunnel with 49.44.59.XXX. I see isakmp-nat-keep-alive packets every 20 seconds when I do tcpdump on the router.
  3. On my cell phone (which is an Apple iOS device) I see a tunnel is opened on IPSEC1 port. IPSEC2 & IPSEC3 are there for VoLTE.
    IPSEC1 has a link-local IPv6 and an IPv6 from 2405:205:3000::/36 range. The IPv6 on all three IPSEC tunnels is the same as of pdp_ip1 (Cellular data port). I guess that’s probably how they are handing the handovers between VoWifi and VoLTE.
  4. The IPv6 on pdp_ip0 for cellular data is from a different range which again is being announced in the global table as 2409:4051::/36. I wonder why Jio is using publically routable pool for VoLTE and VoWifi. In case of Airtel it a non-routed address space and routes are not announced in the global routing table.
  5. In my test VoWifi to VoLTE was reasonably OK. I tested by shutting off wifi on the cell phone as well as unplugging the cable from wifi AP to test. In both cases call went silent for 1-2 seconds and resumed over VoLTE. It did not come back on VoWifi once Wifi was back live.
  6. Latency from both connections to other endpoint (the router before the IPSEC endpoint) is around 60ms. Either the endpoint is somewhere far off or (more likely) it’s a case of not-so-good routing between Airtel (upstream of my ISP) and Jio.
  7. There’s a quite noticeable improvement in connection time. For IVR based numbers it’s close to 2-3 seconds. Which is a great improvement for someone like me who was on 2G non-VoLTE calls for December!

How to check interfaces and routing table on your phone?

Well use HE.NET Network App on Android or Apple iPhone and you will be able to check that!

That’s about it. Back to the world of routing. 🙂