Transit manipulation across Indian IXPs

Isp-Column

It’s a known secret across many ISPs in India that IP transit manipulation/stealing happens at IXPs. This practice has persisted for several years. In this post, I am going to analyse the number of ISPs who seem to be doing it intentionally or unintentionally.


Background

If you are from the BGP routing world and reading this for the first time, you would be wondering how is that even possible. Well, traditionally networks could steal IP transit in the outbound direction by randomly pointing the default route to a peer (who has not agreed to transit them) and this would give them “outbound bandwidth”. Unethical, and very likely illegal as well. But since it’s free outbound, most of the inbound heavy eyeball ISPs had nothing to gain from it. This blog post does not cover that well-known angle.

There’s an Indian-specific condition which allows eyeball ISPs to “steal” / “get free” IP transit bandwidth from some IXPs in some specific cases.

  1. Presence at IXPs: Large telco-run ISPs are at NIXI with limited capacity/routes but maintain a presence for mostly compliance reasons as many tenders/RFPs require them to be peering at NIXI. Some non-telco but reasonably large networks are also at some of the private IXPs as well.
  2. Overlapping Relationships: These same set of players sell IP transit to a similar set of eyeball-heavy ISPs some of which are present at these IXPs.
  3. Common Routing Tables: FIB for ISPs is mostly common for downstream as well as peering routes.
  4. Routing Behavior: In IP routing, a more specific route always wins.

So here’s how it works: an ISP connects at NIXI with a peering port, and they also take IP transit from one of the ISPs. Now, if the ISP announces a /22 on the IP transit port and 2 x /23s (or 4 x /24s) on the peering port, in many cases they will get almost all the traffic via IXP. ISP would learn /22, tag it as a customer route and announce it globally. They would usually tag /23 differently, keep announcements locally within India, and avoid announcing it to their peers as upstreams.

Now, /22 will “catch traffic” and bring it to India but /23 will take over as a more specific route and traffic will exit via a peering port. So, the ISP from whom bandwidth is being taken/stolen will appear to be sitting between its upstream (or a peer) on the international side and another peer on the Indian side. This is a classical case of peering with your customers while also selling them transit. It’s a tricky problem to solve. Furthermore, it’s a good practice to announce more specific at IX and less specific at transit and thus people doing that may not be all doing it with the intention of transit stealing.


Telcos know

When I originally found it (years ago), I shared it with friends in the larger networks. I showed them some of those interesting traceroutes and they agreed but not much happened. I had no personal stake in it and did not bother much beyond that point. Later, I was contacted by my good friend Mr Venkat from Worldphone. He lost a customer in Chennai and when he checked the local market to find out who the customer went to, he learnt “nowhere!” and that made him curious about what was happening. We had a long chat and discussed in detail how it works. He eventually reached out to senior management at one of the telcos to push them to drop their customer ASNs from peerings. Not technically the best (more on this later) but somewhat worked.


Analysing the impact

To find out who is doing it, I decided to trace to .1 of all possible unique /24s in India. That would give a clear FIB view instead of an RIB view. Selling of IP transit across IX fabric is not happening in India as far as I know. There could be some corner case but I don’t know of any larger networks doing it as of now. Now source of such a trace has to be outside of India for two specific reasons:

  1. By tracing from outside of India, there’s a very chance that entities outside of India will NOT be on the customer side of Indian ISPs. That way we can be sure that traces go either via upstream of Indian telcos or peering ports.
  2. Some hardware vendors like Mikrotik Router OS reply to TTL time exceeded with the SRC IP of the port which is returning packets. That creates confusing traceroutes during asymmetric routing. By tracing from outside of India source, there’s an extremely high chance that return packets will not be from a peering link and hence if we see IX IPs in the traces, those are indeed the cases of routing manipulation.

So with that - I put 10 Hetzner VMs to trace in a batch of 1000 IPs each to get me the result in few mins.

As I have said in past, to define the Indian routing table, I can pick routes behind a small set of ASNs which sell IP transit in India and connect to the outside world. Some of them connect to the outside world with the same ASN like Airtel AS9498 or Sify AS9583, some do so via different ASNs like AS4755-AS6453 in the case of Tata Communications or AS55836-AS64049 in the case of Jio. I can pick the domestic ones as they are the ones which connect at IXP (NIXI in the case of these telcos).

ASNs operating in India which connect to the outside world:

  • Bharti Airtel AS9498
  • Tata Comm AS4755
  • Jio AS55836
  • VI AS55410 / AS55644
  • BSNL AS9829
  • Sify AS9583

Notes:

  1. I picked routes from RIPE RIS RRC01 and RRC03. There isn’t a need to diversify much because the goal is not to get all routes and all possible peering but simply prefixes seen behind these ASNs to get the Indian table. This gives me 55621 routes.
  2. I skipped the own originated massive table by Airtel via AS24560, AS45609, BSNL AS9829 etc and looked only at their downstream routes. Technically AS24560 is a downstream but pointless for this analysis. Airtel would not steal bandwidth from itself via NIXI. This reduces the overall unique IPs to trace to just 37061 prefixes. When breaking to /24 levels, it gives me 94423 prefixes i.e. 94423 IPs to trace to.
  3. This check is only done for IPv4. It would take a while to map to each possible /48 in the Indian table due to the large bit size.
  4. There are many cases where a network is on both sides i.e. for some routes they are getting free transit via IX, while in some other cases, their customer is getting free transit out of them.



List of networks

Here is a list of networks which appear to be getting IP transit out of internet exchanges.

Note: It’s important to realise that this list is NOT reflecting people who are “stealing” bandwidth but effectively getting IP transit via an ISP out of IX. Some of them are purely following the good practice of announcing more specifics at IX to get more traffic.

Beneficiary AS Number Beneficiary AS Name Impacted network ASN Impacted Network Name Name of Exchange Trace links
AS133001 Airnet Cable And Datacom AS17665 One OTT AMS-IX Mumbai Click here
AS141519 Yuvaa Communications AS134674 Tata Play Broadband Extreme IX Bangalore Click here
AS17665 One OTT AS18229 Ctrls Extreme IX Bangalore Click here
AS147279 OPTO COMMUNICATIONS AS17665 One OTT Extreme IX Bangalore Click here
AS136956 Assistive Networks and technologies AS18229 Ctrls Extreme IX Bangalore Click here
AS134911 Sigaram Networks AS58898 Rainbow Communications India Extreme IX Chennai Click here
AS151732 Onremote Telecom AS45820 Tata Tele (TTSL) Extreme IX Delhi Click here
AS135692 Global Ra Net Services AS17665 One OTT Extreme IX Delhi Click here
AS24550 Websurfer Nepal AS9583 Sify Extreme IX Delhi Click here
AS17665 One OTT AS133676 PNPL-AS Precious netcom Extreme IX Delhi Click here
AS17665 One OTT AS136362 Yes India Communications Extreme IX Delhi Click here
AS135716 Sat International AS135848 Digitax India Communications Extreme IX Delhi Click here
AS18002 Worldphone AS17665 One OTT Extreme IX Delhi Click here
AS134294 R P World Telecom AS45820 Tata Tele (TTSL) Extreme IX Delhi Click here
AS142524 Poem Techno AS135848 Digitax India Communications Extreme IX Delhi Click here
AS142488 Nextia Broadband AS9583 Sify Extreme IX Delhi Click here
AS134922 Renu Broadband AS9583 Sify NIXI Delhi Click here
AS134922 Renu Broadband AS133647 ELXIRE DATA SERVICES Extreme IX Delhi Click here
AS141529 VMPS Broadband AS9583 Sify Extreme IX Delhi Click here
AS137678 Protoact Digital Network AS9583 Sify Extreme IX Delhi Click here
AS45117 Ishan Netsol AS55836 Jio NIXI Mumbai Click here
AS134331 Smartlink Solutions AS9583 Sify NIXI Mumbai Click here
AS58659 Quest Consultancy AS9583 Sify NIXI Mumbai Click here
AS136334 VORTEX NETSOL AS55836 Jio NIXI Mumbai Click here
AS133001 Airnet Cable And Datacom AS55352 Microscan NIXI Mumbai Click here
AS55352 Microscan AS55836 Jio NIXI Mumbai Click here
AS9885 NKN - National Knowledge Network AS55836 Jio NIXI Mumbai Click here
AS24554 Five Network AS55352 Microscan NIXI Mumbai Click here
AS132296 Seven Star Digital Network AS17762 Tata Teleservices (Maharastra) NIXI Mumbai Click here

Solution to the problem

To solve this problem, a not-so-good but easy way is to drop your customer’s ASNs from peers. However this is not efficient as many networks are multi-homed, they may announce some routes via one transit and some other routes via other transit. So it’s always possible that a given network may learn a few routes directly from customers and a few routes via its peers. So ultimately solution lies in the per-prefix action. It can be done with a basic script with certain logic:

  1. Run a BGP collector/software BGP speaker like FRR or BIRD etc on a server and feed it downstream and peer routes marked by clear BGP communities.
  2. Next, let a script read and compare peering routes with downstream routes to find cases where there is a more specific route on the peering table compared to the downstream table.
  3. Learnt routes can be sent back to the router in the form of a prefix list to drop those on peering for say next 4 hours.
  4. Reset after 4hrs and re-compute to ensure the filter stays dynamic.

Why only at IXP?

A similar issue is highly unlikely outside of IXPs between two large transit players. Imagine the case of Airtel and Tata Comm in the Indian context. If a network announces /22 on Airtel and /23 on Tata Comm, then Tata Comm will carry traffic and exit traffic from its customer port. Only if routing is influenced by the use of BGP communities in a way to keep /23 on Tata Comm local with India (i.e. AS4755 boundary) but /22 via Airtel goes through. In this case, Airtel will carry traffic from outside of India to India and will exit to Tata Comm who will deliver bit to the customer. Possible but unlikely. In the absence of such a limited announcement of more specific prefixes, this behaviour will not occur at large.