NIXI expansion & some thoughts

Isp-Column

Background

Lately, NIXI has been making a bit of news in the Indian peering ecosystem. NIXI for those who may not be aware is the National Internet Exchange of India. It was founded in 2003 with the idea to provide inter-connection layer 2 peering fabric for local Indian ISPs. They were supposed to ensure domestic Indian traffic is exchanged within India and not outside of India. In my previous post, I did cover how that is not true for now. They never picked up much interconnection due to a number of fundamental issues with their policies.

I have written about their issues in the past multiple times in my blog post including (but not limited to) charging for traffic on the “requester pays” model of x-y and charging as high as 50Rs/GB during early days. Next was the issue of not allowing non-licensed (including content networks) to peer at the IX. And lastly the policy of forced multilateral peering policy. Out of these three last still exists.
The other two are gone for a good reason. So no more x-y charge from the last few years and non-ISP license networks are now welcome to connect.
Besides running IX, NIXI is also the administrative contact point for .in domains and runs National Internet Registry - IRINN which allocates IP addresses and AS numbers for networks in India.

 

Latest expansion plans

NIXI has announced plans to open internet exchanges in the following UP cities:

  • Lucknow
  • Prayagraj
  • Kanpur
  • Meerut
  • Varanasi
  • Gorakhpur
  • Bareilly
  • Agra
  • Noia

Besides those cities in UP, there have been news announcements about Dehradun in Uttarakhand, Shimla and a few other places. They are essentially looking for setting up micro PoPs at various locations to facilitate peering. Going by this public document, shows network topology with lots of cities & NLD circuits:

This topology will give a nightmare to any network engineer looking at it. There are many problems with this model of “connecting IXes”. It’s technically as well as commercially a bad idea.

 

Problems with connecting exchanges

  1. Connecting internet exchanges which are far away is problematic because one cannot scale up optical capacity exponentially or even linearly with the demand at such distances. IX will not have access to raw fibre and essentially would be relying on n x 10G and n x 100G waves mostly. Turn around time on changes in that sort of capacity is months as opposed to running a network within say Delhi NCR, Mumbai or Chennai and adding massive DWDM capacity.

  2. At its core, NIXI is a layer 2 exchange. It would be very hard to keep it layer 2 with this sort of design. If NIXI Delhi, Mumbai, and Chennai are connected ….essentially it would be done over an extended layer 2 network. Either raw layer 2 or VXLANs over layer 3 circuits. In either case, it would require changing the IP addressing of all except one IX and fitting everyone on that common IP address block.

  3. If NIXI moves away from being layer 2 IX to achieve this, then we are looking at a layer 3 routed network. That is nothing but just another ISP network. Govt. of India already runs a bunch of those - BSNL, MTNL, Powergrid Telecom, RailTel, NKN, NIC etc. The topology shows switches and not routers which makes me believe that they are sticking with the idea of keeping layer 2 switched network.

  4. Extending layer 2 on long-haul is tricky especially to add redundancy in a mesh design. IX’es running within a single city across multiple datacenters do get issues of peer isolation. In this issue, peers can reach the route server but cannot reach others & which results in BGP routes being in the table but those next-hops not being reachable. Imagine where route servers will be in such design for best redundancy while keeping their number to a logical 2-3 since one cannot expect members to peer with route servers of all locations.

  5. NIXI will end up in subsiding traffic flow from cities like Delhi/Mumbai towards smaller cities. Networks like Google, Facebook, Netflix, Microsoft etc are pumping multiple terabits of traffic towards Indian ISPs. Even if 4%-5% of that multi-terabit traffic moves over these paths, they would be struggling with NLD capacity.

  6. NIXI can charge for local Vs core-node traffic (via NLD) by looking at sflow data but there is going to be extremely low local traffic. Almost all of it would be via Delhi/Mumbai/Chennai for inbound heavy eyeball networks. If NIXI does not charge for it, then it’s just a Govt. network indirectly subsidising private networks. That is non-sustainable and if Govt. badly wants to do it, probably cheaper or free IP transit from BSNL/RailTel/Powergrid nodes would be a better design rather than turning NIXI upside down & making it an ISP.

  7. This design will remove the incentive for the private players to pay for NLD circuits whenever the option would exist. Besides pushing peering traffic, an ISP can simply take port at two points in the topology and start pumping traffic. The current cost of 10G port at NIXI is 5.8lakh per year. That’s way cheaper than what they will pay for the NLD circuits to carry it.

  8. Troubleshooting issues like packet loss and congestion would be extremely hard for ISPs in this design as they won’t have visibility on the actual route taken by the packets (view them as layer 2 frames) over this mesh NLD connectivity.

  9. The above issue along with the policy for forced multi-lateral peering is a recipe for disaster unless that is removed. Imagine Airtel, Tata Comm, Jio, Sify, Spectra, Worldphone etc being connected in Delhi, Mumbai, and Chennai while all on the same single layer 2 fabric and now injecting their table to route servers. Which next-hop peering ISPs should be picking?

  10. There is a lack of a basic datacenter and more importantly a facility with access to raw fibre from the outside pit to inside the building. And hence ISPs in these cities will have their own nightmare in reaching these PoP locations.

So ultimately it’s going to have limited benefits and more market manipulation impact by cross-subsiding NLD in a poorly designed layer 2 network. This is a strange move for BJP lead NDA Govt. in the Center. On one end they are pushing the private sector while on other hand they are making NIXI become one more ISP network which will end up manipulating demand & supply instead of letting on the market forces.

 

Disclaimer

Thoughts are strictly personal. I did interact with NIXI in past quite a few times & whenever asked for an opinion. I did share exactly the same opinion about connecting IXPs as I posted here.

Also, I am on NIXI’s IPv6 Expert Panel group. Though not interacting with NIXI management or their team that actively these days.