Understanding NIXI and it's policies

Networking

NIXI i.e National Internet Exchange of India is well known for it’s inefficiency and for its bad policies. I am posting this blog post to discuss some of them.  

Bit of background:

NIXI is one (and only) Indian IXP i.e Internet Exchange Point established in 2003 so as to facilitate peering between Indian ISPs. Before this, there were lot of cases when Indian ISP’s were connecting to each other from outside India in Singapore and Europe. Thus NIXI established few exchanges in key cities where necessary infrastructure was provided to ISP’s to “peer”.  With peering, the strict technical meaning is that exchange of traffic between ISPs.    


Understanding peering…

E.g if ISP A is peering with ISP B, ISP A will announce it’s and it’s downstream customer prefixes to B and so does B will do with A. If A is peering with C, and B is peering with D, C and D will NOT be able to reach to each other via A and B’s interconnection. While say if E is (downstream) customer of A and F is downstream customer of B, then there will be a route between E to F as E > A > B > F and vice versa (depending on prefixes announced). Thus, if big ISPs like Airtel and Tata peer at NIXI, it will help to keep traffic between them local + also their customers (not peers).  


What is benefit of keeping traffic local?

A lot! :) Firstly that’s way more efficient. Why to have packets to go from India > Europe > US > Singapore > India just because source and destination networks are not connected within India? Also, theoretically it will save us on bandwidth costs (in real it is not yet, which I will discuss in this post). Other then that there are huge technical advantages of keeping traffic i.e low latency which matters a lot when latency within India is usually less then 100ms, while with US it’s 300ms. And worst if we are hitting US twice in a round trip connection, we will get latency of over 500ms! Also, networks work lot faster on direct peering due to TCP window size factor apart from fact that having 4-5 networks between source and destination effects a lot.  


So how much NIXI has done so far?

Well not much! Let’s talk about some numbers. India has around 10 million broadband users and a little over 100 million users (so a lot still on slow non-broadband usage) as per this data. Aggregated traffic passing from all NIXI exchanges is around 20Gbps as per their official data. If we look at Europe, number of Internet users around 50 million as per this data, while aggregated traffic of Amsterdam Internet Exchange (AMX) is over 1.6Tbps alone! (Source). Other important exchanges like London Internet Exchange carry 1.4Tbps, while DE-CIX exchange in Germany carry around 2Tbps of aggrigated traffic. It’s very hard to count total data of all these exchanges but clearly AMX + LINX + DECIX alone carry 5Tbps i.e 5000Gbps of aggregated bandwidth which is 250 times more then that of Indian exchange NIXI. (I am still ignoring dozen of big exchanges here. You can find IXP directory at PCH’s website here)  


At this point one can think of fact that is NIXI only exchange in India?

Well, yes. Indian “licence raj” as usual! Also at AMX, we can see around 514 connected networks, at LINX 435 members while at DECIX around 465 participants. At all NIXI we don’t see more then 30-40 connected networks! I can guess aggrigated number of unique networks to be say 50 or so. Look at list of members in AMX and compare with NIXI. Some big names are missing at NIXI. Few of them are - Google, Akamai, Limelight, Microsoft, root servers networks, Verisign, Yahoo etc. One common thing among all these networks is the fact they are content providers and not really consumers. They send more traffic (way more) then what they receive. NIXI has a requirement that only ISPs can join NIXI and with ISP it means you must have license to operate in India. There’s no place for content provider! This is strange and an absurd because traffic between content consuming ISPs is very low as compared to content between content providers like CDN, datacenters, etc. Unless they join in the exchange, there won’t be really much traffic for ISP participants to consume.   At this point one can think of fact that if content provider like Google is connected to an NIXI participating ISP, won’t it be available at NIXI via that ISP? OK! Let’s look at this. Google.co.in uses IPs from 173.194.36.0/24 subnet. It is announcement by Google’s AS 15169.  Let’s look at this prefix at NIXI Delhi, Mumbai and Chennai using their looking glass.

NIXI Looking Glass - show ip bgp 173.194.36.0/24 Router: NIXI Delhi (Noida) Command: show ip bgp 173.194.36.0/24 % Network not in table NIXI Looking Glass - show ip bgp 173.194.36.0/24 Router: NIXI Mumbai Command: show ip bgp 173.194.36.0/24 % Network not in table NIXI Looking Glass - show ip bgp 173.194.36.0/24 Router: NIXI Chennai Command: show ip bgp 173.194.36.0/24 show ip bgp 173.194.36.0/24 BGP4 : None of the BGP4 routes match the display condition

So - no where. One cannot reach Google AS15169 from NIXI and reason for that is the fact that Google just peers with Indian providers like Airtel, Tata & Reliance and doesn’t really has a transit relationship. Likely Google doesn’t pays to any of them and they only to Google’s Indian PoPs and caching servers.  Since Google has a peering relationship, ISPs do not share Google’s route with other peers. If we look at say Akamai India - I see they have one of DNS servers ns1-2.akam.net in India. It is using IP 193.108.91.2 which is coming from BGP announcement of 193.108.91.0/24 by AS 21342. Looking at prefix at NIXI Delhi:

NIXI Looking Glass - show ip bgp 193.108.91.0/24 Router: NIXI Delhi (Noida) Command: show ip bgp 193.108.91.0/24 BGP routing table entry for 193.108.91.0/24, version 62228 Paths: (1 available, best #1, table Default-IP-Routing-Table) Advertised to update-groups: 1 2 3 9498 20940 20940 20940 21342, (received & used) 218.100.48.20 from 218.100.48.20 (203.101.87.9) Origin IGP, localpref 100, valid, external, best

  Clearly route available via Airtel (AS 9498) and so does at NIXI Mumbai and Chennai too. Here Akamai is visible because Akamai seems to be a downstream customer of Bharti Airtel AS9498.   This is a problem! So if you are content provider, NIXI damages you as:

  • You cannot connect to NIXI’s peering fabric directly unless you have a license to operate in India (NIXI's requirement here)
  • You cannot reach to NIXI via other ISPs unless you become their direct customers. No transit routes, only peering routes. This is normal for an exchange though.

  Also problem is just not limited to this. Another issue at NIXI is cost factor. It’s really terrible how economics work here. As per NIXI’s routing & tariff policy, there is really crazy pricing.  


Counting on costs…

Joining charges - 1000Rs (just $20!), port charges for 100Mbps around $2000, while Gig port costs around $5000 yearly. Comparing it to AMX they don’t seem selling less then 500Mbps directly (well ….) while for 1Gbps it costs 500 Euro monthly which will around $7800 per year. So NIXI seems cheap? Well no! There’s a catch here that NIXI is likely world’s only exchange which has concept of charging on data transferred too! It goes as for exchange of traffic between ISP A and ISP B, the “requester pays” logic goes. So if B’s customer are pulling off data from A then B will have to pay 12Rs/GB to ISP A. Now here catch is if you are a datacenter (well yeah there are datacenters too connected to NIXIs with ISP license) then requester pays concept doesn’t works but datacenter does has to pay for it’s own downstream.  


How does cost is so damaging for a small ISP?

Let’s assume if a small ISP connects to NIXI. Since it is small and has consumers who are just pulling data, it would have to pay 12Rs/GB to big old incumbent operators while they would be taking relatively very less content from that network and would be paying negligible in return (apart from fact that small ISP would be surely using one of them for Internet transit to have global reach ability). So for say a 100Mbps connection in full use, it will transfer around 8640000 Mega bits in one day or 394200000 Mega bytes in one year which is around 394200 GB. At 12Rs/GB pull off costs, it will be around $94k yearly bill! Apart from  $2000 port charges! :) So, we are talking about partial routing tables with almost no content provider and paying around $96k for 100Mbps peering capacity. That too when retail ISP would be likely selling connections at around $10/user on average. So we are talking about atleast 9600 users alone to cover cost of peering apart from layer 2 circuit cost to reach NIXI and surely a HUGE costs of Internet transit to cover rest of non-NIXI routes of global Internet.   Also, if we look at cost of $96k for 100Mbps, it might be little less due to fact that consumers will upload a bit during that time and ISP will also earn. So doing a rough assuming ISP would be paying atleast $80k  in real world case when burst port completely. So $80k for 100Mbps means $800 for each mega bit per second. One can easily get a STM 1 i.e OC3 circuit (for American readers!) for around 25 lakh which will be $50,000. I have ignored fact that ISP has not to pay if it is taking content from datacenters because there are simply not many datacenters in list. Effect will be negligible here! So we are looking at STM 1 - 155Mbps Global transit bandwidth for $50,000 Vs NIXI peering partial routes with missing major content providers for around $80,000 an year. Amazing? As you might have already guessed - one can get way more cheaper transit bandwidth from ISPs then reaching (only) them via NIXI. Also, not to forget that transit matters a lot because lot of content is still in US and not hosted in India. Also providers like Google who are building their own network have their own PoPs at Delhi, Mumbai and Chennai and one can connect there directly rather then reaching via NIXI (Google’s Peering info map here). So clearly NIXI is failing heavily here. Also, if we talk about datacenters like Control S and Netmagic who hold ISP license also and are connected to NIXI.  If we look at Control S AS18229 - they are announcing hundreds of prefixes (check here) in global IPv4 table. They participate at NIXI Delhi, Mumbai and Chennai with name Pioneer E labs and use same ASN. Looking at their prefixes at each of these exchanges via NIXI’s looking glass:

NIXI Looking Glass - show ip bgp neighbors 218.100.48.89 routes Router: NIXI Mumbai Command: show ip bgp neighbors 218.100.48.89 routes BGP table version is 5482108, local router ID is 218.100.48.65 Status codes: s suppressed, d damped, h history, \* valid, > best, i - internal, S Stale Origin codes: i - IGP, e - EGP, ? - incomplete Network Next Hop Metric LocPrf Weight Path \*
103.1.112.0/24 218.100.48.89 0 0 18229 i \*
103.1.113.0/24 218.100.48.89 0 0 18229 i \*
103.1.114.0/24 218.100.48.89 0 0 18229 i \*
> 103.1.115.0/24 218.100.48.89 0 0 18229 i \*
103.8.124.0/24 218.100.48.89 0 0 18229 i \*
> 103.8.126.0/24 218.100.48.89 0 0 18229 i \*
103.13.97.0/24 218.100.48.89 0 0 18229 i Total number of prefixes 7   

NIXI Looking Glass - show ip bgp neighbors 218.100.48.22 routes Router: NIXI Delhi (Noida) Command: show ip bgp neighbors 218.100.48.22 routes NIXI Looking Glass - show ip bgp neighbors 218.100.48.151 received-routes Router: NIXI Chennai Command: show ip bgp neighbors 218.100.48.151 received-routes show ip bgp neighbors 218.100.48.151 received-routes There are 13 received routes from neighbor 218.100.48.151 Searching for matching routes, use ^C to quit... Status A:AGGREGATE B:BEST b:NOT-INSTALLED-BEST C:CONFED\_EBGP D:DAMPED E:EBGP H:HISTORY I:IBGP L:LOCAL M:MULTIPATH S:SUPPRESSED F:FILTERED s:STALE Prefix Next Hop Metric LocPrf Weight Status 1 111.118.212.0/22 218.100.48.151 1 100 0 BE AS\_PATH: 18229 2 182.18.129.0/24 218.100.48.151 1 100 0 BE AS\_PATH: 18229 3 182.18.131.0/24 218.100.48.151 1 100 0 BE AS\_PATH: 18229 4 182.18.132.0/24 218.100.48.151 1 100 0 BE AS\_PATH: 18229 5 182.18.142.0/24 218.100.48.151 1 100 0 BE AS\_PATH: 18229 6 182.18.182.0/24 218.100.48.151 1 100 0 BE AS\_PATH: 18229 7 202.65.135.0/24 218.100.48.151 1 100 0 BE AS\_PATH: 18229 8 202.65.137.0/24 218.100.48.151 1 100 0 BE AS\_PATH: 18229 9 202.65.145.0/24 218.100.48.151 1 100 0 BE AS\_PATH: 18229 10 202.65.148.0/24 218.100.48.151 1 100 0 BE AS\_PATH: 18229 11 202.65.152.0/24 218.100.48.151 1 100 0 BE AS\_PATH: 18229 12 202.65.156.0/24 218.100.48.151 1 100 0 BE AS\_PATH: 18229 13 202.65.157.0/24 218.100.48.151 1 100 0 BE AS\_PATH: 18229

So 7 prefixes at Mumbai and 13 at Chennai. None at Delhi and rest all 100 (along with these) via their transit providers - Tata, Reliance, Airtel, Tulip etc.  This tells the state of peering in India. Netmagic seems little better here. They are using AS17439. They are arouncing close to 200 prefixes in Global IPv4 routing table. They also participate at NIXI Mumbai, Delhi and Chennai. They are announcing 57 prefixes at Mumbai, 12 at Delhi and around 40 at Chennai. NIXI isn’t really supporting datacenters in any sense. Their concept that “requester pays” sucks badly. As per NIXI’s policy if you are a datacenter i.e if you send out 5x times more then what you receive, then you will not be paid by requesting ISP. So e.g if a small ISP having 100Mbps shared peering at NIXI Mumbai starts sucking lot of traffic say 20Mbps from Control S datacenter alone. So datacenter is sending at 20Mbps to that ISP. In other way datacenter would be still getting request of say 1/5th of size so 4Mbps. Thus datacenter is pulling off content from that ISP at 4Mbps for which it has to pay even when it’s peering! So for say 100Mbps connection, datacenter would be consuming say roughly 1/5th i.e 20Mbps, it will consume over  78TB of data annually. This turns out to be $18k per FE port in peering costs alone for sending domestic traffic to domestic users. This is an absurd!


Changes needed in NIXI and IXP part as whole:

  1. IXP sector should be open and there should be private players in IXP sector like Equinix, Coresite etc. Forget Walmart, please bring Equinix! I would love to see Airtel-Equinix rather then Bharti Walmart! :)
  2. NIXI should be open to content provider. Without content provider there’s just no one injecting the content into these networks.
  3. NIXI should end cost based on metering and big ISPs should be rather forced to peer with small ISPs. If destination traffic is for their or their downstream network, they should take it and route it across their network.
  4. More easy availability of layer 2 optical circuits to connect to NIXI fabric.

  Local time check: 4:37am here. Time for me to get some sleep! :)