06 Oct

F root server, Chennai down from 5 months. Who cares?

Time for a quick followup blog post. On 26th April of this year I blogged about broken connectivity of F root server which was hosted in NIXI Chennai. Apart from that blog post, I did informed ISC which operates F root (NIXI was host on behalf of them in India). In my open email on APNIC mailing list, I got a reply from Network Operations Center of ISC that they will verify and will take necessary action. Within 48 hours of that email they figured out root cause and since they couldn’t fix it right at that point, they pulled plug off from that root server.

This was one of 3 Global root DNS servers hosted in India. I am sad to post that till date they have not been able to turn server back live. No blame to ISC but this is how serious Indian bodies are about internet and infrastructure.

My current traceroute to F root:

traceroute to f.root-servers.net (, 30 hops max, 60 byte packets
1 router.local ( [AS1] 0.969 ms 1.168 ms 1.488 ms
2 ( [AS9829] 19.203 ms 20.001 ms 22.286 ms
3 ( [AS9829] 26.905 ms 28.801 ms 29.490 ms
4 ( [AS4755] 64.299 ms 66.175 ms 68.068 ms
5 ( [*] 95.702 ms ( [*] 96.813 ms 98.304 ms
6 ix-4-2.tcore1.CXR-Chennai.as6453.net ( [*] 304.038 ms 280.526 ms 280.544 ms
7 if-3-3.tcore2.CXR-Chennai.as6453.net ( [*] 330.969 ms if-5-2.tcore1.SVW-Singapore.as6453.net ( [*] 327.010 ms if-3-3.tcore2.CXR-Chennai.as6453.net ( [*] 333.282 ms
8 if-5-2.tcore2.SVW-Singapore.as6453.net ( [*] 319.188 ms if-2-2.tcore2.SVW-Singapore.as6453.net ( [*] 319.458 ms if-5-2.tcore2.SVW-Singapore.as6453.net ( [*] 341.489 ms
9 Vlan1870.icore1.HK2-HongKong.as6453.net ( [*] 339.646 ms Vlan1850.icore1.HK2-HongKong.as6453.net ( [*] 337.416 ms Vlan1779.icore1.HK2-HongKong.as6453.net ( [*] 338.317 ms
10 isc2-FE.hkix.net ( [AS2687/AS4862/AS9498/AS10026/AS1221] 340.247 ms 339.589 ms 344.179 ms
11 f.root-servers.net ( [AS55440/AS3557/AS23708/AS8167] 340.218 ms 341.172 ms 341.604 ms


So I am still hitting Hong Kong.

Please note ultra high latency here is due to usual old problem of BSNL that they have broken return paths. We can see that as soon as traffic is handed over to AS6453 on hop 6, there is a huge spike in latency. Since AS6453 – Tata has a publically available looking glass, I can traceroute back to my IP from there and see the path:


Router: gin-cfo-core1
Site: IN, Chennai – CFO, VSNL
Command: traceroute ip

Tracing the route to

1 if-11-0-2-0.tcore1.CXR-Chennai.as6453.net ( [MPLS: Label 613458 Exp 0] 268 msec
if-1-0-0-0.tcore1.CXR-Chennai.as6453.net ( [MPLS: Label 613458 Exp 0] 252 msec
if-1-3-0-0.tcore1.CXR-Chennai.as6453.net ( [MPLS: Label 613458 Exp 0] 280 msec
2 if-7-2.tcore1.MLV-Mumbai.as6453.net ( [MPLS: Label 508693 Exp 0] 248 msec 304 msec
if-3-3.tcore2.CXR-Chennai.as6453.net ( [MPLS: Label 557305 Exp 0] 400 msec
3 if-9-2.tcore2.MLV-Mumbai.as6453.net ( [MPLS: Label 320866 Exp 0] 412 msec 404 msec 404 msec
4 if-6-2.tcore1.L78-London.as6453.net ( [MPLS: Label 731443 Exp 0] 400 msec 248 msec
if-2-2.tcore2.WYN-Marseille.as6453.net ( [MPLS: Label 404482 Exp 0] 244 msec
5 if-2-2.tcore2.L78-London.as6453.net ( [MPLS: Label 515300 Exp 0] 256 msec 256 msec 256 msec
6 if-20-2.tcore2.NYY-NewYork.as6453.net ( [MPLS: Label 300800 Exp 0] 260 msec 268 msec 260 msec
7 if-9-0-0-19.mcore4.NYY-NewYork.as6453.net ( 252 msec 252 msec 252 msec
8 ix-14-2.mcore4.NYY-NewYork.as6453.net ( 484 msec 476 msec 480 msec
9 [AS 9829] 500 msec 612 msec 604 msec
10 [AS 9829] 624 msec 504 msec 504 msec
11 [AS 9829] 500 msec 504 msec 504 msec
12 * * *
13 * * *
14 * * *
15 * * *


So path is like Chennai > Mumbai > London > New York > back to India BSNL. This is completely due to negligence of BSNL. They are doing BGP announcement only at New York which is why India to Hong Kong packets go straight but return is via New York taking latency super high. Anyways this is separate issue on it’s own. Coming back on main issue of this post i.e F root server – it is yet not up and things are still “moving” but slowly. 


Looking at last week latency to F root server from home hosted RIPE NCC probe:



What exactly was cause of problem?

The cause of problem was forced MLP and regional only MLP. Here’s the exact NIXI’s policy which says:

An ISP at any NIXI node must at a minimum announce all its regional routes to the NIXI router at that NIXI location. All ISPs connecting to that NIXI node are entitled to receive these routes using a single BGP session with the NIXI router. This will guarantee the exchange of regional traffic within a NIXI node. This is referred to as forced regional multi-lateral peering under the policy.


Now ISC was running F root server without any transit and was relying completely on peering sessions in Chennai region. If you recall at that time problem effecting few networks only. For networks like Sify, IDEA Cellular it was all running well while for BSNL it was failing. The reason is when ISPs like BSNL participate at NIXI, they announce ONLY regional routes. So BSNL was getting BGP announcement of ISC which was sitting below NIXI Chennai router, while BSNL itself was announcing prefixes only at New Delhi (closest to Haryana) exchange and not at Chennai exchange. Since node was without any transit, it was not able to reach BSNL users outside Chennai at all (and so does for many other big ISPs). As of now ISC is working on deal with NIXI to get a basic transit pipe from STPI (well another Govt. ISP). Since it will be transit pipe, it will provide full global routing table feed including BSNL Haryana and other routes. 


This is truly an absurd that Indian Govt. is terribly slow with this critical part of Internet infrastructure and still has as high as $5 billion to invest in to connect Gram panchayats over fiber even when there’s no electricity to quite a few of them. The prime problem for now is that there are SOOOOO many Govt. departments dealing with “problem” that they themselves constitute a significant part of “problem”. There is terrible co-ordination between all these Govt. bodies & companies.


List of Govt. bodies involved in telecommunications:


So many departments. BSNL holds domestic fiber everywhere except metros where Govt. replies on MTNL. Then we have PowerGrid which puts fibers along with power lines and so does RailTel which does with Railway lines. I wonder why this work can’t be done via single body BSNL alone? Railtel has ambition of building Nation wide broadband network via RailWire project. I thought Govt. relied on BSNL’s 4.5lakh exchanges with over 100 pair of fiber capable of running 10G DWDM pipes for that!

Then on top of that we have NIXI which is all different i.e an IXP and not ISP and it has no direct relation with BSNL or other fiber holding bodies. STPI i.e State Technology Parks of India itself sounds like a funny name for an ISP but it exists and actually is more popular for layer 2 circuits nearby NIXI exchanges. NIC works to hold datacenters for Govt. websites (BSNL or none of other previous bodies have no clue how to run datacenter?) and then we get NKN which is running MPLS over BSNL+RailTel+Powergrid to provide 1Gbps connectivity to IITs + IIM’s + NIT’s etc. And if you are from a private small state college like mine – you can’t do much other then writing blog posts like these to yell out result frustration after years! 🙂

With hope we will have some better policies and governance, private sector will do way too better then these Govt. bodies. Time for me to get back to my work! 🙂

05 Oct

Understanding NIXI and it's policies

NIXI i.e National Internet Exchange of India is well known for it’s inefficiency and for its bad policies. I am posting this blog post to discuss some of them.

Bit of background:

NIXI is one (and only) Indian IXP i.e Internet Exchange Point established in 2003 so as to facilitate peering between Indian ISPs. Before this, there were lot of cases when Indian ISP’s were connecting to each other from outside India in Singapore and Europe. Thus NIXI established few exchanges in key cities where necessary infrastructure was provided to ISP’s to “peer”.  With peering, the strict technical meaning is that exchange of traffic between ISPs.

Understanding peering…

E.g if ISP A is peering with ISP B, ISP A will announce it’s and it’s downstream customer prefixes to B and so does B will do with A. If A is peering with C, and B is peering with D, C and D will NOT be able to reach to each other via A and B’s interconnection. While say if E is (downstream) customer of A and F is downstream customer of B, then there will be a route between E to F as E > A > B > F and vice versa (depending on prefixes announced). Thus, if big ISPs like Airtel and Tata peer at NIXI, it will help to keep traffic between them local + also their customers (not peers).

What is benefit of keeping traffic local?

A lot! 🙂
Firstly that’s way more efficient. Why to have packets to go from India > Europe > US > Singapore > India just because source and destination networks are not connected within India? Also, theoretically it will save us on bandwidth costs (in real it is not yet, which I will discuss in this post). Other then that there are huge technical advantages of keeping traffic i.e low latency which matters a lot when latency within India is usually less then 100ms, while with US it’s 300ms. And worst if we are hitting US twice in a round trip connection, we will get latency of over 500ms! Also, networks work lot faster on direct peering due to TCP window size factor apart from fact that having 4-5 networks between source and destination effects a lot.

So how much NIXI has done so far?

Well not much!
Let’s talk about some numbers. India has around 10 million broadband users and a little over 100 million users (so a lot still on slow non-broadband usage) as per this data. Aggregated traffic passing from all NIXI exchanges is around 20Gbps as per their official data. If we look at Europe, number of Internet users around 50 million as per this data, while aggregated traffic of Amsterdam Internet Exchange (AMX) is over 1.6Tbps alone! (Source). Other important exchanges like London Internet Exchange carry 1.4Tbps, while DE-CIX exchange in Germany carry around 2Tbps of aggrigated traffic. It’s very hard to count total data of all these exchanges but clearly AMX + LINX + DECIX alone carry 5Tbps i.e 5000Gbps of aggregated bandwidth which is 250 times more then that of Indian exchange NIXI. (I am still ignoring dozen of big exchanges here. You can find IXP directory at PCH’s website here)

At this point one can think of fact that is NIXI only exchange in India?

Well, yes. Indian “licence raj” as usual! Also at AMX, we can see around 514 connected networks, at LINX 435 members while at DECIX around 465 participants. At all NIXI we don’t see more then 30-40 connected networks! I can guess aggrigated number of unique networks to be say 50 or so.
Look at list of members in AMX and compare with NIXI. Some big names are missing at NIXI. Few of them are – Google, Akamai, Limelight, Microsoft, root servers networks, Verisign, Yahoo etc. One common thing among all these networks is the fact they are content providers and not really consumers. They send more traffic (way more) then what they receive. NIXI has a requirement that only ISPs can join NIXI and with ISP it means you must have license to operate in India. There’s no place for content provider! This is strange and an absurd because traffic between content consuming ISPs is very low as compared to content between content providers like CDN, datacenters, etc. Unless they join in the exchange, there won’t be really much traffic for ISP participants to consume.
At this point one can think of fact that if content provider like Google is connected to an NIXI participating ISP, won’t it be available at NIXI via that ISP?
OK! Let’s look at this. Google.co.in uses IPs from subnet. It is announcement by Google’s AS 15169.  Let’s look at this prefix at NIXI Delhi, Mumbai and Chennai using their looking glass.

NIXI Looking Glass – show ip bgp
Router: NIXI Delhi (Noida)
Command: show ip bgp
% Network not in table
NIXI Looking Glass – show ip bgp
Router: NIXI Mumbai
Command: show ip bgp
% Network not in table
NIXI Looking Glass – show ip bgp
Router: NIXI Chennai
Command: show ip bgp
show ip bgp
BGP4 : None of the BGP4 routes match the display condition

So – no where. One cannot reach Google AS15169 from NIXI and reason for that is the fact that Google just peers with Indian providers like Airtel, Tata & Reliance and doesn’t really has a transit relationship. Likely Google doesn’t pays to any of them and they only to Google’s Indian PoPs and caching servers.  Since Google has a peering relationship, ISPs do not share Google’s route with other peers. If we look at say Akamai India – I see they have one of DNS servers ns1-2.akam.net in India. It is using IP which is coming from BGP announcement of by AS 21342. Looking at prefix at NIXI Delhi:

NIXI Looking Glass – show ip bgp
Router: NIXI Delhi (Noida)
Command: show ip bgp
BGP routing table entry for, version 62228
Paths: (1 available, best #1, table Default-IP-Routing-Table)
Advertised to update-groups:
1 2 3
9498 20940 20940 20940 21342, (received & used) from (
Origin IGP, localpref 100, valid, external, best

Clearly route available via Airtel (AS 9498) and so does at NIXI Mumbai and Chennai too. Here Akamai is visible because Akamai seems to be a downstream customer of Bharti Airtel AS9498.
This is a problem! So if you are content provider, NIXI damages you as:

  • You cannot connect to NIXI’s peering fabric directly unless you have a license to operate in India (NIXI’s requirement here)
  • You cannot reach to NIXI via other ISPs unless you become their direct customers. No transit routes, only peering routes. This is normal for an exchange though.

Also problem is just not limited to this. Another issue at NIXI is cost factor. It’s really terrible how economics work here. As per NIXI’s routing & tariff policy, there is really crazy pricing.

Counting on costs…

Joining charges – 1000Rs (just $20!), port charges for 100Mbps around $2000, while Gig port costs around $5000 yearly. Comparing it to AMX they don’t seem selling less then 500Mbps directly (well ….) while for 1Gbps it costs 500 Euro monthly which will around $7800 per year. So NIXI seems cheap? Well no! There’s a catch here that NIXI is likely world’s only exchange which has concept of charging on data transferred too!
It goes as for exchange of traffic between ISP A and ISP B, the “requester pays” logic goes. So if B’s customer are pulling off data from A then B will have to pay 12Rs/GB to ISP A. Now here catch is if you are a datacenter (well yeah there are datacenters too connected to NIXIs with ISP license) then requester pays concept doesn’t works but datacenter does has to pay for it’s own downstream.

How does cost is so damaging for a small ISP?

Let’s assume if a small ISP connects to NIXI. Since it is small and has consumers who are just pulling data, it would have to pay 12Rs/GB to big old incumbent operators while they would be taking relatively very less content from that network and would be paying negligible in return (apart from fact that small ISP would be surely using one of them for Internet transit to have global reach ability). So for say a 100Mbps connection in full use, it will transfer around 8640000 Mega bits in one day or 394200000 Mega bytes in one year which is around 394200 GB. At 12Rs/GB pull off costs, it will be around $94k yearly bill! Apart from  $2000 port charges! 🙂
So, we are talking about partial routing tables with almost no content provider and paying around $96k for 100Mbps peering capacity. That too when retail ISP would be likely selling connections at around $10/user on average. So we are talking about atleast 9600 users alone to cover cost of peering apart from layer 2 circuit cost to reach NIXI and surely a HUGE costs of Internet transit to cover rest of non-NIXI routes of global Internet.
Also, if we look at cost of $96k for 100Mbps, it might be little less due to fact that consumers will upload a bit during that time and ISP will also earn. So doing a rough assuming ISP would be paying atleast $80k  in real world case when burst port completely. So $80k for 100Mbps means $800 for each mega bit per second. One can easily get a STM 1 i.e OC3 circuit (for American readers!) for around 25 lakh which will be $50,000. I have ignored fact that ISP has not to pay if it is taking content from datacenters because there are simply not many datacenters in list. Effect will be negligible here!
So we are looking at STM 1 – 155Mbps Global transit bandwidth for $50,000 Vs NIXI peering partial routes with missing major content providers for around $80,000 an year. Amazing?
As you might have already guessed – one can get way more cheaper transit bandwidth from ISPs then reaching (only) them via NIXI. Also, not to forget that transit matters a lot because lot of content is still in US and not hosted in India. Also providers like Google who are building their own network have their own PoPs at Delhi, Mumbai and Chennai and one can connect there directly rather then reaching via NIXI (Google’s Peering info map here). So clearly NIXI is failing heavily here.
Also, if we talk about datacenters like Control S and Netmagic who hold ISP license also and are connected to NIXI.  If we look at Control S AS18229 – they are announcing hundreds of prefixes (check here) in global IPv4 table. They participate at NIXI Delhi, Mumbai and Chennai with name Pioneer E labs and use same ASN.
Looking at their prefixes at each of these exchanges via NIXI’s looking glass:

NIXI Looking Glass – show ip bgp neighbors routes
Router: NIXI Mumbai
Command: show ip bgp neighbors routes
BGP table version is 5482108, local router ID is
Status codes: s suppressed, d damped, h history, * valid, > best, i – internal,
S Stale
Origin codes: i – IGP, e – EGP, ? – incomplete
Network Next Hop Metric LocPrf Weight Path
*> 0 0 18229 i
*> 0 0 18229 i
*> 0 0 18229 i
*> 0 0 18229 i
*> 0 0 18229 i
*> 0 0 18229 i
*> 0 0 18229 i
Total number of prefixes 7
NIXI Looking Glass – show ip bgp neighbors routes
Router: NIXI Delhi (Noida)
Command: show ip bgp neighbors routes
NIXI Looking Glass – show ip bgp neighbors received-routes
Router: NIXI Chennai
Command: show ip bgp neighbors received-routes
show ip bgp neighbors received-routes
There are 13 received routes from neighbor
Searching for matching routes, use ^C to quit…
Prefix Next Hop Metric LocPrf Weight Status
1 1 100 0 BE
AS_PATH: 18229
2 1 100 0 BE
AS_PATH: 18229
3 1 100 0 BE
AS_PATH: 18229
4 1 100 0 BE
AS_PATH: 18229
5 1 100 0 BE
AS_PATH: 18229
6 1 100 0 BE
AS_PATH: 18229
7 1 100 0 BE
AS_PATH: 18229
8 1 100 0 BE
AS_PATH: 18229
9 1 100 0 BE
AS_PATH: 18229
10 1 100 0 BE
AS_PATH: 18229
11 1 100 0 BE
AS_PATH: 18229
12 1 100 0 BE
AS_PATH: 18229
13 1 100 0 BE
AS_PATH: 18229

So 7 prefixes at Mumbai and 13 at Chennai. None at Delhi and rest all 100 (along with these) via their transit providers – Tata, Reliance, Airtel, Tulip etc.  This tells the state of peering in India.
Netmagic seems little better here. They are using AS17439. They are arouncing close to 200 prefixes in Global IPv4 routing table. They also participate at NIXI Mumbai, Delhi and Chennai. They are announcing 57 prefixes at Mumbai, 12 at Delhi and around 40 at Chennai.
NIXI isn’t really supporting datacenters in any sense. Their concept that “requester pays” sucks badly. As per NIXI’s policy if you are a datacenter i.e if you send out 5x times more then what you receive, then you will not be paid by requesting ISP. So e.g if a small ISP having 100Mbps shared peering at NIXI Mumbai starts sucking lot of traffic say 20Mbps from Control S datacenter alone. So datacenter is sending at 20Mbps to that ISP. In other way datacenter would be still getting request of say 1/5th of size so 4Mbps. Thus datacenter is pulling off content from that ISP at 4Mbps for which it has to pay even when it’s peering!
So for say 100Mbps connection, datacenter would be consuming say roughly 1/5th i.e 20Mbps, it will consume over  78TB of data annually. This turns out to be $18k per FE port in peering costs alone for sending domestic traffic to domestic users. This is an absurd!

Changes needed in NIXI and IXP part as whole:

  1. IXP sector should be open and there should be private players in IXP sector like Equinix, Coresite etc. Forget Walmart, please bring Equinix! I would love to see Airtel-Equinix rather then Bharti Walmart! 🙂
  2. NIXI should be open to content provider. Without content provider there’s just no one injecting the content into these networks.
  3. NIXI should end cost based on metering and big ISPs should be rather forced to peer with small ISPs. If destination traffic is for their or their downstream network, they should take it and route it across their network.
  4. More easy availability of layer 2 optical circuits to connect to NIXI fabric.

Local time check: 4:37am here. Time for me to get some sleep! 🙂