10 Sep

NIXI permits content players!

I am in Chiang Mai, Thailand for APNIC 48 conference. Earlier today attended APIX meeting where many IX members from Asian community gave an update including NIXI i.e National Internet Exchange of India.

As per the update NIXI now allows content players to peer at the exchange. NIXI earlier had a strict requirement of telecom license for anyone to peer but as of now it allows anyone with IP address and AS number to be part of the exchange just like all other exchanges. This is a really good development coming this year after their announcement of the removal of x-y charge. One strange thing remains that their website is still not updated to reflect that which is probably just work in progress. As per representative from NIXI they now openly welcome all content players to peer at NIXI.

What more needs to be done?

Well, a couple more changes are needed.
Here’s what I requested to the representative from NIXI:

  1. Removal of forced multi-lateral peering policy. Forced multilateral is bad idea in modern times. Many large networks (especially the ones dealing with anycast based service) would usually not like to peer at route server.
  2. So far NIXI has discouraged bilateral BGP sessions in the policy. Technically any members can create bilateral sessions but were denied in the policy. That needs to be changed. Bilateral/multilateral peering is a technical decision and should be left to individual networks operators.
  3. NIXI needs to migrate to software-based route servers like bird with auto-config generation to include features like IRR filtering, RPKI filtering, BGP communities support and much more.

Remove forced multilateral, but what about Indian incumbents?

This is an old interesting discussion. Basically due to “forced” multilateral peering policy – everyone is on route servers and that includes incumbents like Tata Communications, Airtel, BSNL and other large networks like Jio and Voda/IDEA as well. The argument there is if multi-lateral peering is not forced, local ISPs won’t be able to peer with these networks. Part of these arguments comes from a mindset which still believes the world’s traffic flows from tier 1 operators to tier 2 and tier 3. A very large part of modern internet traffic flows from content players to eyeball networks (where “eye balls” are). Content players deliver this traffic through a mix of backbone + peering as well as by putting caching nodes inside the ISPs network (like Google’s GGC, Facebook’s FNA, Netflix OCA, Akamai caching node etc). The success of an IX depends on meeting the content players and eyeball players. To connect eyeball players (which are usually spread across the region) one needs circuits i.e dark fibre, DWDM waves, or any sort of transport network. As long as these three things are in place, IX can be a success.

Look at any large IX doing over few hundred Gigabits of traffic or even terabit of traffic and ask does local incumbent telco peers openly at that IX? Does BT openly peers at LINX in London? Do Deutsche Telekom peers openly at DECIX Frankfurt? Does AT&T, Verizon, Comcast peer openly at various Equinix and Coresite exchanges spread across the US? Answer to most of these is no and that is just fine. I would personally hope that these networks do but if they do not it’s not something which blocks the development. In the same way, having Airtel/Tata/Jio/Voda-IDEA at NIXI is great and I would hope they stay but the success of NIXI does not depend on these. As long as transport circuits are available (which they are!) from enough players with competition it would be fine. As of now across India, one can take transport circuit from Airtel, Tata, Voda/IDEA, Powergrid. Railtel, Reliance Communications and more!

Forced multilateral and routing issues…

One recent issue which E2E Networks (a cloud provider based in India) faced was to send traffic to Jio. Suddenly traffic going from E2E to Jio via Tata Comm was going from outside India. As per discussions, routing was tweaked to send traffic out to Jio via Airtel and still, there were issues of routing from outside India. The issue went on for a couple of days and was eventually fixed by speaking to both ends. Issue was there due to ongoing peering issues between Jio – Tata Comm and Jio – Airtel.

Now the funny thing here is that Netmagic (which takes care of routing for E2E) connects to NIXI and Jio also connects to NIXI. During this entire time, both Netmagic (on behalf of E2E) and Jio had enough capacity at NIXI but that could not be utilised because of forced multilateral peering policy. If routes were announced to NIXI route server, that would have probably fixed the Jio issue but would also attract traffic from Airtel/Tata which was not specifically desired due to their known port capacity issues at NIXI.

Ending this post with my quote which I often give in these discussions – “Remember somewhere up the transit path there’s always peering!” 🙂

Disclaimer:

  1. This post is done in my personal capacity and nowhere reflect the views of my employer.
  2. I am on board of the mentioned organisation E2E Networks (more on it here)

05 Apr

Tata – Airtel domestic peering IRR filtering and OpenDNS latency!

Last month I noticed quite high latency with Cisco’s OpenDNS from my home fibre connection. The provider at home is IAXN (AS134316) which is peering with content folks in Delhi besides transit from Airtel.

ping -c 5 208.67.222.222
PING 208.67.222.222 (208.67.222.222) 56(84) bytes of data.
64 bytes from 208.67.222.222: icmp_seq=1 ttl=51 time=103 ms
64 bytes from 208.67.222.222: icmp_seq=2 ttl=51 time=103 ms
64 bytes from 208.67.222.222: icmp_seq=3 ttl=51 time=103 ms
64 bytes from 208.67.222.222: icmp_seq=4 ttl=51 time=103 ms
64 bytes from 208.67.222.222: icmp_seq=5 ttl=51 time=103 ms


--- 208.67.222.222 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4005ms
rtt min/avg/max/mdev = 103.377/103.593/103.992/0.418 ms

This is bit on the higher side as from Haryana to Mumbai (OpenDNS locations list here). My ISP is backhauling from Faridabad which is probably 6-8ms away from my city and 2-3ms further to Delhi and from there to Mumbai around 30ms. Thus latency should be around ~40-45ms.

Here’s how forward trace looked like

traceroute 208.67.222.222
traceroute to 208.67.222.222 (208.67.222.222), 30 hops max, 60 byte packets
 1  172.16.0.1 (172.16.0.1)  0.730 ms  0.692 ms  0.809 ms
 2  axntech-dynamic-218.140.201.103.axntechnologies.in (103.201.140.218)  4.904 ms  4.314 ms  4.731 ms
 3  10.10.26.1 (10.10.26.1)  6.000 ms  6.414 ms  6.326 ms
 4  10.10.26.9 (10.10.26.9)  6.836 ms  7.135 ms  7.047 ms
 5  nsg-static-77.249.75.182-airtel.com (182.75.249.77)  9.344 ms  9.416 ms  9.330 ms
 6  182.79.243.201 (182.79.243.201)  62.274 ms 182.79.177.69 (182.79.177.69)  66.874 ms 182.79.239.193 (182.79.239.193)  61.297 ms
 7  121.240.1.201 (121.240.1.201)  85.789 ms  82.250 ms  79.591 ms
 8  172.25.81.134 (172.25.81.134)  110.049 ms 172.31.29.245 (172.31.29.245)  114.350 ms  113.673 ms
 9  172.31.133.210 (172.31.133.210)  112.598 ms 172.19.138.86 (172.19.138.86)  114.889 ms 172.31.133.210 (172.31.133.210)  113.415 ms
10  115.110.234.50.static.mumbai.vsnl.net.in (115.110.234.50)  125.770 ms  125.056 ms  123.779 ms
11  resolver1.opendns.com (208.67.222.222)  113.648 ms  115.044 ms  106.066 ms

Forward trace looks fine except that latency jumps as soon as we hit Tata AS4755 backbone. OpenDNS connects with Tata AS4755 inside India and announces their anycast prefixes to them. If the forward trace is logically correct but has high latency, it often reflects the case of bad return path. Thus I requested friends at OpenDNS to share the return path towards me. As expected, it was via Tata AS6453 Singapore.

Here’s what Tata AS4755 Mumbai router had for IAXN prefix:

BGP routing table entry for 14.102.188.0/22 
Paths: (1 available, best #1, table Default-IP-Routing-Table) 
Not advertised to any peer 
6453 9498 134316 134316 134316 134316 134316 134316 134316 134316 134316 134316 
192.168.203.194 from 192.168.199.193 (192.168.203.194) 
Origin IGP, localpref 62, valid, internal, best 
Community: 4755:44 4755:97 4755:888 4755:2000 4755:3000 4755:47552 6453:50 6453:3000 6453:3400 6453:3402 
Originator: 192.168.203.194, Cluster list: 192.168.199.193 192.168.194.15 
Last update: Mon Mar 25 15:26:36 2019

Thus what was happening is this:

Forward path: IAXN (AS134316) > Airtel (AS9498) > Tata (AS4755) > OpenDNS (AS36692)

Return path: OpenDNS (AS36692) > Tata (AS4755) > Tata (AS6453) > Airtel (AS9498) > IAXN (AS134316)

While this may seem like a Tata – Airtel routing issue but it wasn’t. I could see some of the prefixes with a direct path as well. Here’s a trace from Tata AS4755 Mumbai PoP to an IP from a different pool of IAXN:

traceroute to 103.87.46.1 (103.87.46.1), 15 hops max, 60 byte packets
1 * * *
2 172.31.170.210 (172.31.170.210) 0.911 ms 0.968 ms 0.643 ms
3 172.23.78.233 (172.23.78.233) 1.233 ms 0.821 ms 0.810 ms
4 172.17.125.249 (172.17.125.249) 23.540 ms 23.454 ms 23.367 ms
5 115.110.232.174.static.Delhi.vsnl.net.in (115.110.232.174) 49.175 ms 48.832 ms 49.107 ms
6 182.79.153.87 (182.79.153.87) 48.777 ms 182.79.153.83 (182.79.153.83) 49.043 ms 182.79.177.127 (182.79.177.127) 54.879 ms
7 103.87.46.1 (103.87.46.1) 60.865 ms 60.540 ms 60.644 ms

This clearly was fine. So why Tata was treating 103.87.46.0/24 different from 14.102.188.0/22? The reason for that lies in following:

  • Airtel (AS9498) very likely peers with Tata (AS4755). They do interconnect for sure as we see in traceroutes and my understanding is that it’s based on settlement-free peering for Indian traffic.
  • Airtel (AS9498) buys IP transit from Tata (AS6453) (besides a few others). Tata AS6453 is carrying the routing announcements to other networks in the transit free zone and that confirms that Airtel (at least technically) has a downstream customer relationship here.
  • Tata (AS4755) has IRR based filters on peering but not the Tata (AS6453) for it’s downstream. Hence while Tata rejected the route in India, they did accept that in Singapore PoP.
  • My IP was from prefix 14.102.188.0/22 and there was no valid route object for it at any of key IRRs like ATLDB, APNIC or RADB. But other prefix 103.87.46.0/24 did had a valid route object on APNIC.

Now after almost 10 days of it, my ISP has changed the BGP announcement and announcing 14.102.189.0/24 (which does a valid route object on APNIC). This fixes the routing problem and give me pretty decent latency with OpenDNS:

ping -c 5 208.67.222.222
PING 208.67.222.222 (208.67.222.222): 56 data bytes
64 bytes from 208.67.222.222: icmp_seq=0 ttl=55 time=52.552 ms
64 bytes from 208.67.222.222: icmp_seq=1 ttl=55 time=53.835 ms
64 bytes from 208.67.222.222: icmp_seq=2 ttl=55 time=53.330 ms
64 bytes from 208.67.222.222: icmp_seq=3 ttl=55 time=52.700 ms
64 bytes from 208.67.222.222: icmp_seq=4 ttl=55 time=52.504 ms

--- 208.67.222.222 ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 52.504/52.984/53.835/0.518 ms

So if you are a network operator and originating prefixes, please do document them in any of the IRRs. You can do that via IRR of your RIR (APNIC, ARIN etc) or a free IRR like ALTDB. If you have downstreams, make sure to create AS SET, add downstreams ASNs in your AS SET and also include that AS SET on peeringdb for the world to see!

Misc Notes

  • Posted strictly in my personal capacity and has nothing to do with my employrer.
  • Thanks for folks from Cisco/OpenDNS for quick replies with relevant data which helped in troubleshooting. 🙂
26 Feb

NIXI finally removing the x-y charge!

It’s kind of fun times in India with many IX’es showing up and now NIXI finally removing their traffic wise charge.

I came across this email which they sent to their member networks recently:

From: I X <ix@nixi.in>
Sent: Thu, 21 Feb 2019 12:48:31 GMT+0530
To: “members ” <members@nixi.in>
Cc: “ceo@nixi.in” <ceo@nixi.in>
Subject: Change in Data Transfer Charges (X-Y)

Dear Members,

In reference to the Board’s direction to reduce the current data transfer (X-Y) charges from Re. 1/GB to Rs. 0/GB. The said reduction in X-Y charges to Rs. 0/GB is being implemented with effect from 1st March 2019.
This is for your kind information please.

Thanks & Regards.
National Internet Exchange of India (NIXI)
www.nixi.in

This is really good. I hope next they start allowing the content networks i.e they do not demand for an ISP license to anyone who wants to connect to the IX.