25 Oct

NIXI root DNS servers and updates

Has been a while since I checked the status of root servers which are hosted at NIXI. The list as per their official member list stays the same i.e i root in Mumbai, K root in Noida and F root in Chennai. 

i root seems to be up!

show ip bgp neighbors 218.100.48.75 received-routes
       There are 5 received routes from neighbor 218.100.48.75
Searching for matching routes, use ^C to quit...
Status A:AGGREGATE B:BEST b:NOT-INSTALLED-BEST C:CONFED_EBGP D:DAMPED
       E:EBGP H:HISTORY I:IBGP L:LOCAL M:MULTIPATH m:NOT-INSTALLED-MULTIPATH
       S:SUPPRESSED F:FILTERED s:STALE
       Prefix             Next Hop        MED        LocPrf     Weight Status
1      192.36.148.0/24    218.100.48.75   0          100        0      BE    
         AS_PATH: 8674 29216
2      194.58.198.0/24    218.100.48.75   0          100        0      BE    
         AS_PATH: 8674 56908
3      194.58.199.0/24    218.100.48.75   0          100        0      BE    
         AS_PATH: 8674 56908
4      194.146.106.0/24   218.100.48.75   0          100        0      BE    
         AS_PATH: 8674
5      194.146.107.0/24   218.100.48.75   0          100        0      BE    
         AS_PATH: 8674

K root seems to be down!

Router: NIXI Delhi (Noida)

Command: show ip bgp neighbors 218.100.48.6 received-routes


show ip bgp neighbors 218.100.48.6 received-routes
Inbound soft reconfiguration not enabled for neighbor 218.100.48.6

F root seems to be up!

show ip bgp neighbors 218.100.48.135 received-routes
       There are 1 received routes from neighbor 218.100.48.135
Searching for matching routes, use ^C to quit...
Status A:AGGREGATE B:BEST b:NOT-INSTALLED-BEST C:CONFED_EBGP D:DAMPED
       E:EBGP H:HISTORY I:IBGP L:LOCAL M:MULTIPATH m:NOT-INSTALLED-MULTIPATH
       S:SUPPRESSED F:FILTERED s:STALE
       Prefix             Next Hop        MED        LocPrf     Weight Status
1      192.5.5.0/24       218.100.48.135  10         100        0      ME    
         AS_PATH: 24049 3557 3557

Atleast while 2 out of 3 root servers seems to be up but for some reason my connection in Haryana isn’t  hitting i root. F root instance it is taking me there for sure. 

i root latency check from my home: 

ping -c 5 i.root-servers.net.
PING i.root-servers.net (192.36.148.17) 56(84) bytes of data.
64 bytes from i.root-servers.net (192.36.148.17): icmp_seq=1 ttl=52 time=156 ms
64 bytes from i.root-servers.net (192.36.148.17): icmp_seq=2 ttl=52 time=155 ms
64 bytes from i.root-servers.net (192.36.148.17): icmp_seq=3 ttl=52 time=155 ms
64 bytes from i.root-servers.net (192.36.148.17): icmp_seq=4 ttl=52 time=156 ms
64 bytes from i.root-servers.net (192.36.148.17): icmp_seq=5 ttl=52 time=155 ms

--- i.root-servers.net ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4004ms
rtt min/avg/max/mdev = 155.817/156.034/156.481/0.551 ms

That’s clearly too high latency. Latency from my location to Mumbai is typically 30-40ms. Let’s trace to the i root. 

traceroute i.root-servers.net.
traceroute to i.root-servers.net. (192.36.148.17), 30 hops max, 60 byte packets
 1  172.16.0.1 (172.16.0.1)  0.590 ms  0.664 ms  0.774 ms
 2  103.201.140.218 (103.201.140.218)  3.500 ms  3.413 ms  3.495 ms
 3  10.10.26.1 (10.10.26.1)  6.182 ms  6.304 ms  6.012 ms
 4  10.10.26.9 (10.10.26.9)  6.318 ms  6.047 ms  5.964 ms
 5  nsg-static-77.249.75.182-airtel.com (182.75.249.77)  45.636 ms  44.225 ms  44.144 ms
 6  182.79.191.89 (182.79.191.89)  56.411 ms 182.79.181.218 (182.79.181.218)  59.690 ms 182.79.153.86 (182.79.153.86)  66.090 ms
 7  182.79.149.95 (182.79.149.95)  207.319 ms 182.79.217.94 (182.79.217.94)  57.670 ms 182.79.149.95 (182.79.149.95)  207.285 ms
 8  182.79.177.101 (182.79.177.101)  187.845 ms 182.79.224.134 (182.79.224.134)  183.999 ms 182.79.224.124 (182.79.224.124)  180.657 ms
 9  182.79.146.218 (182.79.146.218)  211.258 ms 182.79.154.2 (182.79.154.2)  187.929 ms 182.79.154.10 (182.79.154.10)  192.907 ms
10  182.79.149.103 (182.79.149.103)  183.405 ms  181.645 ms  181.540 ms
11  peering.r1.lnx.dnsnode.net (195.66.225.151)  157.300 ms  157.214 ms  157.293 ms
12  i.root-servers.net (192.36.148.17)  157.364 ms  156.423 ms *

Thus Airtel is taking me to all the way to London (while LNX = Airport code for Smolensk Airport, Smolensk, Russia but route clearly shows it’s being exchanged at LINX. Someone in Netnod got into habit of writing LINX as LNX which is confussing). 

I see the same by querying id.server and hostname.bind in CHAOS class.

dig chaos @192.36.148.17 id.server txt  +short
"s1.lnx"

dig chaos @192.36.148.17 hostname.bind  txt  +short
"s1.lnx"

So, for now, Airtel is preferring route learnt via LINX peering over route learnt at NIXI. In a check by all Indian RIPE Atlas probes, I see that out of 50 RIPE Atlas probes, 23 are hitting s1.mum in Mumbai, 19 are hitting LINX London (s1.lnx) and 1 (which is hosted on NKN) is hitting s1.amx in Amsterdam (json data here). 

Why this happens? 

It’s often the lack of peering and/or case of prefered routes. For smaller networks, it’s simply missing peering. For larger networks, it’s about which route they prefer, which not. Here’s a view of networks with their ASNs sorted by latency (wherever RIPE Atlas Probe) was present (measurement link here). 

So what can be done about it? 

NIXI needs to be more attractive to various (smaller) networks which clearly it is not since it just does not has any content player connected to it due to policy issue. Furthermore customers of Airtel need to buzz it and request for a better route to i root’s local instance. 

Comments & thoughts expressed in the post are personal and have nothing to do with my employer. I am also volunteering for supporting tech platform for BharatIX to facilitate peering.

06 Jul

ISC F root server – IPv6 issue at NIXI Chennai

Last week I noticed that F root was showing poor connectivity with Indian RIPE Atlas probes for F-root. The graph looked really terrible.

 

 

 

I traced to it from one of RIPE Atlas probes and saw this trace:

Probe #6107

  1 2401:7500:fff0:1::1                      0.838 ms     0.747 ms     0.632 ms
  2 2400:5200:1c00:d::1                      1.755 ms     1.745 ms     1.726 ms
  3 2403:0:100::2be                          2.089 ms     2.054 ms     2.049 ms
  4 2404:a800:2a00::13d                     45.589 ms    26.274 ms     33.64 ms
  5 2404:a800::178                          26.376 ms    25.406 ms    25.276 ms
  6 2001:de8:1:2::3                         25.363 ms    25.232 ms    25.223 ms
  7 *                                               *            *            *
  8 *                                               *            *            *
  9 *                                               *            *            *
 10 *                                               *            *            *
 11 *                                               *            *            *

 

Here the last hop before timeout i.e hop 6 is of NIXI Chennai peering subnet 2001:de8:1:2::/64. As soon as I saw it, it reminded me older issue which happened and broke IPv4 connectivity to root DNS servers. I blogged about it here, here and here. So the problem remains that NIXI is broken cost wise due to charge on in – out policy. This leads to people accepting routes at all NIXI’s but they do not announce their routes. Thus return path is broken and essentially traffic is being blackholed. Earlier this issue was fixed by adding IP transit support to these root DNS servers so that a default route stays in case of all other failures.

It seems like same is missing in IPv4 world and routes are not being announced.

During this time, I saw two BGP sessions at NIXI Chennai for F root:

2001:de8:1:2::3 24049 ESTAB 25d 3h10m 1 0 2263 0
2001:de8:1:2::4 24049 ESTAB 125d18h30m 1 0 2264 0

 

Both were announcing prefix covering F root server’s pool:

show ipv6 bgp neighbors 2001:de8:1:2::3 received-routes
       There are 1 received routes from neighbor 2001:de8:1:2::3
Searching for matching routes, use ^C to quit...
Status A:AGGREGATE B:BEST b:NOT-INSTALLED-BEST C:CONFED_EBGP D:DAMPED
       E:EBGP H:HISTORY I:IBGP L:LOCAL M:MULTIPATH m:NOT-INSTALLED-MULTIPATH
       S:SUPPRESSED F:FILTERED s:STALE
       Prefix             Next Hop        MED        LocPrf     Weight Status
1      2001:500:2f::/48   2001:de8:1:2::3 10         100        0      BE    
         AS_PATH: 24049 3557


show ipv6 bgp neighbors 2001:de8:1:2::4 received-routes
       There are 1 received routes from neighbor 2001:de8:1:2::4
Searching for matching routes, use ^C to quit...
Status A:AGGREGATE B:BEST b:NOT-INSTALLED-BEST C:CONFED_EBGP D:DAMPED
       E:EBGP H:HISTORY I:IBGP L:LOCAL M:MULTIPATH m:NOT-INSTALLED-MULTIPATH
       S:SUPPRESSED F:FILTERED s:STALE
       Prefix             Next Hop        MED        LocPrf     Weight Status
1      2001:500:2f::/48   2001:de8:1:2::4 10         100        0      E     
         AS_PATH: 24049 3557

 

I posted about it on SANOG and APNIC mailing list. Though there hasn’t been any reply by ISC, Sunny from APNIC passed info to them and I noticed that prefix announcement from NIXI has been withdrawn. Connectivity to F root now works to the instances outside India.

 

 

 

Waiting to hear from ISC as of now. Time to get back to work!

 

12 Oct

UKNOF32 – Analysis of F-root placement using RIPE Atlas

Enjoyed ISC’s presentation about their analysis of F root server (one 13 root DNS servers which power the Internet) about anycast performance gloablly for 192.5.5.0/24 announced (and anycasted) by AS3557 (ISC). This was presentation at UKNOF 32.

Embedded presentation below (or click here to watch on YouTube directly)