Netnod | anuragbhatia.com

25 Oct

NIXI root DNS servers and updates

Has been a while since I checked the status of root servers which are hosted at NIXI. The list as per their official member list stays the same i.e i root in Mumbai, K root in Noida and F root in Chennai.

i root seems to be up!

show ip bgp neighbors 218.100.48.75 received-routes
       There are 5 received routes from neighbor 218.100.48.75
Searching for matching routes, use ^C to quit...
Status A:AGGREGATE B:BEST b:NOT-INSTALLED-BEST C:CONFED_EBGP D:DAMPED
       E:EBGP H:HISTORY I:IBGP L:LOCAL M:MULTIPATH m:NOT-INSTALLED-MULTIPATH
       S:SUPPRESSED F:FILTERED s:STALE
       Prefix             Next Hop        MED        LocPrf     Weight Status
1      192.36.148.0/24    218.100.48.75   0          100        0      BE    
         AS_PATH: 8674 29216
2      194.58.198.0/24    218.100.48.75   0          100        0      BE    
         AS_PATH: 8674 56908
3      194.58.199.0/24    218.100.48.75   0          100        0      BE    
         AS_PATH: 8674 56908
4      194.146.106.0/24   218.100.48.75   0          100        0      BE    
         AS_PATH: 8674
5      194.146.107.0/24   218.100.48.75   0          100        0      BE    
         AS_PATH: 8674

K root seems to be down!

Router: NIXI Delhi (Noida)

Command: show ip bgp neighbors 218.100.48.6 received-routes


show ip bgp neighbors 218.100.48.6 received-routes
Inbound soft reconfiguration not enabled for neighbor 218.100.48.6

F root seems to be up!

show ip bgp neighbors 218.100.48.135 received-routes
       There are 1 received routes from neighbor 218.100.48.135
Searching for matching routes, use ^C to quit...
Status A:AGGREGATE B:BEST b:NOT-INSTALLED-BEST C:CONFED_EBGP D:DAMPED
       E:EBGP H:HISTORY I:IBGP L:LOCAL M:MULTIPATH m:NOT-INSTALLED-MULTIPATH
       S:SUPPRESSED F:FILTERED s:STALE
       Prefix             Next Hop        MED        LocPrf     Weight Status
1      192.5.5.0/24       218.100.48.135  10         100        0      ME    
         AS_PATH: 24049 3557 3557

Atleast while 2 out of 3 root servers seems to be up but for some reason my connection in Haryana isn’t hitting i root. F root instance it is taking me there for sure.

i root latency check from my home:

ping -c 5 i.root-servers.net.
PING i.root-servers.net (192.36.148.17) 56(84) bytes of data.
64 bytes from i.root-servers.net (192.36.148.17): icmp_seq=1 ttl=52 time=156 ms
64 bytes from i.root-servers.net (192.36.148.17): icmp_seq=2 ttl=52 time=155 ms
64 bytes from i.root-servers.net (192.36.148.17): icmp_seq=3 ttl=52 time=155 ms
64 bytes from i.root-servers.net (192.36.148.17): icmp_seq=4 ttl=52 time=156 ms
64 bytes from i.root-servers.net (192.36.148.17): icmp_seq=5 ttl=52 time=155 ms

--- i.root-servers.net ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4004ms
rtt min/avg/max/mdev = 155.817/156.034/156.481/0.551 ms

That’s clearly too high latency. Latency from my location to Mumbai is typically 30-40ms. Let’s trace to the i root.

traceroute i.root-servers.net.
traceroute to i.root-servers.net. (192.36.148.17), 30 hops max, 60 byte packets
 1  172.16.0.1 (172.16.0.1)  0.590 ms  0.664 ms  0.774 ms
 2  103.201.140.218 (103.201.140.218)  3.500 ms  3.413 ms  3.495 ms
 3  10.10.26.1 (10.10.26.1)  6.182 ms  6.304 ms  6.012 ms
 4  10.10.26.9 (10.10.26.9)  6.318 ms  6.047 ms  5.964 ms
 5  nsg-static-77.249.75.182-airtel.com (182.75.249.77)  45.636 ms  44.225 ms  44.144 ms
 6  182.79.191.89 (182.79.191.89)  56.411 ms 182.79.181.218 (182.79.181.218)  59.690 ms 182.79.153.86 (182.79.153.86)  66.090 ms
 7  182.79.149.95 (182.79.149.95)  207.319 ms 182.79.217.94 (182.79.217.94)  57.670 ms 182.79.149.95 (182.79.149.95)  207.285 ms
 8  182.79.177.101 (182.79.177.101)  187.845 ms 182.79.224.134 (182.79.224.134)  183.999 ms 182.79.224.124 (182.79.224.124)  180.657 ms
 9  182.79.146.218 (182.79.146.218)  211.258 ms 182.79.154.2 (182.79.154.2)  187.929 ms 182.79.154.10 (182.79.154.10)  192.907 ms
10  182.79.149.103 (182.79.149.103)  183.405 ms  181.645 ms  181.540 ms
11  peering.r1.lnx.dnsnode.net (195.66.225.151)  157.300 ms  157.214 ms  157.293 ms
12  i.root-servers.net (192.36.148.17)  157.364 ms  156.423 ms *

Thus Airtel is taking me to all the way to London (while LNX = Airport code for Smolensk Airport, Smolensk, Russia but route clearly shows it’s being exchanged at LINX. Someone in Netnod got into habit of writing LINX as LNX which is confussing).

I see the same by querying id.server and hostname.bind in CHAOS class.

dig chaos @192.36.148.17 id.server txt  +short
"s1.lnx"

dig chaos @192.36.148.17 hostname.bind  txt  +short
"s1.lnx"

So, for now, Airtel is preferring route learnt via LINX peering over route learnt at NIXI. In a check by all Indian RIPE Atlas probes, I see that out of 50 RIPE Atlas probes, 23 are hitting s1.mum in Mumbai, 19 are hitting LINX London (s1.lnx) and 1 (which is hosted on NKN) is hitting s1.amx in Amsterdam (json data here).

Why this happens?

It’s often the lack of peering and/or case of prefered routes. For smaller networks, it’s simply missing peering. For larger networks, it’s about which route they prefer, which not. Here’s a view of networks with their ASNs sorted by latency (wherever RIPE Atlas Probe) was present (measurement link here).

So what can be done about it?

NIXI needs to be more attractive to various (smaller) networks which clearly it is not since it just does not has any content player connected to it due to policy issue. Furthermore customers of Airtel need to buzz it and request for a better route to i root’s local instance.

Comments & thoughts expressed in the post are personal and have nothing to do with my employer. I am also volunteering for supporting tech platform for BharatIX to facilitate peering.

04 Jun

F-root DNS node back up in Chennai!

And finally ACN i.e “Advanced Computer Networks” exam next. Hopefully less to cram in this one and syllabus is pretty interesting.

Talking about networks – I am very happy to post this update. Finally F root server’s node in Chennai is back up!

Though ISC did not updated me about this development but anyways I can always assume they were busy in hitting head with India bureaucratic bodies. 🙂

If you are following my blog, you might have seen my past blog post about “Broken connectivity of F root server” due to NIXI’s routing policies. When I informed ISC (root server operator for F root) about it, they took down the Indian anycasting instance in order to work on fix.

Quoting publically available reply from Mr Leo from ISC on RIPE mailing list in response to my original post about routing glitch:

ISC has experienced a number of routing challenges at our Chennai node over the past year resulting in problems such as this one. We’ve been working with APNIC (the node Sponsor) and NIXI (the node host) to get these resolved. We’ve found the Indian market to be rather unique in the way various large ISP’s chose to announce (or not announce, as the case may be) their routes at various exchanges. In this particular case it appears that the traffic is making it to our Chennai node, but not making it back.

I will work with Anurag directly in the ticket to get this issue resolved. If anyone else is seeing issues in India please mail noc at isc.org to open a ticket. The more reports we have of problems the more attention we can get this issue with the various parties involved.

Here’s updated route:

anurag:~ anurag$ traceroute -a f.root-servers.net
traceroute to f.root-servers.net (192.5.5.241), 64 hops max, 52 byte packets
1 [AS65534] router.home (10.10.0.1) 1.809 ms 0.915 ms 0.981 ms
2 [AS9829] 117.222.224.1 (117.222.224.1) 17.017 ms 18.209 ms 18.170 ms
3 [AS9829] 218.248.169.126 (218.248.169.126) 25.560 ms 29.347 ms 26.233 ms
4 [AS9829] 218.248.250.86 (218.248.250.86) 87.839 ms 86.766 ms 86.485 ms
5 [AS0] 218.100.48.142 (218.100.48.142) 97.747 ms 104.656 ms 98.620 ms
6 [AS55440] f.root-servers.net (192.5.5.241) 97.615 ms 97.153 ms 97.531 ms
anurag:~ anurag$

97 ms latency seems OK. Likely should be slightly less but I assume that’s BSNL problem (return path via Airtel) rather then NIXI or F root’s issue.

I guess Indian instance has been brought up somewhere in mid of April last month (20 days back). I can guess that based on data collected from RIPE Atlas Probe #1032 hosted at my home network along with BGP session details at NIXI’s route server in Chennai.

Screen Shot 2013-06-04 at 8.56.52 PM

It seems like ISC team has done a good job and brought back the node with full connectivity. I can’t blame them for time since it involved Indian bureaucracy.

Going into some of technical details…

Original Problem

The original problem with this node was because NIXI enforces regional route policy. According to that ISPs are forced to announce regional routes and they have option of announcing or not announcing other region routes. E.g BSNL was participating at NIXI Chennai and announcing it’s South Indian prefixes only and not BSNL Haryana (in North of India) prefixes. Now since BSNL was participating at NIXI and had a BGP session with NIXI’s route server on AS24029, it was getting routes to F root while in return it was NOT announcing BSNL Haryana’s prefixes.

There could be multiple solution to this problem

ISP’s like BSNL start announcing all prefixes at all NIXI. (Hard because they won’t let others to “use their backbone” by just peering)
ISP’s like BSNL peer directly with ISC’s router at Chennai and pass their entire table. (Bit hard since number of ISPs & hard to connivence them)
ISC adds transit link defaulting on to transit interface for any non-available routes. It does NOT means announcing F root server prefixes to transit but rather using transit as “default” incase of partial connectivity.

I think ISC went with #3. F root server uses IP address 192.5.5.241 which comes from 192.5.5.0/24 subnet announced by AS3557. In India AS3557 sits under another AS24049 of ISC and router on AS24049 connects to NIXI’s shared peering switch on IP 218.100.48.142.

There was exactly same problem with Netnod’s i root server in Mumbai and they also brought it down after I informed them last year (blog post here). It still seems down. Let’s hope it will be up again very soon.

So that’s all about it. Back to studies for now…

Disclaimer: I work for PCH and part of what we do involves backend DNS hosting. My involvement here is completely out of personal interest about infrastructure in India and has nothing to do with my employeer. All comments are completely personal.

11 Oct

i root server Mumbai node offline

Super dull time here. No classes going on due to “TCS Placement session” at college and this makes me to sit in my room most of time of my day.

Yesterday I tested connectivity to all 13 Global Root DNS Servers and found i root was giving issue.

Here’s a my yesterday’s traceroute to i root:

traceroute to i.root-servers.net. (192.36.148.17), 30 hops max, 60 byte packets
1 router.local (10.0.0.1) 1.470 ms 1.965 ms 2.452 ms
2 117.200.48.1 (117.200.48.1) 26.030 ms 28.857 ms 31.243 ms
3 218.248.173.46 (218.248.173.46) 34.673 ms 37.091 ms 41.025 ms
4 218.248.246.130 (218.248.246.130) 72.853 ms 75.272 ms 77.959 ms
5 * * *
6 * * *

Since i root is another root server hosted within India by NIXI, I was quite sure this was issue again due to NIXI’s regional route enforcement policy along with missing transit link on i root. You can see my last blog post about same issue with F root here.

What was happening here was that Swedish provider Netnod had a anycasting node of i root server at NIXI Mumbai. Netnod uses IP 192.36.148.17 from 192.36.148.0/24 subnet announced by their AS 29216. In current setup Netnod router was connected to NIXI’s Mumbai subnet and was announcing that prefix. Thus all providers including BSNL were getting prefix in their routing table and hence there was a forward path from BSNL to Netnod Mumbai router.
But since ISPs like BSNL are forced to announce regional routes only, BSNL was NOT announcing their prefixes uses in Haryana at Mumbai (they do it at nearest regional exchange which is NIXI Noida) and thus Netnod router was not having any return path. This is true for many other big Indian providers who participate at more then one NIXI.

I informed Netnod Network Operation Center about the issue and they acted promptly by taking Mumbai anycasting instance down. As per their last email to me, they are keeping root server instance down unless they figure out what can be done to prevent this problem.

It is important to note here that if a node is taken down in anycasting that is fine since traffic is routed to other nearest node but keeping a faulty node damages.

Here’s my updated traceroute:

traceroute to 192.36.148.17 (192.36.148.17), 30 hops max, 60 byte packets
1 router.local (10.0.0.1) 1.486 ms 1.965 ms 2.472 ms
2 117.200.48.1 (117.200.48.1) 26.766 ms 30.029 ms 32.558 ms
3 218.248.173.38 (218.248.173.38) 83.640 ms 83.920 ms 84.336 ms
4 115.114.57.165.static-Mumbai.vsnl.net.in (115.114.57.165) 92.011 ms 92.447 ms 92.964 ms
5 ix-0-100.tcore2.MLV-Mumbai.as6453.net (180.87.39.25) 85.625 ms 88.078 ms 90.528 ms
6 180.87.39.58 (180.87.39.58) 227.061 ms 236.796 ms 237.210 ms
7 195.229.3.193 (195.229.3.193) 238.669 ms 196.731 ms 197.479 ms
8 195.229.2.67 (195.229.2.67) 205.832 ms 207.994 ms 210.133 ms
9 195.229.27.22 (195.229.27.22) 204.067 ms 206.465 ms 208.859 ms
10 80.88.240.121 (80.88.240.121) 211.274 ms 213.719 ms 216.668 ms
11 80.88.241.170 (80.88.241.170) 223.069 ms 224.352 ms 225.494 ms
12 i.root-servers.net (192.36.148.17) 227.769 ms 229.160 ms 231.765 ms

With this, India has lost I root server along with F root for time being unless Netnod is able to workout with NIXI on this. Good luck to last one i.e K root in Delhi to handle the load! 🙂

anuragbhatia.com

DNS, BGP, Anycast, IPv6 & more!

Tag Archives: Netnod