06 Jul

ISC F root server – IPv6 issue at NIXI Chennai

Last week I noticed that F root was showing poor connectivity with Indian RIPE Atlas probes for F-root. The graph looked really terrible.
 

 
 
I traced to it from one of RIPE Atlas probes and saw this trace:

Probe #6107
  1 2401:7500:fff0:1::1                      0.838 ms     0.747 ms     0.632 ms
  2 2400:5200:1c00:d::1                      1.755 ms     1.745 ms     1.726 ms
  3 2403:0:100::2be                          2.089 ms     2.054 ms     2.049 ms
  4 2404:a800:2a00::13d                     45.589 ms    26.274 ms     33.64 ms
  5 2404:a800::178                          26.376 ms    25.406 ms    25.276 ms
  6 2001:de8:1:2::3                         25.363 ms    25.232 ms    25.223 ms
  7 *                                               *            *            *
  8 *                                               *            *            *
  9 *                                               *            *            *
 10 *                                               *            *            *
 11 *                                               *            *            *

 
Here the last hop before timeout i.e hop 6 is of NIXI Chennai peering subnet 2001:de8:1:2::/64. As soon as I saw it, it reminded me older issue which happened and broke IPv4 connectivity to root DNS servers. I blogged about it here, here and here. So the problem remains that NIXI is broken cost wise due to charge on in – out policy. This leads to people accepting routes at all NIXI’s but they do not announce their routes. Thus return path is broken and essentially traffic is being blackholed. Earlier this issue was fixed by adding IP transit support to these root DNS servers so that a default route stays in case of all other failures.
It seems like same is missing in IPv4 world and routes are not being announced.
During this time, I saw two BGP sessions at NIXI Chennai for F root:
2001:de8:1:2::3 24049 ESTAB 25d 3h10m 1 0 2263 0
2001:de8:1:2::4 24049 ESTAB 125d18h30m 1 0 2264 0
 
Both were announcing prefix covering F root server’s pool:

show ipv6 bgp neighbors 2001:de8:1:2::3 received-routes
       There are 1 received routes from neighbor 2001:de8:1:2::3
Searching for matching routes, use ^C to quit...
Status A:AGGREGATE B:BEST b:NOT-INSTALLED-BEST C:CONFED_EBGP D:DAMPED
       E:EBGP H:HISTORY I:IBGP L:LOCAL M:MULTIPATH m:NOT-INSTALLED-MULTIPATH
       S:SUPPRESSED F:FILTERED s:STALE
       Prefix             Next Hop        MED        LocPrf     Weight Status
1      2001:500:2f::/48   2001:de8:1:2::3 10         100        0      BE
         AS_PATH: 24049 3557
show ipv6 bgp neighbors 2001:de8:1:2::4 received-routes
       There are 1 received routes from neighbor 2001:de8:1:2::4
Searching for matching routes, use ^C to quit...
Status A:AGGREGATE B:BEST b:NOT-INSTALLED-BEST C:CONFED_EBGP D:DAMPED
       E:EBGP H:HISTORY I:IBGP L:LOCAL M:MULTIPATH m:NOT-INSTALLED-MULTIPATH
       S:SUPPRESSED F:FILTERED s:STALE
       Prefix             Next Hop        MED        LocPrf     Weight Status
1      2001:500:2f::/48   2001:de8:1:2::4 10         100        0      E
         AS_PATH: 24049 3557

 
I posted about it on SANOG and APNIC mailing list. Though there hasn’t been any reply by ISC, Sunny from APNIC passed info to them and I noticed that prefix announcement from NIXI has been withdrawn. Connectivity to F root now works to the instances outside India.
 

 
 
Waiting to hear from ISC as of now. Time to get back to work!
 

15 Jun

Routing with North East India!

A few weeks back I got in touch with Marc from Meghalaya. He offered to host RIPE Atlas probe at Shillong and that’s an excellent location which isn’t there on RIPE Atlas coverage network yet. It took around 5 days for the probe to reach Shillong from Haryana. I think probably this probe is the one at the most beautiful place in India. 🙂
Now that probe is connected, I thought to look into routing which is super exciting for far from places like Shillong. Marc has a BSNL FTTH connection & mentioned about not-so-good latency. Let’s trace to 1st IP of the corresponding /24 pool on which probe is hosted:

traceroute -A 117.247.134.1
traceroute to 117.247.134.1 (117.247.134.1), 30 hops max, 60 byte packets
1 172.16.0.1 (172.16.0.1) [*] 0.552 ms 0.738 ms 0.909 ms
2 192.168.0.1 (192.168.0.1) [*] 3.070 ms 3.330 ms 3.266 ms
3 10.8.64.1 (10.8.64.1) [*] 18.401 ms 18.497 ms 18.097 ms
4 192.168.250.2 (192.168.250.2) [*] 18.655 ms 18.778 ms 24.928 ms
5 125.23.236.157 (125.23.236.157) [AS24560/AS9498] 25.758 ms 25.532 ms 25.251 ms
6 182.79.234.57 (182.79.234.57) [*] 181.730 ms 174.327 ms 174.075 ms
7 182.79.205.173 (182.79.205.173) [*] 50.784 ms 41.060 ms 41.120 ms
8 182.79.178.150 (182.79.178.150) [*] 177.159 ms 176.908 ms 182.79.234.195 (182.79.234.195) [*] 179.786 ms
9 182.79.247.165 (182.79.247.165) [*] 50.855 ms 182.79.177.99 (182.79.177.99) [*] 50.611 ms 182.79.247.115 (182.79.247.115) [*] 50.299 ms
10 182.79.206.46 (182.79.206.46) [*] 175.914 ms 182.79.190.125 (182.79.190.125) [*] 178.664 ms 182.79.222.189 (182.79.222.189) [*] 178.358 ms
11 149.3.183.129 (149.3.183.129) [AS6762] 175.873 ms 175.950 ms 149.3.183.125 (149.3.183.125) [AS6762] 174.901 ms
12 xe11-1-0.marsig2.mar.seabone.net (213.144.176.214) [AS6762] 196.814 ms xe7-3-0.marsig2.mar.seabone.net (213.144.176.172) [AS6762] 169.245 ms xe11-0-0.marsig2.mar.seabone.net (213.144.176.224) [AS6762] 161.673 ms
13 * * *
14 103-16-152-26-noc.bsccl.com (103.16.152.26) [AS132602] 249.369 ms 245.773 ms 249.437 ms
15 163.47.80-138-noc.bsccl.com (163.47.80.138) [AS132602] 213.663 ms 214.164 ms 213.697 ms
16 218.248.255.1 (218.248.255.1) [AS9829] 213.832 ms 210.132 ms 209.828 ms
17 * * *
18 218.248.170.229 (218.248.170.229) [AS9829] 213.674 ms 217.609 ms 217.687 ms
19 117.247.134.1 (117.247.134.1) [AS9829] 223.293 ms 223.366 ms 222.937 ms

 
This is interesting output. So there are two parts of it:

  1. Traffic going via Bangladesh
  2. Traffic to Bangladesh going via Europe!

 
While #1 may look like a routing issue, it’s actually desired result of a deal between BSNL & BSCCL. I blogged about it in last year when it was visible in BGP tables. Eventually, this link was launched by Indian Prime Minister Modi.

 
From the map, it seems like an ideal choice but I really wish BSNL went for some kind of circuits instead of transit with BSCCL. Reason being poor routing across Asian backbones which we see in reason #2.
 
Coming to #2 – this clearly is bad and broken. Traffic is hitting from Siti broadband > Airtel > Telecom Italia > BSCCL and this is resulting in traffic going from India to Europe first before returning to South Asia.
 
Let’s trace to same 1st IP of the pool from all Indian RIPE Atlas probes for a detailed picture:
Measurement result: https://atlas.ripe.net/measurements/8844267/

 
As we can see latency numbers are quite decent from BSNL’s AS9829 itself. 60-70ms seems fine considering it’s from the probes which are in North or South India to far away in North East. Let’s look at some of these traces from probes on BSNL itself:
 



 
This shows that there is indeed a direct backbone circuit of BSNL to that location. There’s a low chance of it being on top of BSCCL infra.

 
Except for BSNL, rest all other Indian networks are routing towards that BSNL segment in Meghalaya from Europe or Singapore/Hong Kong. All the ones from Europe are from Marseille in France. That’s the landing station for 11 cable systems:

  • SEACOM
  • SEA-ME-WE-4
  • EIG
  • I-ME-WE
  • Ariane 2
  • Atlas Offshore
  • Med Cable
  • TE North
  • Tamares Telecom
  • Alexandros
  • AAE-1 (Asia Africa Europe)

 
Out of these Se-Me-We-4 lands in Bangladesh and I guess that is being used by BSCCL for traffic. So coming back to why routing is so terrible from Indian networks towards BSNL in North East? To understand that we need to look at uplinks of BSCCL.
Well, BSNL is announcing 117.247.134.0/24 to BSCCL AS132602 only. BSCCL is buying transit from Telecom Italia AS6762 and NTT AS2914.
http://bgp.he.net/AS132602#_graph4

 
Looking at one of few traces from Europe:

 
213.144.176.194/31 TIS – BSCCL connectivity
213.144.176.194 – 10Gig port on TIS AS6762 router in Marseille
213.144.176.195 – TIS’s IP on BSCCL router in somewhere in Bangladesh
 
Next, looking at NTT AS2914 transit of BSCCL:

 
Here as traffic handoff from Tata AS6453 is happening to NTT AS2914 in Singapore (logical and correct!) and NTT to BSCCL also within Singapore.  The latency is high due to bad return. Here forward is slightly bad but not as bad as return possibly.
Let’s look at return trace to 2nd hop 115.118.168.1 from RIPE probe at destination (measurement here):

So clearly return path i.e Shillong to Hyderabad is via Europe because BSCCL used TIS for forwarding path.
 
So keeping above traces in mind, here’s the reason for high latency:

  1. BSNL is routing traffic over its backbone but rest all traffic i.e which is not going towards BSNL is being routed from Bangladeshi provider BSCCL.
  2. BSCCL is announcing routes to NTT AS2914 in Singapore & TIS AS6762 in France. Thus to send any traffic to BSNL’s segment in Meghalaya, one has to send it either via TIS router in Marseille, France or NTT Singapore. This adds up latency significantly for Indian networks (excluding) towards BSNL Meghalaya.
  3. BSCCL is using TIS AS6762 to reach Tata AS6453 and this is resulting in very bad return route and thus Meghalaya to any other network in India who is Tata AS6453 downstream is via Marseille, France.

Quite a lot seems messed up. BSNL’s should at least start announcing 117.247.134.0/24 immediately across NIXI’s subject to capacity between their core network in North East. If there’s a capacity constrained, they should use L1 circuits from BSCCL to connect network in Shillong instead of IP transit.
 
How is BSNL in North East reaching Google?

Seems direct to BSNL’s PNI with Google within India.