Routing with North East India!
A few weeks back I got in touch with Marc from Meghalaya. He offered to host RIPE Atlas probe at Shillong and that’s an excellent location which isn’t there on RIPE Atlas coverage network yet. It took around 5 days for the probe to reach Shillong from Haryana. I think probably this probe is the one at the most beautiful place in India. 🙂
Now that probe is connected, I thought to look into routing which is super exciting for far from places like Shillong. Marc has a BSNL FTTH connection & mentioned about not-so-good latency. Let’s trace to 1st IP of the corresponding /24 pool on which probe is hosted:
traceroute -A 117.247.134.1 traceroute to 117.247.134.1 (117.247.134.1), 30 hops max, 60 byte packets 1 172.16.0.1 (172.16.0.1) [*] 0.552 ms 0.738 ms 0.909 ms 2 192.168.0.1 (192.168.0.1) [*] 3.070 ms 3.330 ms 3.266 ms 3 10.8.64.1 (10.8.64.1) [*] 18.401 ms 18.497 ms 18.097 ms 4 192.168.250.2 (192.168.250.2) [*] 18.655 ms 18.778 ms 24.928 ms 5 125.23.236.157 (125.23.236.157) [AS24560/AS9498] 25.758 ms 25.532 ms 25.251 ms 6 182.79.234.57 (182.79.234.57) [*] 181.730 ms 174.327 ms 174.075 ms 7 182.79.205.173 (182.79.205.173) [*] 50.784 ms 41.060 ms 41.120 ms 8 182.79.178.150 (182.79.178.150) [*] 177.159 ms 176.908 ms 182.79.234.195 (182.79.234.195) [*] 179.786 ms 9 182.79.247.165 (182.79.247.165) [*] 50.855 ms 182.79.177.99 (182.79.177.99) [*] 50.611 ms 182.79.247.115 (182.79.247.115) [*] 50.299 ms 10 182.79.206.46 (182.79.206.46) [*] 175.914 ms 182.79.190.125 (182.79.190.125) [*] 178.664 ms 182.79.222.189 (182.79.222.189) [*] 178.358 ms 11 149.3.183.129 (149.3.183.129) [AS6762] 175.873 ms 175.950 ms 149.3.183.125 (149.3.183.125) [AS6762] 174.901 ms 12 xe11-1-0.marsig2.mar.seabone.net (213.144.176.214) [AS6762] 196.814 ms xe7-3-0.marsig2.mar.seabone.net (213.144.176.172) [AS6762] 169.245 ms xe11-0-0.marsig2.mar.seabone.net (213.144.176.224) [AS6762] 161.673 ms 13 * * * 14 103-16-152-26-noc.bsccl.com (103.16.152.26) [AS132602] 249.369 ms 245.773 ms 249.437 ms 15 163.47.80-138-noc.bsccl.com (163.47.80.138) [AS132602] 213.663 ms 214.164 ms 213.697 ms 16 218.248.255.1 (218.248.255.1) [AS9829] 213.832 ms 210.132 ms 209.828 ms 17 * * * 18 218.248.170.229 (218.248.170.229) [AS9829] 213.674 ms 217.609 ms 217.687 ms 19 117.247.134.1 (117.247.134.1) [AS9829] 223.293 ms 223.366 ms 222.937 ms
This is interesting output. So there are two parts of it:
- Traffic going via Bangladesh
- Traffic to Bangladesh going via Europe!
While #1 may look like a routing issue, it’s actually desired result of a deal between BSNL & BSCCL. I blogged about it in last year when it was visible in BGP tables. Eventually, this link was launched by Indian Prime Minister Modi.
From the map, it seems like an ideal choice but I really wish BSNL went for some kind of circuits instead of transit with BSCCL. Reason being poor routing across Asian backbones which we see in reason #2.
Coming to #2 – this clearly is bad and broken. Traffic is hitting from Siti broadband > Airtel > Telecom Italia > BSCCL and this is resulting in traffic going from India to Europe first before returning to South Asia.
Let’s trace to same 1st IP of the pool from all Indian RIPE Atlas probes for a detailed picture:
Measurement result: https://atlas.ripe.net/measurements/8844267/
As we can see latency numbers are quite decent from BSNL’s AS9829 itself. 60-70ms seems fine considering it’s from the probes which are in North or South India to far away in North East. Let’s look at some of these traces from probes on BSNL itself:
This shows that there is indeed a direct backbone circuit of BSNL to that location. There’s a low chance of it being on top of BSCCL infra.
Except for BSNL, rest all other Indian networks are routing towards that BSNL segment in Meghalaya from Europe or Singapore/Hong Kong. All the ones from Europe are from Marseille in France. That’s the landing station for 11 cable systems:
- SEACOM
- SEA-ME-WE-4
- EIG
- I-ME-WE
- Ariane 2
- Atlas Offshore
- Med Cable
- TE North
- Tamares Telecom
- Alexandros
- AAE-1 (Asia Africa Europe)
Out of these Se-Me-We-4 lands in Bangladesh and I guess that is being used by BSCCL for traffic. So coming back to why routing is so terrible from Indian networks towards BSNL in North East? To understand that we need to look at uplinks of BSCCL.
Well, BSNL is announcing 117.247.134.0/24 to BSCCL AS132602 only. BSCCL is buying transit from Telecom Italia AS6762 and NTT AS2914.
http://bgp.he.net/AS132602#_graph4
Looking at one of few traces from Europe:
213.144.176.194/31 TIS – BSCCL connectivity
213.144.176.194 – 10Gig port on TIS AS6762 router in Marseille
213.144.176.195 – TIS’s IP on BSCCL router in somewhere in Bangladesh
Next, looking at NTT AS2914 transit of BSCCL:
Here as traffic handoff from Tata AS6453 is happening to NTT AS2914 in Singapore (logical and correct!) and NTT to BSCCL also within Singapore. The latency is high due to bad return. Here forward is slightly bad but not as bad as return possibly.
Let’s look at return trace to 2nd hop 115.118.168.1 from RIPE probe at destination (measurement here):
So clearly return path i.e Shillong to Hyderabad is via Europe because BSCCL used TIS for forwarding path.
So keeping above traces in mind, here’s the reason for high latency:
- BSNL is routing traffic over its backbone but rest all traffic i.e which is not going towards BSNL is being routed from Bangladeshi provider BSCCL.
- BSCCL is announcing routes to NTT AS2914 in Singapore & TIS AS6762 in France. Thus to send any traffic to BSNL’s segment in Meghalaya, one has to send it either via TIS router in Marseille, France or NTT Singapore. This adds up latency significantly for Indian networks (excluding) towards BSNL Meghalaya.
- BSCCL is using TIS AS6762 to reach Tata AS6453 and this is resulting in very bad return route and thus Meghalaya to any other network in India who is Tata AS6453 downstream is via Marseille, France.
Quite a lot seems messed up. BSNL’s should at least start announcing 117.247.134.0/24 immediately across NIXI’s subject to capacity between their core network in North East. If there’s a capacity constrained, they should use L1 circuits from BSCCL to connect network in Shillong instead of IP transit.
How is BSNL in North East reaching Google?
Seems direct to BSNL’s PNI with Google within India.