High latency between India and Europe
I am back from INNOG 8 and still trying to settle down with an overflow of pending tasks.
Since midnight of 31st May 2025, there seems to be a visible jump in latency between India and Europe over Tata Comm’s AS6453 network. I can see this as one of the providers at home is routing forward & return paths via Tata Tele AS45820 which uses Tata Comm’s AS4755 transit (only). If I look up the EU and other points towards India, results are mixed as Airtel & Jio do not seem to be impacted. In Dec 2024 Airtel was routing Europe -> India via the US for couple of months as covered here.
Here’s how it looks:
This is doing an average of latency a few thousand endpoints in the EU and hence jump by only a few will not show a change. Let’s pick some specific ones. Picking my server in Germany on Contabo - server7.anuragbhatia.com as a test point:
I see this jump only in IPv4 traffic and not much in IPv6 traffic. Cannot say much for IPv6 when testing from home ISP is multi-homed and announcing the IPv6 route on Airtel. This also gives a clue that forward (IPv4/IPv6) is the same but return via Tata Comm (IPv4) is changed for sure.
Let’s check forward and return paths over IPv4:
India -> Germany
HOST: desktop.rtk Loss% Snt Last Avg Best Wrst StDev
1. AS??? _gateway (172.16.0.1) 0.0% 10 0.2 0.2 0.1 0.2 0.0
2. AS??? 10.10.11.61 0.0% 10 3.2 3.2 3.2 3.3 0.0
3. AS135817 estointernet.in (43.255.166.81) 50.0% 10 4.1 3.9 3.8 4.1 0.2
4. AS??? 10.11.11.142 0.0% 10 5.1 4.1 3.8 5.1 0.4
5. AS45820 static-217.245.194.14-tataidc.co.in (14.194.245.217) 0.0% 10 4.8 4.6 4.5 4.8 0.1
6. AS4755 14.140.113.197.static-Delhi-vsnl.net.in (14.140.113.197) 0.0% 10 4.7 5.0 4.5 6.8 0.7
7. AS??? 172.23.183.134 0.0% 10 27.5 27.4 27.3 27.5 0.1
8. AS6453 ix-ae-0-100.tcore1.mlv-mumbai.as6453.net (180.87.38.5) 0.0% 10 27.6 28.9 27.3 38.4 3.4
9. AS6453 if-be-13-2.ecore1.mlv-mumbai.as6453.net (180.87.38.29) 60.0% 10 231.1 230.9 230.5 231.3 0.4
10. AS6453 if-be-47-2.ecore1.emrs2-marseille.as6453.net (80.231.217.52) 40.0% 10 226.6 226.5 226.5 226.6 0.0
11. AS6453 if-bundle-33-2.qcore1.emrs2-marseille.as6453.net (80.231.165.48) 70.0% 10 226.7 226.6 226.5 226.7 0.1
12. AS6453 if-bundle-15-2.qcore1.pye-paris.as6453.net (80.231.154.32) 0.0% 10 228.6 228.1 227.6 228.6 0.4
13. AS6453 if-bundle-22-2.qcore1.pvu-paris.as6453.net (80.231.154.199) 10.0% 10 231.6 231.2 230.7 231.6 0.3
14. AS6453 if-bundle-12-2.qcore2.pvu-paris.as6453.net (80.231.245.13) 10.0% 10 230.3 231.1 230.1 234.0 1.2
15. AS6453 80.231.245.15 50.0% 10 226.7 226.9 226.5 227.4 0.4
16. AS1299 prs-bb1-link.ip.twelve99.net (62.115.118.58) 50.0% 10 227.1 227.2 227.1 227.4 0.1
17. AS1299 ffm-bb1-link.ip.twelve99.net (62.115.123.12) 0.0% 10 235.9 235.5 235.4 235.9 0.2
18. AS1299 ddf-b3-link.ip.twelve99.net (62.115.135.151) 0.0% 10 236.8 236.9 236.7 237.2 0.1
19. AS1299 anonymous-ic-386520.ip.twelve99-cust.net (213.248.70.155) 0.0% 10 238.1 244.2 238.1 255.7 5.8
20. AS51167 server7.anuragbhatia.com (213.199.54.67) 0.0% 10 241.7 241.7 241.7 241.8 0.0
There’s a visible latency jump from 27 to 231ms as packets reach Europe. From the DNS PTRs, it seems like a direct forward path and an expected path based on submarine cables. Let’s see if there’s something unusual in the return path. I am going to trace to the .255 IP of the ISP to get an identical trace.
HOST: server7.anuragbhatia.com Loss% Snt Last Avg Best Wrst StDev
1. AS51167 ip-161-97-128-12.static.contabo.net (161.97.128.12) 0.0% 10 11.0 11.7 0.5 38.5 14.5
2. AS??? 10.0.50.1 0.0% 10 0.3 6.0 0.3 24.9 7.7
3. AS3356 et-4-0-8.edge6.Dusseldorf1.Level3.net (62.67.22.193) 0.0% 10 9.6 8.9 7.0 14.4 2.6
4. AS3356 ae2.3218.ear6.frf1.neo.colt.net (171.75.8.136) 70.0% 10 7.9 7.4 7.1 7.9 0.4
5. AS6453 if-ae-68.tcore2.fr0-frankfurt.as6453.net (80.231.65.22) 60.0% 10 4.2 4.1 3.9 4.4 0.2
6. AS6453 if-bundle-56-2.qcore2.pvu-paris.as6453.net (80.231.245.40) 0.0% 10 14.8 14.2 13.8 14.8 0.3
7. AS6453 if-bundle-12-2.qcore1.pvu-paris.as6453.net (80.231.245.12) 70.0% 10 16.4 14.9 13.7 16.4 1.4
8. AS6453 if-bundle-22-2.qcore1.pye-paris.as6453.net (80.231.154.198) 70.0% 10 14.5 14.4 14.0 14.6 0.3
9. AS6453 if-bundle-2-2.qcore2.pye-paris.as6453.net (80.231.154.27) 10.0% 10 14.1 14.2 13.9 14.6 0.3
10. AS6453 if-bundle-13-2.qcore1.ldn-london.as6453.net (80.231.196.37) 70.0% 10 14.6 14.5 14.2 14.6 0.3
11. AS6453 195.219.213.129 10.0% 10 14.2 14.2 13.9 14.5 0.2
12. AS6453 80.231.131.66 0.0% 10 197.0 197.0 196.9 197.2 0.1
13. AS??? ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
14. AS4755 14.140.113.198.static-Delhi-vsnl.net.in (14.140.113.198) 50.0% 10 216.9 216.9 216.8 217.0 0.1
15. AS135817 estointernet.in (103.97.46.255) 0.0% 10 235.1 235.2 234.9 236.1 0.4
Packets seem to be returning via London instead of Marseille, France.
Latency is higher than usual even from London. The jump in latency from London (14.2ms) to Mumbai (197ms) seems high. The expected latency should be around 130ms. This hints at some weird routing. Let’s do the same trace from AS6453 London instead of my server;
Router: gin-ldn-tcore1
Site: GB, London, LDN
Command: traceroute inet4 103.97.46.255 as-number-lookup
traceroute to 103.97.46.255 (103.97.46.255), 30 hops max, 52 byte packets
1 if-ae-26-2.tcore2.ldn-london.as6453.net (80.231.62.57) 1.499 ms 1.163 ms 1.139 ms
MPLS Label=149668 CoS=0 TTL=1 S=1
2 if-bundle-67-2.qcore2.ldn-london.as6453.net (195.219.16.128) 1.223 ms 1.422 ms 1.638 ms
MPLS Label=0 CoS=0 TTL=1 S=1
3 80.231.131.66 (80.231.131.66) 203.189 ms 222.188 ms 202.512 ms
4 172.31.169.86 (172.31.169.86) [AS 65200] 240.769 ms 172.17.125.210 (172.17.125.210) [AS 65200] 243.944 ms 244.206 ms
5 14.140.113.198.static-Delhi-vsnl.net.in (14.140.113.198) [AS 4755] 227.464 ms 223.246 ms 224.161 ms
6 estointernet.in (103.97.46.255) [AS 135817] 227.331 ms 227.854 ms 232.583 ms
This doesn’t give much because hop 2 is in London while hop 3 is in Mumbai. So routing is fine but the latency jump is unusually high and something is wrong with the underlying transport. London - Mumbai should be around 132ms and thus above trace shows an excess of 70ms latency.
Can the transport be possibly take it via Singapore? If that is true then one leg (London - Mumbai would be via Singapore).
Rough half-way latency check:
(Mumbai -> London) 56ms + (London -> Singapore) 77ms + (Singapore -> Mumbai) 29ms = 162ms. That’s still won’t be touching 200ms.
It’s unclear what is causing its routing but the latency jump is very much visible and highly likely not due to congestion as congestion would result in highly variable latency. I still see many AS4755-originated IPs routing directly from Marseille to Mumbai with lower latency.
Let’s see when this gets fixed.
Update - 04 Jun 2025 - issue seems fixed
Since 23:52 on 03 Jun 2025 issue seems to be resolved.
Latest mtr from my server in Europe to Esto (TTSL downstream in Delhi):
1. AS51167 ip-161-97-128-12.static.contabo.net (161.97.128.12) 0.0% 10 10.2 5.7 0.4 15.7 5.4
2. AS??? 10.0.50.1 0.0% 10 0.7 1.2 0.4 3.0 0.9
3. AS3356 et-4-0-8.edge6.Dusseldorf1.Level3.net (62.67.22.193) 0.0% 10 13.1 13.9 12.4 20.3 2.3
4. AS3356 ae2.3218.ear6.frf1.neo.colt.net (171.75.8.136) 60.0% 10 12.4 12.5 12.4 12.7 0.1
5. AS6453 if-ae-68.tcore2.fr0-frankfurt.as6453.net (80.231.65.22) 80.0% 10 4.7 4.5 4.2 4.7 0.3
6. AS6453 if-bundle-56-2.qcore2.pvu-paris.as6453.net (80.231.245.40) 0.0% 10 14.4 14.1 13.7 14.7 0.3
7. AS6453 if-bundle-12-2.qcore1.pvu-paris.as6453.net (80.231.245.12) 30.0% 10 14.9 14.4 13.8 15.0 0.4
8. AS6453 if-bundle-22-2.qcore1.pye-paris.as6453.net (80.231.154.198) 30.0% 10 13.9 14.1 13.7 14.5 0.3
9. AS6453 if-bundle-2-2.qcore2.pye-paris.as6453.net (80.231.154.27) 0.0% 10 14.5 14.5 14.1 14.9 0.2
10. AS??? ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
11. AS6453 195.219.213.129 80.0% 10 13.9 14.3 13.9 14.7 0.6
12. AS6453 80.231.131.66 0.0% 10 144.2 144.3 144.1 144.5 0.1
13. AS??? ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
14. AS4755 14.140.113.198.static-Delhi-vsnl.net.in (14.140.113.198) 80.0% 10 157.0 157.0 157.0 157.0 0.0
15. AS135817 estointernet.in (103.97.46.255) 0.0% 10 175.0 175.0 174.9 175.2 0.1
Based on data from multiple point, here’s rough timeline for the issue:
Issue start: 31 May 2025 - 01:03 (IST)
Issue resolved: 03 June 2025 - 23:53 (IST)
Impact time: 94 hours 50 minutes