24 Sep

Tata Communications (AS4755) pushing traffic to Reliance Jio (AS55836) via Singapore!

Bad route
So it seems like apart from voice interconnect issues, Jio is also facing routing issues on the backbone. I ran a trace to one of IP’s on Jio network allocated to end customer –
I ran trace from all Indian RIPE Atlas probes – https://atlas.ripe.net/measurements/5738489/#!probes
There seem quite a few RIPE Atlas probes which are giving latency on 150ms + range. Seems like they are downstream or downstream of downstream of Tata Comm’s AS4755 and routing is happening via Singapore!
Two of such traces
jio1 screen-shot-2016-09-24-at-11-26-05-pm
This isn’t how regular routing should work since even if both do not announce routes to each other, Jio’s routes are visible at NIXI. E.g NIXI Noida shows visible which covers that test IP. Tata AS4755 is probably rejecting that IP.
And if we look at Tata AS6453 routing table for their Chennai PoP, it shows AS_PATH as AS3431 (PCCW Global) who happens to be upstream of Jio’s International network AS64049.
Clearly, that’s just bad routing!
It might be the case Tata is doing intentionally without any evil plans because NIXI might just not have sufficient capacity. Overall ISPs should peer with PNI’s as well as we need IX’es other than just NIXI to promote more peering and promote better interconnect.
Disclaimer: This post and expressed comments are in my personal capacity and my employer has no relation with it. 

01 Jun

BSNL > Softlayer connectivity problem & possible fix

It’s late night here in India. I am having final 8th semester exams and as usual really bored! 

Though this time we have interesting subjects but still syllabus is pretty boring spreading across multiple books, notes and pdf’s. Anyways I will be out of college after June which sounds good.


Tonight, I found a routing glitch. Yes a routing glitch!! 🙂

These issues somehow keep my life in orbit and give a good understanding on how routing works over the Internet.



OK – so the issue

I noticed a really bad (forward) route from my BSNL’s connection to hostgator.in website hosted in Softlayer Singapore. Let’s look at forward path:

anurag:~ anurag$ traceroute -a hostgator.in
traceroute to hostgator.in (, 64 hops max, 52 byte packets
1 [AS65534] router.home ( 1.189 ms 0.910 ms 0.810 ms
2 [AS9829] ( 17.707 ms 21.147 ms 16.925 ms
3 [AS9829] ( 30.195 ms 29.766 ms 29.976 ms
4 [AS9829] ( 75.432 ms 77.488 ms 76.761 ms
5 [AS6453] if-11-1-1.mcore3.laa-losangeles.as6453.net ( 368.104 ms 303.206 ms 309.964 ms
6 [AS6453] if-10-2-0-14.tcore2.lvw-losangeles.as6453.net ( 309.070 ms 308.725 ms 310.073 ms
7 [AS6453] ( 317.050 ms 318.714 ms 398.408 ms
8 [AS2914] ae-5.r21.lsanca03.us.bb.gin.ntt.net ( 305.672 ms * 304.480 ms
9 [AS2914] as-2.r20.osakjp01.jp.bb.gin.ntt.net ( 414.205 ms
[AS2914] as-1.r21.tokyjp01.jp.bb.gin.ntt.net ( 485.451 ms
[AS2914] as-2.r20.osakjp01.jp.bb.gin.ntt.net ( 414.272 ms
10 [AS2914] ae-3.r24.tokyjp05.jp.bb.gin.ntt.net ( 381.221 ms
[AS2914] ae-1.r23.osakjp01.jp.bb.gin.ntt.net ( 420.412 ms
[AS2914] ae-3.r25.tokyjp05.jp.bb.gin.ntt.net ( 372.768 ms
11 [AS2914] ae-7.r25.tokyjp05.jp.bb.gin.ntt.net ( 394.899 ms
[AS2914] ae-7.r24.tokyjp05.jp.bb.gin.ntt.net ( 406.922 ms
[AS2914] ae-2.r00.tokyjp03.jp.bb.gin.ntt.net ( 491.190 ms
12 [AS2914] ae-3.r00.tokyjp03.jp.bb.gin.ntt.net ( 399.065 ms
[AS2914] xe-0-0-0.bbr01.eq01.tok01.networklayer.com ( 307.955 ms
[AS2914] ae-2.r00.tokyjp03.jp.bb.gin.ntt.net ( 392.937 ms
13 [AS2914] xe-0-0-0.bbr01.eq01.tok01.networklayer.com ( 310.298 ms
[AS36351] ae1.bbr01.eq01.sng02.networklayer.com ( 306.396 ms
[AS2914] xe-0-0-0.bbr01.eq01.tok01.networklayer.com ( 407.191 ms
14 [AS36351] ae5.dar01.sr03.sng01.networklayer.com ( 388.660 ms
[AS36351] ae5.dar02.sr03.sng01.networklayer.com ( 303.546 ms 409.645 ms
15 [AS36351] po2.fcr01.sr03.sng01.networklayer.com ( 407.589 ms
[AS36351] ae5.dar02.sr03.sng01.networklayer.com ( 310.587 ms
[AS36351] po2.fcr01.sr03.sng01.networklayer.com ( 305.969 ms
16 [AS36351] po2.fcr01.sr03.sng01.networklayer.com ( 363.405 ms * 309.151 ms
17 * * *
18 * * *


BSNL (India) >> IPLC circuit >> Tata AS6453 Los Angeles, California >> NTT (US) >> NTT (Asia) >> NTT (Tokyo) >> Softlayer (Tokyo) >> Softlayer (Singapore)


Pretty bad. Ideally route should be BSNL > Upstream – Tata/Reliance/Airtel/Vodafone > Singapore (that’s it. Over!)


Interesting enough that Softlayer operates a nice looking glass and hence I was able to trace return path to my home router from there to get idea of complete route.

bbr02.eq01.sng02> traceroute
HOST: bbr02.eq01.sng02-re0 Loss% Snt Last Avg Best Wrst StDev
1. 0.0% 5 0.4 0.5 0.4 0.5 0.0
2. 0.0% 5 0.6 0.6 0.5 0.7 0.0 <<< PCCW Global
3. 0.0% 5 11.1 7.4 0.6 12.7 5.1 <<< Tata AS6453
4. 0.0% 5 0.6 2.4 0.6 9.6 4.0
5. 0.0% 5 62.1 61.2 60.7 62.1 0.6
6. 0.0% 5 97.7 73.3 60.8 97.7 17.4
7. 0.0% 5 103.2 75.0 59.6 103.2 18.1
8. 0.0% 5 61.1 74.0 61.1 88.9 12.4 <<< Tata AS6453
9. 0.0% 5 91.7 92.6 91.7 96.3 2.0 <<<< VSNL AS4755
10. 0.0% 5 95.5 96.5 95.5 99.7 1.8 <<<< Hits BSNL AS9829
11. 0.0% 5 106.6 110.4 106.4 126.2 8.8
12. 0.0% 5 106.3 107.0 106.3 108.6 1.0
13. ???



Overall pretty good and direct. Basically latency value is also as we expect till hop 12 because forward route (i.e from BSNL > Softlayer) is direct from BSNL router on hop 12 but for routers below it they are taking route via US. Return path trace is not showing those routers because BSNL is dropping ICMP.


Reason for problem:

Forward path is terribly bad here because BSNL let usual BGP route selection algorithm to deal with it. Basically BSNL is getting multiple routes for that prefix from Softlayer. One from it’s IP port in India with Tata-VSNL AS4755 and other from it’s port from Tata in Los Angles (Tata AS6453) over IPLC.


So possible routes as per AS paths are:

AS9829 > AS4755 > AS6453 > AS2914 > AS36351 

AS9829 > AS6453 > AS2914 > AS36351


Based on default property of BGP, it is picking short AS path i.e 2nd one. In case of #1 BGP session between BSNL AS9829 and Tata-VSNL AS4755 is within India. 

For example:

1 [AS65534] router.home ( 1.709 ms 0.912 ms 0.982 ms
2 [AS9829] ( 17.451 ms 18.075 ms 19.029 ms
3 [AS9829] ( 21.843 ms 24.584 ms 22.491 ms
4 [AS4755] ( 57.399 ms 58.563 ms 57.446 ms


Very likely BGP session here is configured on usual /30 subnet with one IP on BSNL side, one on Tata’s side, third one as broadcast and 4th lying useless due to Math game!

So is part of that /30. Let’s ping it:

anurag:~ anurag$ ping -c 5
PING ( 56 data bytes
64 bytes from icmp_seq=0 ttl=58 time=63.286 ms
64 bytes from icmp_seq=1 ttl=58 time=66.029 ms
64 bytes from icmp_seq=2 ttl=58 time=59.063 ms
64 bytes from icmp_seq=3 ttl=58 time=59.439 ms
64 bytes from icmp_seq=4 ttl=58 time=61.719 ms

— ping statistics —
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 59.063/61.907/66.029/2.573 ms
anurag:~ anurag$


60ms latency – for sure Mumbai and all good here.


Now let’s look at IP just next to it:


anurag:~ anurag$ ping -c 5
PING ( 56 data bytes
64 bytes from icmp_seq=0 ttl=251 time=28.784 ms
64 bytes from icmp_seq=1 ttl=251 time=25.586 ms
64 bytes from icmp_seq=2 ttl=251 time=28.631 ms
64 bytes from icmp_seq=3 ttl=251 time=26.905 ms
64 bytes from icmp_seq=4 ttl=251 time=26.213 ms

— ping statistics —
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 25.586/27.224/28.784/1.282 ms
anurag:~ anurag$


Half latency and that’s BSNL router in Delhi/Noida where they are taking drop from Tata. It’s BSNL’s router but sitting on Tata’s IP for BGP session. So this clearly tells that when we see routes from AS9829 to AS4755 Tata-VSNL they are between routers within India.


Now coming back to bad route between BSNL and Softlayer, in that case first few hops are:

1 [AS65534] router.home ( 1.189 ms 0.910 ms 0.810 ms
2 [AS9829] ( 17.707 ms 21.147 ms 16.925 ms
3 [AS9829] ( 30.195 ms 29.766 ms 29.976 ms
4 [AS9829] ( 75.432 ms 77.488 ms 76.761 ms
5 [AS6453] if-11-1-1.mcore3.laa-losangeles.as6453.net ( 368.104 ms 303.206 ms 309.964 ms


Hop 5 has latency of 300ms (usual for India > US routes). Again assuming is coming from /30 and as per usual BSNL practice next IP in that subnet i.e would be on BSNL’s side, let’s ping

anurag:~ anurag$ ping -c 5
PING ( 56 data bytes
64 bytes from icmp_seq=0 ttl=250 time=373.483 ms
64 bytes from icmp_seq=1 ttl=250 time=395.493 ms
64 bytes from icmp_seq=2 ttl=250 time=419.340 ms
64 bytes from icmp_seq=3 ttl=250 time=305.460 ms
64 bytes from icmp_seq=4 ttl=250 time=362.598 ms

— ping statistics —
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 305.460/371.275/419.340/38.232 ms
anurag:~ anurag$



Hmm….300ms latency. Unexpected. I thought this router was in India but this seems slightly complex. Likely BGP session here is using BSNL’s /30 subnet and not via Tata Comm’s subnet. 

OK – let’s see last IP from BSNL on that trace – it was Let’s ask Tata AS6453 Los Angles LAA router via AS6453 Looking Glass for BGP table:


Router: gin-laa-mcore3
Site: US, Los angeles, LAA
Command: show ip bgp

BGP routing table entry for
Bestpath Modifiers: deterministic-med
Paths: (2 available, best #1)
14 16 17 18
ix-3-2.mcore3.LAA-LosAngeles. from ix-3-2.mcore3.LAA-LosAngeles. (
Origin IGP, valid, external, best
9829, (received-only)
ix-3-2.mcore3.LAA-LosAngeles. from ix-3-2.mcore3.LAA-LosAngeles. (
Origin IGP, valid, external


So BGP route is via –

Let’s trace:

traceroute to (, 64 hops max, 52 byte packets
1 router.home ( 4.047 ms 0.875 ms 0.958 ms
2 ( 18.779 ms 17.490 ms 19.334 ms
3 ( 44.040 ms 32.802 ms 29.831 ms
4 ( 82.626 ms 87.126 ms 84.243 ms
5 ( 86.061 ms 85.503 ms 83.003 ms


Here we go!

So clearly BSNL on is placed in India and is having a BGP session with Tata AS6453 router in Los Angeles. This is over an IPLC circuit of Tata Communications. 


Possible fix…

Following an amazing quote – “Never call it a problem unless you have the solution!

So problem here is not really via Tata’s network. They are just selling bandwidth in form of two products – IP Transit & IPLC. It’s BSNL’s wrong idea of using IPLC carelessly. Likely BSNL won’t care or put much effort in fixing it. 

There can be a possible fix from Softlayer side. If they blackhole prefix announcement to BSNL AS9829 via Tata AS6453, BSNL will never pick their IPLC (or even IP) route. Instead they will just pick route via any other upstream like Airtel or Reliance Globalcom.  


Let’s look at relationship of Tata AS6453 with PCCW Global (upstream for Softlayer)

anurag:~ anurag$ whois -h whois.radb.net as6453 | grep -w AS3491
import: from AS3491 action pref = 100; accept AS-CAIS
export: to AS3491 announce AS-GLOBEINTERNET
import: from AS3491 action pref = 100; accept AS-CAIS
export: to AS3491 announce AS-GLOBEINTERNET
anurag:~ anurag$


Clearly both are peering! 

Based on presentation from Mr Amit Dunga (from Tata Communications) at SANOG, here’s list of BGP communities used by Tata AS6453:

Screen Shot 2013-06-01 at 12.30.35 AM



Thus if Softlayer could get it’s upstream providers (like PCCW in this specific case) to use 65009:9829 – this will ensure that route learnt by Tata AS6453 from PCCW Global AS3491 is NOT exported to BSNL AS9829. Thus BSNL will instead get route via Bharti Airtel AS9498 or Reliance AS18101.


I just sent this detailed info as email to Softlayer and BSNL. And oh yes – I don’t know why hostgator.in is hosted in Softlayer Singapore anyways. They provide hosting in India out of Ctrls datacenter. Why they host their own home site in Singapore is something beyond my understanding!


With hopes that your packets to Singapore are not routing via US, time for me to get back to my “cramming” for exams. 🙂

07 Nov

Google's routing issues because of an Indonesian ISP

Yesterday it was reported across networking community that Google’s prefixes were having issue due to an Indonesian ISP Moratel AS23947.


Quick analysis of what happened

From data logged by routeviews it seems like it wasn’t exactly a prefix hijack. AS23947 did not originated prefixes but rather had a route leak leading to path leak of AS23947 > AS15169

Here’s a view of global routing table for Google’s prefix at 15:57 GMT on 4th Nov:



Next at 02:07:27 GMT on 6th Nov morning, a route change is logged.

FROM 4436 15169 
TO 4436 3491 23947 15169


This path change is observed by routeviews only for one of its participating networks – nLayer AS4436. Rest participating networks seem not having any change. By 02:07:31 i.e within 4 seconds entire route goes via AS23947 for this specific network (and likely few more). At 02:35:06 i.e after 28 mins of this route leak, it is withdrawn and we can see within next few seconds direct route is preferred again. 


The impact was only on very limited part of internet because of fact that Google peers with lot of big networks directly and thus a short path is preferred  E.g Comcast in US will ignore this because Comcast AS36732 > Google AS15169 is short AS path as compared to Comcast > Someone else > Google. 


In above specific part – nLayer preferred long path likely because of its relation with Google. Most of networks prefer Customer routes (in which they get paid) over settlement free peering routes (in which they don’t get anything) over transit routes (in which they have to pay). The Indonesian ISP Moratel seems to have transit from few major players including PCCW Global which was likely running an unfiltered BGP session with its client. Thus largest impact came on PCCW Global Network which is Hong Kong based and fairly large in Asia. 


These glitches remind network engineers to careful configure router to avoid screwups! 🙂