05 Apr

Tata – Airtel domestic peering IRR filtering and OpenDNS latency!

Last month I noticed quite high latency with Cisco’s OpenDNS from my home fibre connection. The provider at home is IAXN (AS134316) which is peering with content folks in Delhi besides transit from Airtel.

ping -c 5 208.67.222.222
PING 208.67.222.222 (208.67.222.222) 56(84) bytes of data.
64 bytes from 208.67.222.222: icmp_seq=1 ttl=51 time=103 ms
64 bytes from 208.67.222.222: icmp_seq=2 ttl=51 time=103 ms
64 bytes from 208.67.222.222: icmp_seq=3 ttl=51 time=103 ms
64 bytes from 208.67.222.222: icmp_seq=4 ttl=51 time=103 ms
64 bytes from 208.67.222.222: icmp_seq=5 ttl=51 time=103 ms


--- 208.67.222.222 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4005ms
rtt min/avg/max/mdev = 103.377/103.593/103.992/0.418 ms

This is bit on the higher side as from Haryana to Mumbai (OpenDNS locations list here). My ISP is backhauling from Faridabad which is probably 6-8ms away from my city and 2-3ms further to Delhi and from there to Mumbai around 30ms. Thus latency should be around ~40-45ms.

Here’s how forward trace looked like

traceroute 208.67.222.222
traceroute to 208.67.222.222 (208.67.222.222), 30 hops max, 60 byte packets
 1  172.16.0.1 (172.16.0.1)  0.730 ms  0.692 ms  0.809 ms
 2  axntech-dynamic-218.140.201.103.axntechnologies.in (103.201.140.218)  4.904 ms  4.314 ms  4.731 ms
 3  10.10.26.1 (10.10.26.1)  6.000 ms  6.414 ms  6.326 ms
 4  10.10.26.9 (10.10.26.9)  6.836 ms  7.135 ms  7.047 ms
 5  nsg-static-77.249.75.182-airtel.com (182.75.249.77)  9.344 ms  9.416 ms  9.330 ms
 6  182.79.243.201 (182.79.243.201)  62.274 ms 182.79.177.69 (182.79.177.69)  66.874 ms 182.79.239.193 (182.79.239.193)  61.297 ms
 7  121.240.1.201 (121.240.1.201)  85.789 ms  82.250 ms  79.591 ms
 8  172.25.81.134 (172.25.81.134)  110.049 ms 172.31.29.245 (172.31.29.245)  114.350 ms  113.673 ms
 9  172.31.133.210 (172.31.133.210)  112.598 ms 172.19.138.86 (172.19.138.86)  114.889 ms 172.31.133.210 (172.31.133.210)  113.415 ms
10  115.110.234.50.static.mumbai.vsnl.net.in (115.110.234.50)  125.770 ms  125.056 ms  123.779 ms
11  resolver1.opendns.com (208.67.222.222)  113.648 ms  115.044 ms  106.066 ms

Forward trace looks fine except that latency jumps as soon as we hit Tata AS4755 backbone. OpenDNS connects with Tata AS4755 inside India and announces their anycast prefixes to them. If the forward trace is logically correct but has high latency, it often reflects the case of bad return path. Thus I requested friends at OpenDNS to share the return path towards me. As expected, it was via Tata AS6453 Singapore.

Here’s what Tata AS4755 Mumbai router had for IAXN prefix:

BGP routing table entry for 14.102.188.0/22 
Paths: (1 available, best #1, table Default-IP-Routing-Table) 
Not advertised to any peer 
6453 9498 134316 134316 134316 134316 134316 134316 134316 134316 134316 134316 
192.168.203.194 from 192.168.199.193 (192.168.203.194) 
Origin IGP, localpref 62, valid, internal, best 
Community: 4755:44 4755:97 4755:888 4755:2000 4755:3000 4755:47552 6453:50 6453:3000 6453:3400 6453:3402 
Originator: 192.168.203.194, Cluster list: 192.168.199.193 192.168.194.15 
Last update: Mon Mar 25 15:26:36 2019

Thus what was happening is this:

Forward path: IAXN (AS134316) > Airtel (AS9498) > Tata (AS4755) > OpenDNS (AS36692)

Return path: OpenDNS (AS36692) > Tata (AS4755) > Tata (AS6453) > Airtel (AS9498) > IAXN (AS134316)

While this may seem like a Tata – Airtel routing issue but it wasn’t. I could see some of the prefixes with a direct path as well. Here’s a trace from Tata AS4755 Mumbai PoP to an IP from a different pool of IAXN:

traceroute to 103.87.46.1 (103.87.46.1), 15 hops max, 60 byte packets
1 * * *
2 172.31.170.210 (172.31.170.210) 0.911 ms 0.968 ms 0.643 ms
3 172.23.78.233 (172.23.78.233) 1.233 ms 0.821 ms 0.810 ms
4 172.17.125.249 (172.17.125.249) 23.540 ms 23.454 ms 23.367 ms
5 115.110.232.174.static.Delhi.vsnl.net.in (115.110.232.174) 49.175 ms 48.832 ms 49.107 ms
6 182.79.153.87 (182.79.153.87) 48.777 ms 182.79.153.83 (182.79.153.83) 49.043 ms 182.79.177.127 (182.79.177.127) 54.879 ms
7 103.87.46.1 (103.87.46.1) 60.865 ms 60.540 ms 60.644 ms

This clearly was fine. So why Tata was treating 103.87.46.0/24 different from 14.102.188.0/22? The reason for that lies in following:

  • Airtel (AS9498) very likely peers with Tata (AS4755). They do interconnect for sure as we see in traceroutes and my understanding is that it’s based on settlement-free peering for Indian traffic.
  • Airtel (AS9498) buys IP transit from Tata (AS6453) (besides a few others). Tata AS6453 is carrying the routing announcements to other networks in the transit free zone and that confirms that Airtel (at least technically) has a downstream customer relationship here.
  • Tata (AS4755) has IRR based filters on peering but not the Tata (AS6453) for it’s downstream. Hence while Tata rejected the route in India, they did accept that in Singapore PoP.
  • My IP was from prefix 14.102.188.0/22 and there was no valid route object for it at any of key IRRs like ATLDB, APNIC or RADB. But other prefix 103.87.46.0/24 did had a valid route object on APNIC.

Now after almost 10 days of it, my ISP has changed the BGP announcement and announcing 14.102.189.0/24 (which does a valid route object on APNIC). This fixes the routing problem and give me pretty decent latency with OpenDNS:

ping -c 5 208.67.222.222
PING 208.67.222.222 (208.67.222.222): 56 data bytes
64 bytes from 208.67.222.222: icmp_seq=0 ttl=55 time=52.552 ms
64 bytes from 208.67.222.222: icmp_seq=1 ttl=55 time=53.835 ms
64 bytes from 208.67.222.222: icmp_seq=2 ttl=55 time=53.330 ms
64 bytes from 208.67.222.222: icmp_seq=3 ttl=55 time=52.700 ms
64 bytes from 208.67.222.222: icmp_seq=4 ttl=55 time=52.504 ms

--- 208.67.222.222 ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 52.504/52.984/53.835/0.518 ms

So if you are a network operator and originating prefixes, please do document them in any of the IRRs. You can do that via IRR of your RIR (APNIC, ARIN etc) or a free IRR like ALTDB. If you have downstreams, make sure to create AS SET, add downstreams ASNs in your AS SET and also include that AS SET on peeringdb for the world to see!

Misc Notes

  • Posted strictly in my personal capacity and has nothing to do with my employrer.
  • Thanks for folks from Cisco/OpenDNS for quick replies with relevant data which helped in troubleshooting. šŸ™‚
03 Jul

Reliance Jio orignating Charter’s /16 pool

 

Just noticed that Reliance Jio (AS55836) seems to be originatingĀ a /16 which is for Charter Communications (AS20115) –Ā 47.35.0.0/16.

 

route-views>sh ip bgp 47.35.0.0/16 long | exclude 20115
BGP table version is 18764390, local router ID is 128.223.51.103
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network          Next Hop            Metric LocPrf Weight Path
*   47.35.0.0/16     195.208.112.161                        0 3277 3267 174 64049 55836 i
*                    217.192.89.50                          0 3303 6762 64049 55836 i
*                    212.66.96.126                          0 20912 1267 64049 55836 i
*                    162.243.188.2                          0 393406 6453 6762 64049 55836 i
*                    192.241.164.4                          0 62567 2914 174 64049 55836 i
*                    129.250.0.11          1007             0 2914 174 64049 55836 i
*                    104.192.216.1                          0 46450 174 64049 55836 i
*                    202.93.8.242                           0 24441 3491 3491 174 64049 55836 i
*                    66.59.190.221                          0 6539 577 6762 64049 55836 i
*                    144.228.241.130         80             0 1239 174 64049 55836 i
*                    207.172.6.20             0             0 6079 3356 174 64049 55836 i
*                    203.62.252.83                          0 1221 4637 174 64049 55836 i
*                    93.104.209.174                         0 58901 51167 3356 6762 64049 55836 i
Network          Next Hop            Metric LocPrf Weight Path
*                    162.250.137.254                        0 4901 174 64049 55836 i
*                    4.69.184.193             0             0 3356 174 64049 55836 i
*                    208.51.134.254           1             0 3549 3356 174 64049 55836 i
*                    89.149.178.10           10             0 3257 174 64049 55836 i
*                    66.110.0.86                            0 6453 6762 64049 55836 i
*                    134.222.87.1           650             0 286 174 64049 55836 i
*                    95.85.0.2                              0 200130 6453 174 64049 55836 i
*                    12.0.1.63                              0 7018 174 64049 55836 i
*                    173.205.57.234                         0 53364 3257 174 64049 55836 i
*                    206.24.210.80                          0 3561 174 64049 55836 i
*                    5.101.110.2                            0 202018 2914 174 64049 55836 i
*                    207.172.6.1              0             0 6079 3356 174 64049 55836 i
*                    154.11.98.225            0             0 852 174 64049 55836 i
*                    194.85.40.15                           0 3267 174 64049 55836 i
*                    208.74.64.40                           0 19214 174 64049 55836 i
*                    209.124.176.223                        0 101 101 174 64049 55836 i
*                    66.185.128.48            6             0 1668 174 64049 55836 i
*                    203.181.248.168                        0 7660 2516 6762 64049 55836 i
*                    202.232.0.2                            0 2497 701 6762 64049 55836 i
*                    103.247.3.45                           0 58511 64049 55836 i
*                    193.0.0.56                             0 3333 1103 64049 55836 i
Network          Next Hop            Metric LocPrf Weight Path
*                    80.241.176.31                          0 20771 47872 64049 55836 i
*>                   216.218.252.164                        0 6939 64049 55836 i
*                    132.198.255.253                        0 1351 174 64049 55836 i
*                    103.255.249.22                         0 58443 45177 64049 55836 i
*                    114.31.199.1                           0 4826 174 64049 55836 i
route-views>

 

This shows Reliance Jio’s ASN AS55836 announcingĀ 47.35.0.0/16. Charter Communications (AS20115) is originating multiple of /18s out of the same pool.

 

route-views>sh ip bgp 47.35.0.0/16 long
BGP table version is 18764237, local router ID is 128.223.51.103
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network          Next Hop            Metric LocPrf Weight Path
*   47.35.0.0/18     195.208.112.161                        0 3277 3267 6939 20115 i
*                    212.66.96.126                          0 20912 6939 20115 i
*                    162.243.188.2                          0 393406 6939 20115 i
*                    192.241.164.4                          0 62567 6939 20115 i
*                    129.250.0.11          1006             0 2914 1299 20115 i
*                    4.69.184.193             0             0 3356 20115 i
*                    89.149.178.10           10             0 3257 3356 20115 i
*                    194.85.40.15                           0 3267 6939 20115 i
*                    216.218.252.164                        0 6939 20115 i

 

Let’s look for whois record of this /16 pool:Ā https://whois.arin.net/rest/net/NET-47-32-0-0-1

NetRange: 47.32.0.0 - 47.51.255.255
CIDR: 47.32.0.0/12, 47.48.0.0/14
NetName: CC04
NetHandle: NET-47-32-0-0-1
Parent: NET47 (NET-47-0-0-0-0)
NetType: Direct Allocation
OriginAS:
Organization: Charter Communications (CC04)
RegDate: 2014-12-23
Updated: 2014-12-23
Ref: https://whois.arin.net/rest/net/NET-47-32-0-0-1

 

It seems highly unlikely that this pool has been transferred over to Jio and hence it’s highly likely that Reliance Jio is falsely advertising this pool. Possible reason for such mistake can be because they have gotĀ 47.29.0.0/16,Ā 47.30.0.0/16 andĀ 47.31.0.0/16 but beyond that there are few Charter Pools. So likely someone just mistyped an IP. Another possible reason for such mistype can be that they have also gotĀ 49.35.0.0/16 (it’s 49, not 47).

What is much more fascinating is that they even have a RADB route object registered for the same.

whois -h whois.radb.net 47.35.0.0
route:      47.35.0.0/16
descr:      Route Set JIO Maharastra LTE USER Pool
origin:     AS55836
notify:     Ip.abuse@ril.com
mnt-by:     MAINT-AS55836
changed:    ip.management@ril.com 20160624
source:     RADB

 

World of BGP is interesting!

 

Update:

As per update from Mr Morlay Gosh from Reliance Jio on SANOG mailing list- This was mistake and they are removing the same.

http://article.gmane.org/gmane.org.operators.sanog/2929