04 Jan

VoWifi experience on Jio

Since last week of Nov 2019, I am having serious issues with Airtel at my home. Somehow 4G signal SNR is very poor and most of the calls just fail on that. Airtel support just mentioned that they are putting a new site in my area in Jan 2020 but fail to explain why suddenly it went so bad. I can imagine that support team staff does not have visibility to network in real-time and likely it would be an issue with the 4G antenna on one or more towers. 2G signal was good but latency was extremely high to connect call plus calls still failed regardless (maybe due to high strain on 2G).

So by the end of December, I just gave up and gave the request for porting to Jio. Yesterday my number started on Jio and around 24 hours later I got access to VoWifi which is quite nice. Both Airtel & Jio are rolling our VoWifi but for Airtel, many users have reported it for being locked on Airtel fixed line only.

What is VoWifi?

Well, it’s just native offloading of voice calls over the fixed-line network and is extremely useful for both network operators as well as end-users. Native IP equipment is cheap and it’s way easy to extend cell coverage at home via Wifi instead of proprietary inbuilding solutions. Plus landline was good for quality but form factor and usability was pain. Now VoWifi kind of merges the fixed-line and cell phones and we get best of both worlds.

I think many people confused BSNL Wings with VoWifi. VoWifi is seamless offloading of calls natively while BSNL Wings offered a separate number, an app to run and a separate billing account to maintain with the operator. BSNL Wings was a bad product idea and really awful execution.

Coming to Jio VoWifi, some misc observations:

  1. It’s working for me on the non-Jio fixed-line connection in Haryana circle. I have IAXN (GEPON FTTH) and Siti broadband (DOCSIS 3.0) at home running in active/backup configuration for high availability.
  2. The device has an IPsec tunnel with 49.44.59.XXX. I see isakmp-nat-keep-alive packets every 20 seconds when I do tcpdump on the router.
  3. On my cell phone (which is an Apple iOS device) I see a tunnel is opened on IPSEC1 port. IPSEC2 & IPSEC3 are there for VoLTE.
    IPSEC1 has a link-local IPv6 and an IPv6 from 2405:205:3000::/36 range. The IPv6 on all three IPSEC tunnels is the same as of pdp_ip1 (Cellular data port). I guess that’s probably how they are handing the handovers between VoWifi and VoLTE.
  4. The IPv6 on pdp_ip0 for cellular data is from a different range which again is being announced in the global table as 2409:4051::/36. I wonder why Jio is using publically routable pool for VoLTE and VoWifi. In case of Airtel it a non-routed address space and routes are not announced in the global routing table.
  5. In my test VoWifi to VoLTE was reasonably OK. I tested by shutting off wifi on the cell phone as well as unplugging the cable from wifi AP to test. In both cases call went silent for 1-2 seconds and resumed over VoLTE. It did not come back on VoWifi once Wifi was back live.
  6. Latency from both connections to other endpoint (the router before the IPSEC endpoint) is around 60ms. Either the endpoint is somewhere far off or (more likely) it’s a case of not-so-good routing between Airtel (upstream of my ISP) and Jio.
  7. There’s a quite noticeable improvement in connection time. For IVR based numbers it’s close to 2-3 seconds. Which is a great improvement for someone like me who was on 2G non-VoLTE calls for December!

How to check interfaces and routing table on your phone?

Well use HE.NET Network App on Android or Apple iPhone and you will be able to check that!

That’s about it. Back to the world of routing. 🙂

05 Apr

Tata – Airtel domestic peering IRR filtering and OpenDNS latency!

Last month I noticed quite high latency with Cisco’s OpenDNS from my home fibre connection. The provider at home is IAXN (AS134316) which is peering with content folks in Delhi besides transit from Airtel.

ping -c 5 208.67.222.222
PING 208.67.222.222 (208.67.222.222) 56(84) bytes of data.
64 bytes from 208.67.222.222: icmp_seq=1 ttl=51 time=103 ms
64 bytes from 208.67.222.222: icmp_seq=2 ttl=51 time=103 ms
64 bytes from 208.67.222.222: icmp_seq=3 ttl=51 time=103 ms
64 bytes from 208.67.222.222: icmp_seq=4 ttl=51 time=103 ms
64 bytes from 208.67.222.222: icmp_seq=5 ttl=51 time=103 ms


--- 208.67.222.222 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4005ms
rtt min/avg/max/mdev = 103.377/103.593/103.992/0.418 ms

This is bit on the higher side as from Haryana to Mumbai (OpenDNS locations list here). My ISP is backhauling from Faridabad which is probably 6-8ms away from my city and 2-3ms further to Delhi and from there to Mumbai around 30ms. Thus latency should be around ~40-45ms.

Here’s how forward trace looked like

traceroute 208.67.222.222
traceroute to 208.67.222.222 (208.67.222.222), 30 hops max, 60 byte packets
 1  172.16.0.1 (172.16.0.1)  0.730 ms  0.692 ms  0.809 ms
 2  axntech-dynamic-218.140.201.103.axntechnologies.in (103.201.140.218)  4.904 ms  4.314 ms  4.731 ms
 3  10.10.26.1 (10.10.26.1)  6.000 ms  6.414 ms  6.326 ms
 4  10.10.26.9 (10.10.26.9)  6.836 ms  7.135 ms  7.047 ms
 5  nsg-static-77.249.75.182-airtel.com (182.75.249.77)  9.344 ms  9.416 ms  9.330 ms
 6  182.79.243.201 (182.79.243.201)  62.274 ms 182.79.177.69 (182.79.177.69)  66.874 ms 182.79.239.193 (182.79.239.193)  61.297 ms
 7  121.240.1.201 (121.240.1.201)  85.789 ms  82.250 ms  79.591 ms
 8  172.25.81.134 (172.25.81.134)  110.049 ms 172.31.29.245 (172.31.29.245)  114.350 ms  113.673 ms
 9  172.31.133.210 (172.31.133.210)  112.598 ms 172.19.138.86 (172.19.138.86)  114.889 ms 172.31.133.210 (172.31.133.210)  113.415 ms
10  115.110.234.50.static.mumbai.vsnl.net.in (115.110.234.50)  125.770 ms  125.056 ms  123.779 ms
11  resolver1.opendns.com (208.67.222.222)  113.648 ms  115.044 ms  106.066 ms

Forward trace looks fine except that latency jumps as soon as we hit Tata AS4755 backbone. OpenDNS connects with Tata AS4755 inside India and announces their anycast prefixes to them. If the forward trace is logically correct but has high latency, it often reflects the case of bad return path. Thus I requested friends at OpenDNS to share the return path towards me. As expected, it was via Tata AS6453 Singapore.

Here’s what Tata AS4755 Mumbai router had for IAXN prefix:

BGP routing table entry for 14.102.188.0/22 
Paths: (1 available, best #1, table Default-IP-Routing-Table) 
Not advertised to any peer 
6453 9498 134316 134316 134316 134316 134316 134316 134316 134316 134316 134316 
192.168.203.194 from 192.168.199.193 (192.168.203.194) 
Origin IGP, localpref 62, valid, internal, best 
Community: 4755:44 4755:97 4755:888 4755:2000 4755:3000 4755:47552 6453:50 6453:3000 6453:3400 6453:3402 
Originator: 192.168.203.194, Cluster list: 192.168.199.193 192.168.194.15 
Last update: Mon Mar 25 15:26:36 2019

Thus what was happening is this:

Forward path: IAXN (AS134316) > Airtel (AS9498) > Tata (AS4755) > OpenDNS (AS36692)

Return path: OpenDNS (AS36692) > Tata (AS4755) > Tata (AS6453) > Airtel (AS9498) > IAXN (AS134316)

While this may seem like a Tata – Airtel routing issue but it wasn’t. I could see some of the prefixes with a direct path as well. Here’s a trace from Tata AS4755 Mumbai PoP to an IP from a different pool of IAXN:

traceroute to 103.87.46.1 (103.87.46.1), 15 hops max, 60 byte packets
1 * * *
2 172.31.170.210 (172.31.170.210) 0.911 ms 0.968 ms 0.643 ms
3 172.23.78.233 (172.23.78.233) 1.233 ms 0.821 ms 0.810 ms
4 172.17.125.249 (172.17.125.249) 23.540 ms 23.454 ms 23.367 ms
5 115.110.232.174.static.Delhi.vsnl.net.in (115.110.232.174) 49.175 ms 48.832 ms 49.107 ms
6 182.79.153.87 (182.79.153.87) 48.777 ms 182.79.153.83 (182.79.153.83) 49.043 ms 182.79.177.127 (182.79.177.127) 54.879 ms
7 103.87.46.1 (103.87.46.1) 60.865 ms 60.540 ms 60.644 ms

This clearly was fine. So why Tata was treating 103.87.46.0/24 different from 14.102.188.0/22? The reason for that lies in following:

  • Airtel (AS9498) very likely peers with Tata (AS4755). They do interconnect for sure as we see in traceroutes and my understanding is that it’s based on settlement-free peering for Indian traffic.
  • Airtel (AS9498) buys IP transit from Tata (AS6453) (besides a few others). Tata AS6453 is carrying the routing announcements to other networks in the transit free zone and that confirms that Airtel (at least technically) has a downstream customer relationship here.
  • Tata (AS4755) has IRR based filters on peering but not the Tata (AS6453) for it’s downstream. Hence while Tata rejected the route in India, they did accept that in Singapore PoP.
  • My IP was from prefix 14.102.188.0/22 and there was no valid route object for it at any of key IRRs like ATLDB, APNIC or RADB. But other prefix 103.87.46.0/24 did had a valid route object on APNIC.

Now after almost 10 days of it, my ISP has changed the BGP announcement and announcing 14.102.189.0/24 (which does a valid route object on APNIC). This fixes the routing problem and give me pretty decent latency with OpenDNS:

ping -c 5 208.67.222.222
PING 208.67.222.222 (208.67.222.222): 56 data bytes
64 bytes from 208.67.222.222: icmp_seq=0 ttl=55 time=52.552 ms
64 bytes from 208.67.222.222: icmp_seq=1 ttl=55 time=53.835 ms
64 bytes from 208.67.222.222: icmp_seq=2 ttl=55 time=53.330 ms
64 bytes from 208.67.222.222: icmp_seq=3 ttl=55 time=52.700 ms
64 bytes from 208.67.222.222: icmp_seq=4 ttl=55 time=52.504 ms

--- 208.67.222.222 ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 52.504/52.984/53.835/0.518 ms

So if you are a network operator and originating prefixes, please do document them in any of the IRRs. You can do that via IRR of your RIR (APNIC, ARIN etc) or a free IRR like ALTDB. If you have downstreams, make sure to create AS SET, add downstreams ASNs in your AS SET and also include that AS SET on peeringdb for the world to see!

Misc Notes

  • Posted strictly in my personal capacity and has nothing to do with my employrer.
  • Thanks for folks from Cisco/OpenDNS for quick replies with relevant data which helped in troubleshooting. 🙂
26 Aug

Facebook FNA Nodes Updates

Earlier this year after APRICOT 2018, I posted a list of visible Facebook FNA (CDN caching) nodes across the world with IPv4, IPv6 and the AS name. I got quite a few mails in following months about people mentioning that they installed nodes but do not see their names in the list (and that was normal since list was static). 

I re-ran my script to see emailslatest status of nodes. During last check I saw 1689  nodes (3rd March). Now on 26th Aug i.e after close to 6 months, the total number of nodes has increased to 2204.

Here is the latest sheet containing the list of nodes with ASN, network name, IPv4 and IPv6 – http://link.anuragbhatia.com/fna30aug

Summary of the data and some findings for India

  1. The number of nodes increased by 27% within the last 6 months. 

  2. On Reliance Jio network number of node increased by just 1 – which is a new node they put in Ludhiana, Punjab. 

  3. In Delhi, a number of FNA nodes went up from 16 to 21. Four new additions are ACT (AS18209), MNR Broadband (AS133648), Facebook itself (AS63293) which is worth exploring) and GEONET GEOCITY NETWORK SOLUTIONS (AS45235). This actually makes me wonder why I do not see any FNA nodes on my ex-employer Spectra AS10029 as yet.  (30 Aug 2018 Update: I missed this, please see footer below)

  4. In case of Mumbai (or Bombay as used for BOM airport code), the number went up from 17 to 21. New additions HNS (AS38457), Airtel (AS9498) and Vortex Netsol  (AS136334). 

  5. For Chennai number stayed same at 6 (4 telco – Airtel, Jio, Vodafone, IDEA) and 2 broadband ISPs (ACT and Hathway). 

  6. In Kolkata IDEA added the new FNA node. Rest all seems the same. 

  7. I see zero active nodes in Dishnet Wireless (Aircel) now. Earlier there was one in Kolkata and one in Chennai. 

  8. Still zero active nodes in India’s largest govt. incumbent operator BSNL AS9289. They clearly do not understand the value of content caching nodes. 

  9. There’s a major growth in a number of FNA nodes in Airtel from 9 (in March) to 16 (now at the end of Aug). And for IDEA number went from 6 to 12. While the number of nodes in Vodafone stays same (14). 

  10. There’s no node in any of Tata telecom companies. 

Well, that’s all about for now. Have a good Rakshabandhan. 🙂

Update: 30th Aug 2018

My friends from Spectra pointed out that they do have a node and that made me to re-look at my scripts. Due to a bug in the scripts, I was not getting all the nodes. I have fixed the bug and updated the data in this post.