26 Aug

Facebook FNA Nodes Updates

Earlier this year after APRICOT 2018, I posted a list of visible Facebook FNA (CDN caching) nodes across the world with IPv4, IPv6 and the AS name. I got quite a few mails in following months about people mentioning that they installed nodes but do not see their names in the list (and that was normal since list was static). 

I re-ran my script to see emailslatest status of nodes. During last check I saw 1689  nodes (3rd March). Now on 26th Aug i.e after close to 6 months, the total number of nodes has increased to 2204.

Here is the latest sheet containing the list of nodes with ASN, network name, IPv4 and IPv6 – http://link.anuragbhatia.com/fna30aug

Summary of the data and some findings for India

  1. The number of nodes increased by 27% within the last 6 months. 

  2. On Reliance Jio network number of node increased by just 1 – which is a new node they put in Ludhiana, Punjab. 

  3. In Delhi, a number of FNA nodes went up from 16 to 21. Four new additions are ACT (AS18209), MNR Broadband (AS133648), Facebook itself (AS63293) which is worth exploring) and GEONET GEOCITY NETWORK SOLUTIONS (AS45235). This actually makes me wonder why I do not see any FNA nodes on my ex-employer Spectra AS10029 as yet.  (30 Aug 2018 Update: I missed this, please see footer below)

  4. In case of Mumbai (or Bombay as used for BOM airport code), the number went up from 17 to 21. New additions HNS (AS38457), Airtel (AS9498) and Vortex Netsol  (AS136334). 

  5. For Chennai number stayed same at 6 (4 telco – Airtel, Jio, Vodafone, IDEA) and 2 broadband ISPs (ACT and Hathway). 

  6. In Kolkata IDEA added the new FNA node. Rest all seems the same. 

  7. I see zero active nodes in Dishnet Wireless (Aircel) now. Earlier there was one in Kolkata and one in Chennai. 

  8. Still zero active nodes in India’s largest govt. incumbent operator BSNL AS9289. They clearly do not understand the value of content caching nodes. 

  9. There’s a major growth in a number of FNA nodes in Airtel from 9 (in March) to 16 (now at the end of Aug). And for IDEA number went from 6 to 12. While the number of nodes in Vodafone stays same (14). 

  10. There’s no node in any of Tata telecom companies. 

Well, that’s all about for now. Have a good Rakshabandhan. 🙂

Update: 30th Aug 2018

My friends from Spectra pointed out that they do have a node and that made me to re-look at my scripts. Due to a bug in the scripts, I was not getting all the nodes. I have fixed the bug and updated the data in this post. 

07 Jul

Indian telecom voice market and updates

 

Suddenly the voice market in India is becoming very interesting. Earlier it was the case of Jio (and competitors) launching unlimited voice plans and now it’s the case of Govt. of India permitting IP telephony.

IP Telephony i.e networks where telephony happens over IP (not to be confused with IP to IP calls but) where IP to PSTN interconnects happen. Till a few months ago IP telephony (or IP-PSTN) interconnection was allowed only under certain conditions like doing it inside a building only for purpose of call centres (with OSP license) or running SIP trunks over private networks. Things like termination of calls originated from the apps was not allowed (where IP-PSTN was happening within India) as well as DID or Direct Inward Dialing numbers were not allowed. There were even cases where apps/businesses had to shut down due to confusing regulation. Here’s a nice article from Medianama about it. But all those were things of past.

In May Wifi calling or calls via Wifi where wifi is used loosely and it’s essentially called via any sort of Internet connections were permitted (news here). Later after TRAI’s clarification it now has been formally allowed. While it may not look as attractive as it should have been in the age of WhatsApp calling (IP to IP, not PSTN mess involved!), it still is quite interesting and going to bring some major change.

 

Here some of the upcoming things we all can expect to see in the next few months:

  1. All key operators will launch native wifi call offload for flagship phones (Google Pixel, iPhone, Samsung Galaxy’s etc). This will offload a hell lot of voice traffic from the cell towards home wifi. Various fixed wired ISPs would now be carrying a significant chunk of voice traffic.
  2. All key operators will launch an app for making phone calls and it would not only be for their users but also for other users. So while at this point one has to have a SIM card from the provider, next it would be sim card as well as “virtual connection” in form of a sort of KYC followed by an app essentially making use of SIP for call routing.
  3. SIP trunks over IP networks will become common and that would be huge. In present times if someone needed 5-10 connections for official use with call haunting etc, it was either POTS analogue phones or PRIs (yuck!) or SIP trunks running over the private network. Going forward now it would be SIP trunks offered over the regular internet all would be facilitated via closed systems (apps and portals) as well as open systems based on SIP. This would help significantly to businesses which have direct customer interaction.
  4. Market of DIDs or 10 digit virtual phone numbers will become very common. Telcos would be offering it directly and various platforms like Microsoft’s Skype, Google Voice, Vonage etc would also join in and resell those.

 

An interesting case of above is BSNL’s recent announcement of their platform “Wings”. Though based on their usual track record of totally screwing up, I would keep my expectations low, but still offering seems interesting and gives an idea of the updated regulatory framework.

15 Jun

Routing with North East India!

A few weeks back I got in touch with Marc from Meghalaya. He offered to host RIPE Atlas probe at Shillong and that’s an excellent location which isn’t there on RIPE Atlas coverage network yet. It took around 5 days for the probe to reach Shillong from Haryana. I think probably this probe is the one at the most beautiful place in India. 🙂

Now that probe is connected, I thought to look into routing which is super exciting for far from places like Shillong. Marc has a BSNL FTTH connection & mentioned about not-so-good latency. Let’s trace to 1st IP of the corresponding /24 pool on which probe is hosted:

traceroute -A 117.247.134.1
traceroute to 117.247.134.1 (117.247.134.1), 30 hops max, 60 byte packets
1 172.16.0.1 (172.16.0.1) [*] 0.552 ms 0.738 ms 0.909 ms
2 192.168.0.1 (192.168.0.1) [*] 3.070 ms 3.330 ms 3.266 ms
3 10.8.64.1 (10.8.64.1) [*] 18.401 ms 18.497 ms 18.097 ms
4 192.168.250.2 (192.168.250.2) [*] 18.655 ms 18.778 ms 24.928 ms
5 125.23.236.157 (125.23.236.157) [AS24560/AS9498] 25.758 ms 25.532 ms 25.251 ms
6 182.79.234.57 (182.79.234.57) [*] 181.730 ms 174.327 ms 174.075 ms
7 182.79.205.173 (182.79.205.173) [*] 50.784 ms 41.060 ms 41.120 ms
8 182.79.178.150 (182.79.178.150) [*] 177.159 ms 176.908 ms 182.79.234.195 (182.79.234.195) [*] 179.786 ms
9 182.79.247.165 (182.79.247.165) [*] 50.855 ms 182.79.177.99 (182.79.177.99) [*] 50.611 ms 182.79.247.115 (182.79.247.115) [*] 50.299 ms
10 182.79.206.46 (182.79.206.46) [*] 175.914 ms 182.79.190.125 (182.79.190.125) [*] 178.664 ms 182.79.222.189 (182.79.222.189) [*] 178.358 ms
11 149.3.183.129 (149.3.183.129) [AS6762] 175.873 ms 175.950 ms 149.3.183.125 (149.3.183.125) [AS6762] 174.901 ms
12 xe11-1-0.marsig2.mar.seabone.net (213.144.176.214) [AS6762] 196.814 ms xe7-3-0.marsig2.mar.seabone.net (213.144.176.172) [AS6762] 169.245 ms xe11-0-0.marsig2.mar.seabone.net (213.144.176.224) [AS6762] 161.673 ms
13 * * *
14 103-16-152-26-noc.bsccl.com (103.16.152.26) [AS132602] 249.369 ms 245.773 ms 249.437 ms
15 163.47.80-138-noc.bsccl.com (163.47.80.138) [AS132602] 213.663 ms 214.164 ms 213.697 ms
16 218.248.255.1 (218.248.255.1) [AS9829] 213.832 ms 210.132 ms 209.828 ms
17 * * *
18 218.248.170.229 (218.248.170.229) [AS9829] 213.674 ms 217.609 ms 217.687 ms
19 117.247.134.1 (117.247.134.1) [AS9829] 223.293 ms 223.366 ms 222.937 ms

 

This is interesting output. So there are two parts of it:

  1. Traffic going via Bangladesh
  2. Traffic to Bangladesh going via Europe!

 

While #1 may look like a routing issue, it’s actually desired result of a deal between BSNL & BSCCL. I blogged about it in last year when it was visible in BGP tables. Eventually, this link was launched by Indian Prime Minister Modi.

 

From the map, it seems like an ideal choice but I really wish BSNL went for some kind of circuits instead of transit with BSCCL. Reason being poor routing across Asian backbones which we see in reason #2.

 

Coming to #2 – this clearly is bad and broken. Traffic is hitting from Siti broadband > Airtel > Telecom Italia > BSCCL and this is resulting in traffic going from India to Europe first before returning to South Asia.

 

Let’s trace to same 1st IP of the pool from all Indian RIPE Atlas probes for a detailed picture:

Measurement result: https://atlas.ripe.net/measurements/8844267/

 

As we can see latency numbers are quite decent from BSNL’s AS9829 itself. 60-70ms seems fine considering it’s from the probes which are in North or South India to far away in North East. Let’s look at some of these traces from probes on BSNL itself:

 

 

This shows that there is indeed a direct backbone circuit of BSNL to that location. There’s a low chance of it being on top of BSCCL infra.

 

Except for BSNL, rest all other Indian networks are routing towards that BSNL segment in Meghalaya from Europe or Singapore/Hong Kong. All the ones from Europe are from Marseille in France. That’s the landing station for 11 cable systems:

  • SEACOM
  • SEA-ME-WE-4
  • EIG
  • I-ME-WE
  • Ariane 2
  • Atlas Offshore
  • Med Cable
  • TE North
  • Tamares Telecom
  • Alexandros
  • AAE-1 (Asia Africa Europe)

 

Out of these Se-Me-We-4 lands in Bangladesh and I guess that is being used by BSCCL for traffic. So coming back to why routing is so terrible from Indian networks towards BSNL in North East? To understand that we need to look at uplinks of BSCCL.

Well, BSNL is announcing 117.247.134.0/24 to BSCCL AS132602 only. BSCCL is buying transit from Telecom Italia AS6762 and NTT AS2914.

http://bgp.he.net/AS132602#_graph4

 

Looking at one of few traces from Europe:

 

213.144.176.194/31 TIS – BSCCL connectivity
213.144.176.194 – 10Gig port on TIS AS6762 router in Marseille
213.144.176.195 – TIS’s IP on BSCCL router in somewhere in Bangladesh

 

Next, looking at NTT AS2914 transit of BSCCL:

 

Here as traffic handoff from Tata AS6453 is happening to NTT AS2914 in Singapore (logical and correct!) and NTT to BSCCL also within Singapore.  The latency is high due to bad return. Here forward is slightly bad but not as bad as return possibly.

Let’s look at return trace to 2nd hop 115.118.168.1 from RIPE probe at destination (measurement here):

So clearly return path i.e Shillong to Hyderabad is via Europe because BSCCL used TIS for forwarding path.

 

So keeping above traces in mind, here’s the reason for high latency:

  1. BSNL is routing traffic over its backbone but rest all traffic i.e which is not going towards BSNL is being routed from Bangladeshi provider BSCCL.
  2. BSCCL is announcing routes to NTT AS2914 in Singapore & TIS AS6762 in France. Thus to send any traffic to BSNL’s segment in Meghalaya, one has to send it either via TIS router in Marseille, France or NTT Singapore. This adds up latency significantly for Indian networks (excluding) towards BSNL Meghalaya.
  3. BSCCL is using TIS AS6762 to reach Tata AS6453 and this is resulting in very bad return route and thus Meghalaya to any other network in India who is Tata AS6453 downstream is via Marseille, France.

Quite a lot seems messed up. BSNL’s should at least start announcing 117.247.134.0/24 immediately across NIXI’s subject to capacity between their core network in North East. If there’s a capacity constrained, they should use L1 circuits from BSCCL to connect network in Shillong instead of IP transit.

 

How is BSNL in North East reaching Google?

Seems direct to BSNL’s PNI with Google within India.