03 Mar

Mapping Facebook's FNA (CDN) nodes across the world!

Just back from APRICOT 2018. As I mentioned in my previous blog post, APNIC had its first Hackathon and it was fun (blog post of APNIC here). There was one project on the ranking of CDNs using RIPE Atlas data. To achieve this team was trying to find strings/hostnames which they can trace to and figure out nearby CDN.
As part of that, I suggested them to look into www.facebook.com and carefully noting the sources from where elements get loaded. It’s quite common that Facebook.com (or Google.com for the logic) would be hosted on some server at a large PoP while FNA (or GGC) would serve only specific static content out of it. FNA, of course, sits on the IPs of the ISP hosting it.
So in the source list, we found scontent.fktm1-1.fna.fbcdn.net and that gives an idea that FNA strings are around logic: scontent.fxxx1-1.fna.fbcdn.net where xxx is the airport code. 1-1 means 1st PoP in 1st ISP over there probably (strong guess!). If there are more FNA nodes in a given area, the number goes further up. The team used it and for now, the project is over. But while I was on the way back to India, I thought that this is very interesting data if we pull the full picture by querying all possible IATA airport codes with a logic.
This logic can be used for two things:

  1. Finding locations of all FNA nodes and plot them on the map
  2. Find which networks/ASNs in the world host it

In my check with a script querying it all, I found there are 1689 of FNA’s nodes deployed across the world. Here’s a map of the presence of FNA:

Some of the interesting regional stats from the data

India

Bharti Airtel AS9498 is hosting Facebook caching FNA nodes at Bangalore, Mumbai, Calcutta, Delhi, Hyderabad, Chandigarh, Jaipur, Chennai and Patna i.e 9 cities while in case of Jio they are hosted in 19 cities – Agra, Ahmedabad, Bhubaneswar, Bhopal, Bangalore, Mumbai, Calcutta, Cochin, Delhi, Delhi, Guwahati, Hyderabad, Jammu, Jaipur, Madras, Nagpur, Patna, Simla, Srinagar. IDEA Cellular has them at 6 locations Bangalore
Mumbai, Delhi, Hyderabad, Indore and Pune.

FNA nodes in Indian metro cities

ASNAS NameIPv4IPv6Airport CodeAirport Desc
9498BBIL-AP BHARTI Airtel Ltd., IN157.240.191.172a03:2887:ff05:0:face:b00c:0:a7BOMBombay
45769DVOIS-IN D-Vois Broadband Pvt Ltd, IN1.186.190.172400:1f00:0:d:face:b00c:0:a7BOMBombay
55836RELIANCEJIO-IN Reliance Jio Infocomm Limited, IN49.44.63.172405:200:1602:2885:face:b00c:0:a7BOMBombay
55644IDEANET1-IN Idea Cellular Limited, IN223.196.3.812400:c700:0:5:face:b00c:0:a7BOMBombay
58678INTECHONLINE-IN Intech Online Private Limited, IN124.108.16.2092404:bd00:1:4:face:b00c:0:a7BOMBombay
55352MCPL-IN Microscan Computers Private Limited, IN103.226.191.2092406:9e00:3:1:face:b00c:0:a7BOMBombay
18196SEVENSTAR-AS Seven Star Internet Service Provider, IN202.134.164.2092402:f200:301:0:face:b00c:0:a7BOMBombay
45194SIPL-AS Syscon Infoway Pvt. Ltd., IN110.5.75.172401:a100:2:4:face:b00c:0:a7BOMBombay
45117INPL-IN-AP Ishan_s Network, IN103.214.129.17BOMBombay
133720SOFTCALLCOC-AS SOFT CALL CUST-O-CARE PRIVATE LIMITED, IN139.5.47.209BOMBombay
133232SAMPARKESTATES-AS-IN SAMPARK ESTATES PVT. LTD., IN103.69.221.172001:df7:6e00:3:face:b00c:0:a7BOMBombay
17665IN2CABLE-AP AS Number of In2cable.com (India) Ltd., IN203.192.223.145BOMBombay
55832HOMESYSTEM-AS-AP HOME SYSTEMS PVT.LTD, IN120.88.176.145BOMBombay
55862WNET-IN Wan & Lan Internet Pvt Ltd, IN49.128.164.17BOMBombay
24554FIVE-NET-AS-IN Fivenetwork Solution India Pvt Ltd Internet, IN202.177.230.145BOMBombay
17488HATHWAY-NET-AP Hathway IP Over Cable Internet, IN202.88.184.17BOMBombay
38266HUTCHVAS-AS Vodafone Essar Ltd., Telecommunication – Value Added Services, IN1.38.8.172402:3a80:c002:13:face:b00c:0:a7BOMBombay
17747ZIML-AP SITI NETWORKS LIMITED, IN103.225.178.2092406:3c80:1:3:face:b00c:0:a7CCUCalcutta
55836RELIANCEJIO-IN Reliance Jio Infocomm Limited, IN49.44.63.2092405:200:1606:2885:face:b00c:0:a7CCUCalcutta
23860ALLIANCE-GATEWAY-AS-AP Alliance Broadband Services Pvt. Ltd., IN203.171.243.145CCUCalcutta
9498BBIL-AP BHARTI Airtel Ltd., IN157.240.185.172a03:2887:ff06:0:face:b00c:0:a7CCUCalcutta
45804MEGHBELA-IN MEGHBELA BROADBAND, IN103.56.238.17CCUCalcutta
45334AIRCEL-AS-AP Dishnet Wireless Limited, IN202.148.205.172402:4c00:ffff:ffed:face:b00c:0:a7CCUCalcutta
38266HUTCHVAS-AS Vodafone Essar Ltd., Telecommunication – Value Added Services, IN1.38.7.2092402:3a80:c009:13:face:b00c:0:a7CCUCalcutta
NANA223.196.149.1452400:c700:a000:0:face:b00c:0:a7CCUCalcutta
9498BBIL-AP BHARTI Airtel Ltd., IN157.240.189.172a03:2887:ff03:0:face:b00c:0:a7DELDelhi
55644IDEANET1-IN Idea Cellular Limited, IN223.196.64.172400:c700:4000:0:face:b00c:0:a7DELDelhi
55836RELIANCEJIO-IN Reliance Jio Infocomm Limited, IN49.44.63.1452405:200:160b:2885:face:b00c:0:a7DELDelhi
17747ZIML-AP SITI NETWORKS LIMITED, IN103.217.246.172402:ea80:0:1:face:b00c:0:a7DELDelhi
132116ANINETWORK-IN Ani Network Pvt Ltd, IN45.248.174.81DELDelhi
132453TRIPLE-PLAY-IN TRIPLE PLAY BROADBAND PRIVATE LIMITED, IN150.242.84.145DELDelhi
58965ABSPL-AS-IN ANJANI BROADBAND SOLUTIONS PVT.LTD., IN103.233.117.81DELDelhi
133982EXCITEL-AS-IN Excitel Broadband Private Limited, IN103.56.230.209DELDelhi
45184DEN-ISP-AS-IN-AP Den Digital Entertainment Pvt. Ltd. AS ISP india, IN112.196.177.17DELDelhi
133982EXCITEL-AS-IN Excitel Broadband Private Limited, IN103.48.198.209DELDelhi
55836RELIANCEJIO-IN Reliance Jio Infocomm Limited, IN49.44.62.1452405:200:1605:2885:face:b00c:0:a7DELDelhi
17488HATHWAY-NET-AP Hathway IP Over Cable Internet, IN202.88.147.145DELDelhi
38266HUTCHVAS-AS Vodafone Essar Ltd., Telecommunication – Value Added Services, IN1.38.7.812402:3a80:c005:13:face:b00c:0:a7DELDelhi
NANA233.196.145.1452400:c700:a001:0:face:b00c:0:a7DELDelhi
133275GIGANTIC-AS Gigantic Infotel Pvt Ltd, IN103.59.199.145DELDelhi
38266HUTCHVAS-AS Vodafone Essar Ltd., Telecommunication – Value Added Services, IN1.38.5.172402:3a80:c006:13:face:b00c:0:a7DELDelhi
9498BBIL-AP BHARTI Airtel Ltd., IN157.240.190.172a03:2887:ff04:0:face:b00c:0:a7MAAMadras
55836RELIANCEJIO-IN Reliance Jio Infocomm Limited, IN49.44.63.812405:200:1607:2885:face:b00c:0:a7MAAMadras
24309CABLELITE-AS-AP Atria Convergence Technologies Pvt. Ltd. Broadband Internet Service Provider INDIA, IN49.207.35.172406:7400:b7:0:face:b00c:0:a7MAAMadras
45334AIRCEL-AS-AP Dishnet Wireless Limited, IN202.148.204.1452402:4c00:ffff:ffec:face:b00c:0:a7MAAMadras
38266HUTCHVAS-AS Vodafone Essar Ltd., Telecommunication – Value Added Services, IN1.38.8.1452402:3a80:c00b:13:face:b00c:0:a7MAAMadras
17488HATHWAY-NET-AP Hathway IP Over Cable Internet, IN202.88.160.81MAAMadras

South Asian neighbours of India

Kathmandu

ASNAS NameIPv4IPv6Airport CodeAirport Desc
23752NPTELECOM-NP-AS Nepal Telecommunications Corporation, Internet Services, NP120.89.99.17KTMKathmandu
38565NCELL-AS-NP Ncell Pvt. Ltd., NP116.68.215.17KTMKathmandu
17501WLINK-NEPAL-AS-AP WorldLink Communications Pvt Ltd, NP139.5.68.172400:1a00:4:139:face:b00c:0:a7KTMKathmandu
45845NEPAL-IIG-AS NEPAL INTERNATIONAL INTERNET GATEWAY, NP202.51.79.172405:6600:709:0:face:b00c:0:a7KTMKathmandu
4613MOS-NP Mercantile Office Systems, NP27.111.22.17KTMKathmandu
55915CLASSIC-NP Classic Tech Pvt. Ltd., NP202.94.66.145KTMKathmandu
45650VIANET-NP Vianet Communications Pvt. Ltd., NP110.44.120.812404:7c00:1:1:face:b00c:0:a7KTMKathmandu
17501WLINK-NEPAL-AS-AP WorldLink Communications Pvt Ltd, NP139.5.68.1452400:1a00:4:13a:face:b00c:0:a7KTMKathmandu
4007SUBISU-CABLENET-AS-AP Subisu Cablenet (Pvt) Ltd, Baluwatar, Kathmandu, Nepal, NP182.93.66.172403:3800::face:b00c:0:a7KTMKathmandu

Dhaka

ASNAS NameIPv4IPv6Airport CodeAirport Desc
24432AXIATA-ROBI-AS-AP TM International Bangladesh Ltd.Internet service Provider,Gulshan-1,Dhaka-1212, BD202.134.8.17DACDhaka
24389GRAMEENPHONE-AS-AP GrameenPhone Ltd., BD123.108.241.17DACDhaka
24389GRAMEENPHONE-AS-AP GrameenPhone Ltd., BD123.108.242.17DACDhaka
45245BANGLALINK-AS banglalink an Orascom Telecom Company, providing GSM Telecom service in Bangladesh, BD203.223.95.209DACDhaka
58715EARTHTELECOMMUNICATION-AS EARTH TELECOMMUNICATION (Pvt) LTD., BD45.113.132.17DACDhaka
23688LINK3-TECH-AS-BD-AP Link3 Technologies Ltd., BD203.76.97.1452400:ca00:1fb:fb01:face:b00c:0:a7DACDhaka
58689ICCNET-DHK-BD ICC Communication, BD103.41.213.145DACDhaka
134371CIRCLENETWORK-BD CIRCLE NETWORK BANGLADESH, BD202.136.90.2092400:3dc0:200:1:face:b00c:0:a7DACDhaka
63996MNL-AS-AP Mazeda Networks Limited, BD103.60.174.145DACDhaka
58601AAMRA-ATL-BD Aamra technologies limited, BD43.245.195.81DACDhaka
134204BUSINESSNETWORK-AS-AP Business Network, BD203.76.223.2092400:4d40:1:101:face:b00c:0:a7DACDhaka
45925TELETALK-BD ASN For Teletalk Bangladesh Ltd., BD103.230.105.145DACDhaka
59362KSNETWORK-AS-AP KS Network Limited, BD110.76.130.81DACDhaka
58682LEVEL3-BD Level3 Carrier Ltd., BD103.15.41.2092404:c900:8:0:face:b00c:0:a7DACDhaka
45766TRIANGLESERVICES Triangle Services Limited., BD103.40.227.81DACDhaka
133854X-PRESS-AS-AP X-press Technologies Limited., BD103.43.149.81DACDhaka
58889ZOL-BD Zx Online Ltd, BD103.19.254.145DACDhaka
134382ASN134382 Radisson Technologies, BD103.88.235.17DACDhaka
55492DFN-BD Dhaka Fiber Net Limited, BD45.127.244.209DACDhaka
45905STARGATE-AS-AP Stargate Communications Ltd., BD45.118.245.145DACDhaka

Colombo, Sri Lanka

ASNAS NameIPv4IPv6Airport CodeAirport Desc
9329SLTINT-AS-AP Sri Lanka Telecom Internet, LK222.165.163.1452402:d000:130:40:face:b00c:0:a7CMBColombo
9329SLTINT-AS-AP Sri Lanka Telecom Internet, LK222.165.163.2092402:d000:130:48:face:b00c:0:a7CMBColombo
18001DIALOG-AS Dialog Axiata PLC., LK125.214.168.172404:f000:0:e:face:b00c:0:a7CMBColombo
18001DIALOG-AS Dialog Axiata PLC., LK125.214.168.1452404:f000:0:f:face:b00c:0:a7CMBColombo
45224BELLNET-AS-AP Lanka Bell Limited, LK203.81.96.812406:c00:0:1:face:b00c:0:a7CMBColombo
132045AIRTEL-AS-ISP Bharti Airtel Lanka Pvt. Limited, LK223.224.4.172400:ff00:2:0:face:b00c:0:a7CMBColombo
17470ETISALATLK-AS Etisalat Lanka (Pvt) Ltd., LK43.252.12.145CMBColombo

Pakistan

ASNAS NameIPv4IPv6Airport CodeAirport Desc
9541CYBERNET-AP Cyber Internet Services (Pvt) Ltd., PK103.213.110.17KHIKarachi
132165CONNECT-AS-AP Connect Communications, PK111.119.161.209KHIKarachi
24499TPP-AS-PK Telenor Pakistan, PK43.224.238.81KHIKarachi
59257CMPAKLIMITED-AS-AP CMPak Limited, PK45.116.233.145KHIKarachi
23966LDN-AS-PK LINKdotNET Telecom Limited, PK119.30.107.81KHIKarachi
45595PKTELECOM-AS-PK Pakistan Telecom Company Limited, PK182.176.35.1452404:7000:b100:0:face:b00c:0:a7KHIKarachi
9387AUGERE-PK AUGERE-Pakistan, PK103.11.60.145KHIKarachi
58895EBONE1-PK Ebone Network (PVT.) Limited, PK150.129.7.209KHIKarachi
55714APNIC-FIBERLINK-PK Fiberlink Pvt.Ltd, PK103.17.203.145KHIKarachi
38193TWA-AS-AP Transworld Associates (Pvt.) Ltd., PK110.93.194.812404:d400:4000:20:face:b00c:0:a7KHIKarachi
45595PKTELECOM-AS-PK Pakistan Telecom Company Limited, PK182.176.35.2092404:7000:b110:0:face:b00c:0:a7KHIKarachi
45814FARIYA-PK Fariya Networks Pvt. Ltd., PK14.192.149.209KHIKarachi
38547WITRIBE-AS-AP WITRIBE PAKISTAN LIMITED, PK115.167.77.81KHIKarachi
24499TPP-AS-PK Telenor Pakistan, PK43.224.239.81LHELahore
59257CMPAKLIMITED-AS-AP CMPak Limited, PK45.116.235.145LHELahore
45595PKTELECOM-AS-PK Pakistan Telecom Company Limited, PK182.176.35.812404:7000:6100:0:face:b00c:0:a7LHELahore
23966LDN-AS-PK LINKdotNET Telecom Limited, PK119.30.107.17LHELahore
9541CYBERNET-AP Cyber Internet Services (Pvt) Ltd., PK124.29.210.81LHELahore
38264WATEEN-IMS-PK-AS-AP National WiMAX/IMS environment, PK58.27.171.172402:fd00::face:b00c:0:a7LHELahore
45595PKTELECOM-AS-PK Pakistan Telecom Company Limited, PK182.176.36.812404:7000:6000:0:face:b00c:0:a7LHELahore
38547WITRIBE-AS-AP WITRIBE PAKISTAN LIMITED, PK115.167.75.81LHELahore

Male, Maldives

ASNAS NameIPv4IPv6Airport CodeAirport Desc
55944OOREDOO-MV Ooredoo Maldives Plc, MV202.153.85.209MLEMale
7642DHIRAAGU-MV-AP Dhivehi Raajjeyge Gulhun (Dhiraagu), MV103.31.85.812406:e400:1:fb:face:b00c:0:a7MLEMale

What is amazing to see that is there are even nodes of Facebook in areas like Kabul

Kabul, Afghanistan

ASNAS NameIPv4IPv6Airport CodeAirport Desc
38742AWCC-AS-AP AWCC, AF61.5.206.2092400:e500:0:16:face:b00c:0:a7KBLKabul
45178ROSHAN-AF Main Street, House No. 13 Wazir Akbar Khan, AF103.28.134.81KBLKabul
55330GCN-DCN-AS AFGHANTELECOM GOVERNMENT COMMUNICATION NETWORK, AF149.54.4.81KBLKabul
55330GCN-DCN-AS AFGHANTELECOM GOVERNMENT COMMUNICATION NETWORK, AF149.54.4.17KBLKabul

Link to complete data – https://docs.google.com/spreadsheets/d/e/2PACX-1vQ18Ggi9x6QV-kLk6SrYEEQyA2U8gSHKTROfatNbhDISjfNZDEz-h2J8Qb10OIIQnEDvSrKS5Aj5XsP/pubhtml?gid=1674278445&single=true
***Update***
29 Nov 2019: Check the latest post here
Limitations of data

  1. As I often say – what is there is there. There can be more which is not there in data.
  2. Mapping is based on airport codes and it’s a common practice to use airport codes in DNS records. Actual node location may be within a 100-200km radius of the airport code.
01 Mar

Encrypted DNS using DNSCrypt

Writing this post from my hotel room in Kathmandu. I found that many of the servers appear to be DNS resolvers which is unusual.
E.g:

dig @anuragbhatia.com . ns +short
a.root-servers.net.
b.root-servers.net.
c.root-servers.net.
d.root-servers.net.
e.root-servers.net.
f.root-servers.net.
g.root-servers.net.
h.root-servers.net.
i.root-servers.net.
j.root-servers.net.
k.root-servers.net.
l.root-servers.net.
m.root-servers.net.
dig @google.com . ns +short
b.root-servers.net.
c.root-servers.net.
d.root-servers.net.
e.root-servers.net.
f.root-servers.net.
g.root-servers.net.
h.root-servers.net.
i.root-servers.net.
j.root-servers.net.
k.root-servers.net.
l.root-servers.net.
m.root-servers.net.
a.root-servers.net.

 
This seems unusual and is the result of basically port 53 DNS hijack. Let’s try to verify it using popular “whoami.akamai.net” query.

dig @8.8.8.8 whoami.akamai.net a +short
202.79.32.164
dig @9.9.9.9 whoami.akamai.net a +short
202.79.32.164
dig @1.2.3.4 whoami.akamai.net a +short
202.79.32.164

So clearly something in middle is hijacking DNS queries and no matter whichever DNS resolver I try to use, the queries actually hit authoritative DNS via 202.79.32.164. This belongs to WorldLink Communications (ISP here in Nepal) and I am just 5 hops away from it.
 
So what can be done about these cases? Well, one way is VPN of course but with a setup where VPN server’s IP address is hardcoded in the client and not using DNS. It works and does the task but performance can vary greatly depending on how far is the tunnel server. A better and more modern way out of it is by using encryption in DNS by using a protocol named “DNSCrypt“. DNSCrypt offers to encrypt of DNS queries from clients to the DNS resolvers. (Beyond that resolver still, follow usual non-encrypted root chain to reach authoritative DNS servers).
 
So how does it work?
There’s no integrated support of DNSCrypt in OS’es at this time. There are number of projects like dnscrypt-osxclient available on GitHub which enable this support.  Once configured, the client changes system’s DNS resolver to a local IP which listens for port 53 (regular/non-encrypted) requests.

cat /etc/resolv.conf |grep nameserver
nameserver 127.0.0.54

The client often offers support of various open resolvers like OpenDNS, Quad9 etc.

dig @127.0.0.54 whoami.akamai.net a +short
67.215.80.66

 
 
Here it shows that DNS resolver in my case happens to be Cisco’s OpenDNS. As soon as the client gets port 53 DNS queries, it encrypts it and sends via UDP port 443 (UDP or TCP depending on provider and client configuration). The encyption is based on trusted root CA’s and associated chain as popularly used in HTTPS. This is also one of reasons why DNSCrypt is also known as DNS over HTTPS.
 
Here’s an example of a DNS query to resolve A record of google.com while running tcpdumps in parallel:

sudo tcpdump -i lo0 'dst port 53' -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo0, link-type NULL (BSD loopback), capture size 262144 bytes
04:36:04.429212 IP 127.0.0.54.50966 > 127.0.0.54.53: 31576+ A? prd.col.aria.browser.skypedata.akadns.net. (59)
04:36:04.532015 IP 127.0.0.54.54914 > 127.0.0.54.53: 623+ [1au] A? google.com. (39)
^C
2 packets captured
4 packets received by filter
0 packets dropped by kernel

This shows request went in clear text to 127.0.0.54 which is configured on loopback. While in parallel if I watch for traffic towards OpenDNS public IPs, I get:

sudo tcpdump -i en0 'dst 208.67.220.220 or dst 208.67.222.222' -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on en0, link-type EN10MB (Ethernet), capture size 262144 bytes
04:39:56.827824 IP 192.168.0.4.53763 > 208.67.220.220.443: UDP, length 512
^C
1 packet captured
63 packets received by filter
0 packets dropped by kernel

Thus all that appears here is just an encrypted packet to Cisco OpenDNS over UDP port 443.
I ran another query and saved it in pcap file. Here’s how it looks like in wireshark:

 
 
 
That’s all about it for now. I am going to keep encryption enabled especially when travelling from now onwards. Time to get some sleep. 🙂
 
Useful Links:

  1. dnscrypt-osxclient – https://github.com/alterstep/dnscrypt-osxclient
  2. DNSCrypt Wikipedia – https://en.wikipedia.org/wiki/DNSCrypt
  3. DNS Over HTTPS (Google Public DNS) – https://developers.google.com/speed/public-dns/docs/dns-over-https
  4. DNS over TLS (Quad9) – https://quad9.net/faq/#Does_Quad9_support_DNS_over_TLS