Facebook FNA updates (2024) and their IP backbone analysis

Anurag Bhatia
isp-column

Since Nov 2018 I have been posting yearly reports on Facebook’s Cache deployment (also known as FNA). Last year I missed posting an update on this. If you want to read out how I find these nodes, please check out first (Mar 2018 report which describes it).


Previous reports:

  1. March 2018 (here)
  2. Nov 2019 (here)
  3. April 2021 (here)
  4. July 2022 (here)

It’s interesting to see that these posts have been referred to in many of the research papers in Universities around the world.

Some of them:

I find it amusing and quite helpful that Facebook despite being aware of these reports has never gone out of its way to block them. I greatly appreciate that gesture on their part.


Facebook network - How it started…

Internet Routing Registry (IRR) records at times are super old. Facebook likely was using ARIN’s IRR before moving to RADB. A query to ARIN’s IRR shows how it all started 😀

anurag@desktop ~> whois -h rr.arin.net as32934
aut-num: AS32934
descr: Thefacebook.com
import: from AS25658 accept ANY
export: to AS25658 announce AS32934
admin-c: MZU-ARIN
tech-c: MZU-ARIN
mnt-by: MNT-THEFA-3
changed: zuck@thefacebook.com 20040824
source: ARIN
as-name: FACEBOOK

anurag@desktop ~>


FNA Report for 2024



  • Total unique strings show 8014 FNA nodes
  • Facebook FNA caches are in at least 989 cities. I can see 989 unique airport codes and since FNA nodes are often deployed in smaller cities tagged to the nearest airport of a large city, extremely likely these are spread in over 1000+ cities now.
  • FNA is spread across 1795 ASNs (IPv4) and 1748 (IPv6) networks. That way the majority of FNA nodes seem dual-stacked.
  • These caches are spread across 183 unique countries (list here)
  • Top 10 countries based on the number of FNA clusters - Brazil, India, Mexico, Indonesia, Bangladesh, Argentina, Vietnam, Thailand, USA and Pakistan
  • Top 10 ASNs with max number of FNA clusters in the world - Bharti Airtel India (AS9498), Jio India (AS55836), Claro NXT Telecomunicacoes Ltda Brazil (AS28573), BSNL India (AS9829), UNINET Brazil (AS8151), Telefonica Brazil (AS27699), TIM SA Brazil (AS26615), VNPT Corp (AS45899), V Tal Brazil (AS7738) and Mega Cable Mexico (AS13999).

India-specific data

  • FNA caches are spread across 79 Indian cities - List here
  • In terms of 125 unique networks/ASNs have Facebook FNA node - List here

Raw data in Google sheet - Click here



Facebook’s IP backbone

While doing this lookup I wanted to check my latency to the auth DNS since I directly query the auth DNS of Facebook to make queries faster. The traceroute was interesting from my server in Mumbai:

 1.|-- ctpl-demarc.bom.anuragbhatia.com 0.0% 3 5.3 2.6 0.8 5.3 2.3
 2.|-- ae14.pr03.bom1.tfbnw.net 0.0% 3 12.4 4.7 0.6 12.4 6.7
 3.|-- po207.asw02.bom2.tfbnw.net 0.0% 3 0.5 0.5 0.5 0.5 0.0
 4.|-- ae202.ar01.bom1.tfbnw.net 0.0% 3 1.4 1.0 0.4 1.4 0.5
 5.|-- ae101.bb01.bom1.tfbnw.net 0.0% 3 5.2 45.7 1.0 130.9 73.8
 6.|-- ae153.bb01.pnq1.tfbnw.net 0.0% 3 1.8 2.2 1.8 2.8 0.6
 7.|-- ae120.bb01.maa2.tfbnw.net 0.0% 3 23.9 23.0 22.0 23.9 1.0
 8.|-- ae101.ar01.maa2.tfbnw.net 0.0% 3 23.6 27.6 23.2 35.8 7.1
 9.|-- ae110.pr01.maa2.tfbnw.net 0.0% 3 19.6 19.3 18.9 19.6 0.4
 10.|-- po101.psw04.maa2.tfbnw.net 0.0% 3 18.6 18.7 18.6 18.7 0.1
 11.|-- po8.msw1ad.01.maa2.tfbnw.net 0.0% 3 18.8 18.8 18.7 18.8 0.0
 12.|-- c.ns.facebook.com 0.0% 3 18.8 18.8 18.8 18.8 0.0

Reading airport codes here: Mumbai (BOM) -> Pune (PNQ) -> Chennai (MAA)

This is fascinating as Facebook seems to be using airport codes + the fact they have a massive backbone around the world and carry traffic over it. Take e.g if I trace from AT&T AS7018 in Detroit to FB Pune, India - 129.134.37.10

Traceroute from 172.30.120.91 to 129.134.37.10 (129.134.37.10):
1 172.30.120.1 1.032ms 2.812ms 0.938ms 1.668ms 1.043ms 1.582ms 
2 192.168.1.254 25.995ms 1.944ms 1.859ms 
3 2(SERVFAIL (107.143.140.1) 4.278ms 4.29ms 4.274ms 
4 71.151.213.118 4.498ms 4.327ms 4.572ms 
5 * * * 
6 12.83.32.169 11.179ms 5.733ms 5.649ms 
7 12.240.223.133 11.933ms 12.783ms 11.862ms 
8 32.141.192.162 12.71ms 11.775ms 11.946ms 
9 po111.asw04.ord3.tfbnw.net (31.13.29.144) 12.475ms 12.618ms 12.558ms 
10 ae14.ar03.ord3.tfbnw.net (157.240.41.68) 36.562ms 13.312ms 12.091ms 
11 ae7.bb04.ord1.tfbnw.net (129.134.44.152) 47.136ms 11.883ms 16.829ms 
12 ae69.bb04.cle2.tfbnw.net (129.134.45.244) 43.415ms 19.105ms 19.368ms 
13 ae33.bb04.dca1.tfbnw.net (157.240.46.173) 52.256ms 28.122ms 28.887ms 
14 ae6.bb04.vab1.tfbnw.net (129.134.47.67) 58.261ms 33.808ms 39.403ms 
15 ae123.bb02.bio1.tfbnw.net (129.134.40.16) 137.492ms 114.483ms 113.766ms 
16 ae4.bb01.mrs1.tfbnw.net (157.240.44.155) 144.842ms 121.128ms 123.483ms 
17 ae89.bb01.bom1.tfbnw.net (31.13.27.183) 298.682ms 276.003ms 324.732ms 
18 ae154.bb01.pnq1.tfbnw.net (129.134.37.10) 256.464ms 259.672ms 275.302ms

Thus one can make use of this information to map their IP backbone.

So here’s what I did:

  1. Find out their backbone router IPs by looking at the rDNS PTR of their entire routed address space and map where their IP PoPs are located
  2. Find further subset of above which responds to ICMP.



Facebook IP backbone

  1. All connected - As far as I can find all PoPs seem connected and carry traffic over Facebook’s backbone.
  2. Spread around 200 unique cities as measured by airport codes (List here).
  3. While the list of unique airport codes was large, the IPs which respond to ICMP are much lower and this brings down the list to just 140 airport codes. (List here)
  4. In India Facebook has 7 IP PoP locations - Delhi, Mumbai, Chennai, Hyderabad, Pune, Tirupati and Kolkata. I have no idea why Tirupati is here. Not that many known data centres there except STPI. There has to be a good reason for it. There are a lot of visible IPs from Tirupati in Facebook’s backbone range and these are active too. There is also a visible FNA cache node in Tirupati on BSNL, ACT and Excellmedia.

Trace from my Mumbai server -> Facebook IP Node in Tirupati:

traceroute to 129.134.32.173 (129.134.32.173), 30 hops max, 60 byte packets
 1 ctpl-demarc.bom.anuragbhatia.com (45.64.190.61) 2.029 ms 1.979 ms 2.004 ms
 2 ae14.pr03.bom1.tfbnw.net (103.4.97.20) 2.598 ms 2.592 ms 2.605 ms
 3 po207.asw03.bom2.tfbnw.net (129.134.51.82) 0.425 ms 0.514 ms 0.534 ms
 4 ae203.ar02.bom1.tfbnw.net (129.134.33.204) 0.570 ms 0.563 ms 0.557 ms
 5 ae102.bb02.bom1.tfbnw.net (31.13.25.132) 1.109 ms 1.151 ms 1.187 ms
 6 ae154.bb02.pnq1.tfbnw.net (129.134.37.18) 2.220 ms 1.872 ms 1.847 ms
 7 ae120.bb02.maa2.tfbnw.net (129.134.37.44) 42.066 ms 41.998 ms 41.991 ms
 8 ae154.bb02.tir1.tfbnw.net (129.134.101.23) 40.736 ms 40.219 ms 40.161 ms
 9 ae102.ar03.tir3.tfbnw.net (157.240.123.31) 30.420 ms 30.393 ms 30.373 ms
10 po113.asw01.tir3.tfbnw.net (157.240.123.15) 26.874 ms 26.883 ms 26.909 ms
11 ae201.pr01.tir3.tfbnw.net (129.134.32.173) 28.590 ms 28.611 ms 28.673 ms

Mumbai -> Pune -> Chennai -> Tirupati

  1. Facebook’s Indian backbone connects to Marseille (France), Milan (Italy) on the Western side and Singapore on the Eastern side. Interestingly, traces to Marseille go via Mumbai, Milan via Pune and Singapore via Chennai. There seems no direct route to PoPs in the Middle East and they route via Europe (kind of similar to the IP backbones of ISPs as well).