Analysis of Google's IP routing backbone

Isp-Column

Another post on Google’s network and while putting it I realise this is my third post in a row. This is not part of any plan but just a coincidence. 😀

Google operates one of the largest IP backbones (AS15169) in the world and if one has to rate IP backbone based on traffic likely Google will be #1 surpassing any other ISP in the world. AS15169 backbone seems to have only one transit - Tata Comm AS6453 as visible in the routing table. This may or may not be true (more on this in the results below). As is often the case with BGP tables - whatever is there is there but there can be a lot more to it. Google being an outbound heavy network could in theory have more transits while selectively restricting BGP announcements to the network. Coming from India - the fact that Tata Comm is upstream of Google quite fascinates me. We know Tata Comm as the old VSNL which many of us used as our first ISP in life (VSNL auto dialer) or looking at gear in American datacenters labelled in Hindi as “विदेश संचार निगम / Vishesh Sanchar Nigam”. 😀

Due to massive peerings and local caches, likely traffic hitting transit would be quite low but I always wonder who they use Tata Comm for. Is that for a certain set of ASNs, countries, areas etc?

It’s not very easy to find the answer to this because Google’s backbone is not a transit network and they don’t dump their full table in any of the public route collectors or pass the full table to a downstream customer. There is limited information about it on the internet in terms of how many peerings Google has and how that compares to other backbones. At the Google Network India innovation summit in 2022, some of the data was opened up. Bikash Kolay, VP Engineering put this impressive chart comparing reachability study from this paper.



Analysing Google’s interconnection

I don’t have access to their RIB. But since Google sells VMs, I can get my hands on their forwarding table. Furthermore as I posted in past, they do offer both cold and hot potato routing. Thus as long as I am on hot potato routing, as a VM/cloudustomer I should similar set of routing as they have for their application traffic. Thus for fun, I put a Python script to trace to entire IPv4 routing table from a Google Cloud VM. My script traces to .1 of each prefix in the global routing table. This isn’t perfect but helps in finding some answers. I ran this on Google Cloud Iowa region and it took a few hours to finish. It ran mtr in a batch of 1000 at a time. Initially, I put just one VM and it took a few hours to process 3 lakh (300K) out of 8 lakh (800k) routes. To speed it up, I ended up breaking the remaining 500k into 5 x 100k IPs and distributing across 5 VMs. This gets me 870k + traces. This helps with some pointers on Google AS15169’s relation with the rest of the internet including AS6453, various exchanges etc. They do hide internal hops in the trace but the majority of external hops are visible, especially for internet exchanges as well as AS6453 and their other lesser-known transit. 😉


Google’s usage of Tata Comm AS6453

Following are the 2377 ASNs (list here) which Google reached via Tata Comm AS6453. results in around 11k routes via Tata Comm. This would include ASNs not peered to Google as well as ones which are peered but are not announcing all their routes.


Google’s other transit(s)

Tata Comm AS6453 is not the only upstream of Google. Lumen / Level3 AS3356 is also quite heavily used. And I see Google > Lumen > other transit-free networks which by definition means Google is downstream of Lumen. This isn’t visible if I look at BGP announcements.

bgp.he.net/AS15169#_graph4

There seems to be little over 22k routes which Google reaches via Lumen AS3356. Most of it is quite expected / normal considering Lumen/Level3 is a large backbone and has a massive number of downstream customers. But what makes it interesting is the fact that AS3356 seems to be in use for non-AS3356 downstream routes as well. This effectively makes Google downstream of Lumen.

E.g Google > Lumen > Cogent:

  1.|-- 108.170.243.166                                                 0.0%     3   12.6  12.3  11.9  12.6   0.4
  2.|-- lag-119.ear2.Chicago2.Level3.net (4.15.84.173)                 66.7%     3   13.4  13.4  13.4  13.4   0.0
  3.|-- ae2.2.edge2.Chicago10.level3.net (4.69.132.225)                 0.0%     3   17.7  80.9  14.4 210.5 112.3
  4.|-- be3018.ccr41.ord03.atlas.cogentco.com (154.54.12.81)            0.0%     3  211.9 153.1  29.3 217.9 107.2
  5.|-- be2766.ccr42.ord01.atlas.cogentco.com (154.54.46.177)           0.0%     3  260.8 178.7  28.9 260.8 129.9
  6.|-- be2832.ccr22.mci01.atlas.cogentco.com (154.54.44.169)           0.0%     3  225.7 159.2  38.7 225.7 104.6
  7.|-- be3036.ccr22.den01.atlas.cogentco.com (154.54.31.89)            0.0%     3   36.2  35.5  35.0  36.2   0.7
  8.|-- be2668.ccr22.elp02.atlas.cogentco.com (154.54.87.29)            0.0%     3  214.2 222.6 207.9 245.7  20.2
  9.|-- be3872.ccr32.phx01.atlas.cogentco.com (154.54.26.53)            0.0%     3  243.6 250.4 243.6 255.8   6.2
 10.|-- be2932.ccr42.lax01.atlas.cogentco.com (154.54.45.162)           0.0%     3  210.8 229.4 210.8 260.6  27.2
 11.|-- be2336.rcr22.b014796-1.lax01.atlas.cogentco.com (154.54.86.98)  0.0%     3  254.0 245.3 237.5 254.0   8.3
 12.|-- ???

Some routes also seem via Verizon AS701 / ALTER which includes Arelion / Twelve99 though the number of routes via AS701 is low (1538 prefixes).

  1.|-- ???                                                            100.0     3    0.0   0.0   0.0   0.0   0.0
  2.|-- ALTER.NET (152.179.43.157)                                     33.3%     3   13.8  75.6  13.8 137.4  87.4
  3.|-- lag-2.CHCGILDT-PPR01-CC.ALTER.NET (140.222.9.185)              66.7%     3   15.6  15.6  15.6  15.6   0.0
  4.|-- be3076.ccr41.ord03.atlas.cogentco.com (154.54.11.81)            0.0%     3   31.8  69.5  28.7 148.0  68.0
  5.|-- be2765.ccr41.ord01.atlas.cogentco.com (154.54.45.17)            0.0%     3  150.1  70.4  29.8 150.1  69.0
  6.|-- be2717.ccr21.cle04.atlas.cogentco.com (154.54.6.222)            0.0%     3   53.7  80.9  29.0 160.0  69.6
  7.|-- be2891.ccr41.dca01.atlas.cogentco.com (154.54.82.250)           0.0%     3   35.6  37.8  35.6  41.3   3.1
  8.|-- be3083.ccr41.iad02.atlas.cogentco.com (154.54.30.54)            0.0%     3  154.6 109.8  34.7 154.6  65.4
  9.|-- be3620.rcr51.b039343-0.iad02.atlas.cogentco.com (154.54.83.30)  0.0%     3   34.7  75.3  34.7 154.6  68.7
 10.|-- 38.88.170.2                                                     0.0%     3   34.9  34.1  33.7  34.9   0.6
 11.|-- ???

Google > Verizon > Arelion > Akamai

  1.|-- ???                                                             100.0     3    0.0   0.0   0.0   0.0   0.0
  2.|-- ALTER.NET (152.179.43.157)                                       0.0%     3  205.0  78.3  13.4 205.0 109.8
  3.|-- chi-b23-link.ip.twelve99.net (62.115.180.10)                     0.0%     3   13.6  17.7  11.9  27.5   8.6
  4.|-- akamai-ic-326536.ip.twelve99-cust.net (213.248.74.223)          66.7%     3  206.3 206.3 206.3 206.3   0.0
  5.|-- po110.bs-a.sech-ord.netarch.akamai.com (23.57.98.243)           33.3%     3  249.4 249.9 249.4 250.3   0.6
  6.|-- a72-52-1-169.deploy.static.akamaitechnologies.com (72.52.1.169) 66.7%     3  230.6 230.6 230.6 230.6   0.0
  7.|-- ae120.access-a.sech-ord.netarch.akamai.com (23.57.98.249)       66.7%     3   13.4  13.4  13.4  13.4   0.0
  8.|-- 93.191.172.187                                                   0.0%     3  206.1  76.5  11.7 206.1 112.2
  9.|-- ???

Google > Verizon > NTT > Edgecast

  1.|-- ???                                                     100.0     3    0.0   0.0   0.0   0.0   0.0
  2.|-- ALTER.NET (152.179.43.157)                              66.7%     3   13.8  13.8  13.8  13.8   0.0
  3.|-- ALTER.NET.customer.alter.net (152.179.105.22)            0.0%     3   14.5  13.8  11.6  15.3   2.0
  4.|-- ae-6.r22.chcgil09.us.bb.gin.ntt.net (129.250.4.89)       0.0%     3   15.4  13.0  11.7  15.4   2.0
  5.|-- ae-7.r25.sttlwa01.us.bb.gin.ntt.net (129.250.3.42)       0.0%     3  248.6 183.5  56.1 248.6 110.4
  6.|-- ae-6.r26.osakjp02.jp.bb.gin.ntt.net (129.250.3.61)      66.7%     3  245.8 245.8 245.8 245.8   0.0
  7.|-- ae-1.a02.osakjp02.jp.bb.gin.ntt.net (129.250.4.232)      0.0%     3  246.7 248.1 246.7 249.4   1.3
  8.|-- ae-0.edgio.osakjp02.jp.bb.gin.ntt.net (117.103.176.218)  0.0%     3  245.5 248.3 245.5 250.1   2.5
  9.|-- ae-65.core1.itm.edgecastcdn.net (192.16.24.131)          0.0%     3  255.1 249.8 246.1 255.1   4.8
 10.|-- 152.195.61.1

So clearly there are more transit(s) than just Tata Comm.


Before I post IX peering data it is important to remember that IX peering is one of the ways Google uses to connect. They do PNIs (Private Network Interconnect) outside of IXPs quite heavily and hence one can expect to usually ASNs by smaller volume of traffic. As soon as traffic goes beyond a threshold, Google simply moves over that traffic to PNI. (That is a standard practice in the industry). Thus low number of ASNs in any of the lists below means a low number of relatively low-traffic ASNs or ASNs who avoid PNIs. This data can be an interesting metric mostly for content players when they are analysing an IXP. In simpler words - this is how Google seems to be using an IX. Your mileage may/will vary.


Google’s reach via Indian exchanges

This includes networks which directly connect at the IX as well as downstream ASNs of the peers.

  • Google seems to be using NIXI Delhi to reach 7 ASNs, NIXI Mumbai for 42 ASNs and NIXI Chennai for 7 ASNs (list here)
  • 37 ASNs via Extreme IX Delhi, 109 via Extreme IX Mumbai and 30 via Extreme IX Chennai. (list here)
  • 5 ASNs via DE-CIX Delhi, 135 via DE-CIX Mumbai and 8 ASNs via DE-CIX Chennai (list here)
  • 4 ASNs via AMS-IX Mumbai (list here)

These numbers tell the other side story of IXP traffic in India. For inbound heavy eyeball ISPs - they can offload 70-80% of their traffic at these IXPs because content ultimately comes from a very small number of these massive networks like Google, Facebook, Akamai, Netflix etc but for content players’ traffic offload is much lower. I have heard numbers in the range of 12-15% traffic and sometimes even lower in terms of offload. For content players traffic goes to eyeballs and eyeballs are largely behind Airtel, Jio, VI etc on mobile and Airtel/Jio/ACT etc on fixed-line.


Google’s reach via large global exchanges

I have to find a better way to map PNIs out of 800k+ traceroutes where IPs are not well known on the path.

Found any issues with data / interested in any specific IXP or wonder why even your ASN is in the certain list here? Drop a comment or reach out via email and it would be an interesting chat!

Disclaimer: This is my personal blog and has no relation with my employer.