Bypassing CGNAT for GGC traffic

I am just back from SANOG 41 which happened in the last week of April in Mumbai. It was overall a good event and gave an opportunity to see many friends from the industry. In one of the discussions, my friend Aditya Kaul from Juniper Networks mentioned the fact that Google GGC can serve pre-CGNAT traffic and it’s a standardized documented implementation. It’s fascinating to see the extent to which caching servers have evolved. A quick search about the topic leads me to the official Google documentation page on Carrier-grade NAT.

Overview

The broad idea here is that more and more NAT on IPv4 is going to happen because massive shortage of IPv4 and high demand in terms of more users and devices going online. All this puts a massive load on CGNAT i.e Carrier Grade Network Address Translation which is used to put many users (on private IPs) behind a single public IP. NAT cannot scale up infinitely as there are hard limitations in terms of the number of sessions CGNAT devices can support, the number of virtual ports an IP has to initiate connections to the outside world.

Now, while Google supports IPv6 quite nicely, it remains a bit of a challenge to make IPv6 work in the access networks. Even when supported completely from the ISP side, the lack of CPE support and/or lack of configuration on CPE can lead to issues where IPv6 just does not work and IPv4 is ultimately used on the transport. Google uses its caching server platform called GGC (Google Global Cache) to deliver extremely high volumes of traffic with quite low latency. GGC primarily delivers content for YouTube besides other static but widely requested objects like the Android Play store etc. GGCs are deployed inside the ISP network and are usually placed close to the L3 termination points/routers. I documented the presence of GGCs and other CDNs in this old post here.

Usually, GGC would sit on a /28 or so of ISPs’ address pool and traffic is mapped to these GGC by making use of a mix of DNS and BGP. Essentially a passive BGP session is kept to “signal” to GGC the pools which ISP desires to be served from the node. Next, Google’s routing infrastructure ultimately maps end users by giving unique video source strings which map to the nearest GGC node.

Delivery of GGC traffic before NAT

While the traffic profile varies depending on each location, broadly for India almost half of retail traffic is Google and GGC delivers a significant part of it. Thus if GGC can be excluded from CGNAT, it would significantly reduce the load on CGNAT. Plus in India, CGNAT translation logging is a mandatory requirement from the regulator and the translation logs have to be kept for multiple years. This needs some tweaking in the configuration because under normal scenarios end users “speak to” Google’s infrastructure with that CGNAT WAN IP and the same is used for speaking to locally present GGC nodes. But in such a setup, CGNAT WAN IPv4 is used to speak to Google’s infra, but pre-NAT private addresses are used to speak to GGC. Additionally, Google’s documentation suggests excluding NAT for communication with their high-traffic IPs (only) in the GGC cluster instead of all.

The network diagram on Google’s documentation explains it quite well.

Google expects a BGP community tag of 15169:12000 for the CGNAT WAN IPv4 pool and another tag 15169:12100 for the private addresses used behind the CGNAT. If you are an ISP doing a fair amount of CGNAT and have Google GGC nodes deployed in the network - it would be a good idea to make use of these. While writing this post, I am sitting at my home office behind Siti broadband CGNAT. A quick look at tcpdump shows that I am being served from 103.199.225.47 (Siti GGC in Delhi).

anurag@desktop ~> curl --insecure https://103.199.225.47/debug_info
client_ip = '45.249.87.160'
    private_ip_ranges match: false
url_ip = '0.0.0.0'
destination_ip = '103.199.225.47'
date_time = '2024/05/05-11:14:10.462'
anurag@desktop ~> 

Seems like they are not bypassing CGNAT yet. 😀