Category Archives: ISP Column

08 Apr

Espresso: Google’s peering edge architecture

Back in 2017 Google shared details about Espresso which is their SDN solution for scaling up their routing.
Saw this fascinating presentation from Google at SIGCOMM 2017. This blog post covers it in detail besides the talk.

Key design principles for their routing platform

  1. Hierarchical control plane consisting of both global as well as local control. Global takes care of overall traffic flow, inputs coming from performance metric etc while local take care of failure of BGP sessions, port/device failure etc.
  2. Fail static – To ensure that any part of the system can fail and the system keeps working as it was before.
  3. Software Programmability

Key features of the Espresso platform

  1. Peers physically terminate on MPLS switch and BGP feature is in software and hosted on a set of a host (servers). Sessions are spread across different hosts to avoid a single point of failure. If a host fails, it will result in the failure of only a set of peering and not all. Plus, they keep backup hosts in event of failure of the primary.
  2. Single BGP runs on the software, the table goes in RAM of the server giving very high scalability to hold large routing tables.
  3. Google “sprays” small amounts of traffic across all available paths (non-best paths) to have a picture of all available paths and based on that data as well as inputs from applications, it selects the path.
  4. This platform proves that SDN is not only for the jailed gardens and can be used for BGP routing optimisation. Many people believed SDN was for “internal network” only.
  5. Back in 2017, this platform was being used for around 22% of their existing capacity and entire new buildout was using it. Now in 2020 probably number would be much higher.

The talk ended with a nice Q&A where someone asked how they know capacity on other paths because on an “unloaded path” they may see it’s all good but as soon as they send traffic it may actually choke that path. Clearly that is something which does not happen with Google peering that often and hence I must say their platform is very quick in determining and re-routing traffic.

While the presenter did not mention it in response to the question I think that due to distribution of BGP sessions across various host and carrying a large set of a table in such scalable way, they probably do not have BGP convergence issues. Also, since it’s outbound heavy, they can pick the path to send traffic. It will work in all cases where the other side is able to send traffic back to Google (TCP traffic) and their selected path is not dead.

Think about peer on the other side when you bring up your BGP session with AS15169 next time. 🙂

21 Feb

Indian RPKI ROA status

In Melbourne for the week for APRICOT 2020. Someone jokingly said it’s should be “APRICOT and RPKI 2020”. 🙂

It seems like both JPNIC and TWNIC are doing a good job at promoting their member operators in Japan & Taiwan for signing ROA. I thought to check for the status in India to find how India is doing.

RPKI ROA status for India










  1. Total prefixes: 40,834 (IPv4 + IPv6)
  2. Prefixes with valid ROA: 4693
  3. Prefixes with invalid ROA: 354
  4. Prefixes without ROA: 35,787

IRR route objects

  1. Prefixes with at least one valid IRR route object: 38,075
  2. Prefixes with invalid route object: 2213
  3. Prefixes with missing route object: 546

The method used to pull this data

  1. Download APNIC extended data: https://ftp.apnic.net/stats/apnic/delegated-apnic-extended-20200221
  2. Find IN ASNs which is APNIC assigned as well as IRINN delegated prefixes.
  3. Find all prefixes originated by these ASNs (assuming a large of them would be used in India only).
  4. Check for IRR and RPKI status for those prefixes.
13 Feb

Indian IPv6 deployment

I had calls with a couple of friends over this week and somehow discussion IPv6 deployment came up. “How much has been IPv6 deployment in India now in 2020” is a very interesting question. It’s often added with – “how much of my traffic will flow over IPv6 once it is enabled“?

Game of numbers

There is a drastic difference in IPv6 deployment depending on which statistic we are looking at here in India. There can be a bunch of factors based on which we can try to judge IPv6 deployment:

  1. How many operators are offering IPv6 to end users?
  2. How many end-users are on IPv6?
  3. How’s the content available on IPv6 in terms of a number of IPv6 enabled websites?
  4. How’s the content available on IPv6 in terms of traffic volume over IPv6?

First two and last two points are related and point towards from vertically opposite ends. (Call it good or bad) the fact of high centralisation. There has been an ongoing centralisation of mobile operators and in-country like ours they connect a very large number of end-users. The number of fixed-line networks has increased considerably but at the same time in proportion to a number of mobile users they user base growth has been much lower.

On the content side like everywhere else in the world, there’s a lot more centralisation of content. Many of my Indian ISP friends tell me that Google + Akamai + Microsoft + Netflix + Facebook + Cloudflare is way over 75% of their traffic. Think about it, that’s just 6 AS number out of 68000+ odd networks in the world as per BGP routing table. Thus by traffic profile, we are looking at 0.0014% networks serving 75% of content traffic. The reason for such centralisation is actually beyond network and more around the success of products of these organisations followed by factors like the winner (or top 3) gets it all in most of these domains.

For eyeball traffic APNIC IPv6 stats for India (source here) as well as Hurricane Electric’s IPv6 progress report (source here) give us some numbers:

  1. Very few fixed-line operators are offering IPv6 but on the mobile side – a large number of mobile networks are offering IPv6. Jio was on IPv6 since launch and as traffic increased, Airtel as well as Vodafone + IDEA also significantly increased their deployment. On fixed line, it’s just Jio + ACT broadband with any sizable IPv6 footprint. BSNL + Airtel have virtually no deployment. There might be some other network in the list but it’s off the radar.
  2. There are a lot more end users on IPv6 than despite the small number of networks offering it because of the large mobile user base. On Jio 80%+ user base, on Airtel, it’s 45%, on Vodafone & IDEA (merged company but still separate ASNs) it’s close to 50%. That’s the number of users who are connecting over IPv6 when given option of IPv4 and IPv6 as tested by APNIC. That number is huge!
  3. Around 98.5% of TLDs (top-level domain names like .com, .net etc) are IPv6 enabled. For .com & .net domains, only 7% of the domains have an AAAA record (compared to ones having an A record). Most of the numbers are much lower if we look at the content side of IPv6 (ignoring the traffic volume).
  4. If we look at the content players with IPv6 and include the traffic volume numbers then it’s way higher. It is estimated that globally somewhere between 20-30 top ASNs carry 90%+ traffic and almost all of those top 20 are IPv6 enabled.

What do all these numbers actually mean?

  1. If you are an eyeball network in India and you deploy IPv6, you can expect way over 70-80% traffic (by volume) on IPv6.
  2. If you are a content network/datacenter in India and application is targetting to home fixed-line / enterprise network, expect a rather low amount of IPv6 traffic but would be rapidly increasing as more fixed-line networks deploy.
  3. If you are a content network/datacenter in India and content hosted at your end attracts mobile traffic, you can expect way over 50%-60% of that mobile traffic over IPv6.

Some additional reasons to consider deploying IPv6

  1. In India, ISPs need to maintain carrier-grade NAT logs of the translations. If one is doing dual-stack, a large part of traffic will flow over IPv6 saving on those logging requirements.
  2. For ISPs, it will save you from significant strain on CGNAT device.
  3. For content network/webmaster/datacenter – IPv6 will help in delivering your traffic outside of (often) congested CGNAT paths.

Happy IPv6ing! 🙂