Personal blog of Anurag Bhatia

Netflow for home router & Linux servers

Fri, 10 Apr 2026 01:50:00 +0530

For the last few weeks, I have been running a NetFlow collector for the home router. This is something I wanted to do for a long time, but I was missing the time to invest. There are a few open source options, and I guess many commercial solutions offering NetFlow, often bundled with other products.

My personal monitoring is 100% open source and presently running Prometheus + Thanos + Node Exporter + Blackbox Exporter + SNMP Exporter + Grafana + Grafana Loki + a few more exporters. So I started looking for open source options which are still under active development and supported.

Two of them are quite popular - Pmacct and Akvorado. Pmacct is extremely advanced, flexible, but at the same time overall complicated to set up (and maintain). Even their quickstart file 3100-line file is full of setup options. On the other hand, Akvorado seems simpler to maintain. Some complications, anyway, are expected with NetFlow because the goal is not just to collect the data but also to have a system to store it, analyse it, map IPs to location/AS numbers, etc., dashboards, etc. It’s a tool developed by French ISP Free, which is part of the Iliad group, which also owns Scaleway.

Setting up Akvorado is easy if you are familiar with Docker. They have a simple 4 command quick-start which deploys all their containers as part of the stack. They deploy a few containers as part of the overall stack.

The big picture page on their demo site documentation covers the overall architecture. I started feeding it data from the home (Mikrotik) router and later also added various Linux servers/VMs I manage for R&D, DNS, as well as to host this blog and other infrastructure. For Linux, I am using Pmacct as an exporter, as Akvorado’s documentation suggests using it in exporter mode and has a sample config which works.

Some data

Now that it’s been running for a few weeks, I can see major sources of data coming to the home. Before jumping to data, it’s important to note that I have regular camera feeds uploading traffic to my server in Mumbai, and some backups, route collector data pull and automated speedtests at home. All these kinds of “pollute” data, and thus, it is more fun to exclude and look for remaining data coming from regular usage by end devices at home. In the setup, I can easily filter data based on srcAS, dstAS, srcIP, dstIP, port, in and out interfaces, etc.

Here’s the data for the last 10 days towards end devices (excluding home server and camera traffic)

While most of ASNs here are expected, but just for the context, Esto AS135817 is one of the upstreams at home (friendly ISP that I reach over a GRE tunnel over an IX from an underlying ISP), and most of the traffic hitting it is actually CDN traffic for Google GGC, Facebook FNA, etc., sitting on their IPs.

What is more fun here are the Sankey graphs, where for a given interval, I can map ANY attribute -> ANY other attribute. e.g., srcAS -> Home Out interfaces or SrcPort -> SrcAS -> Destination IPs, etc.

Here’s the Sankey Graph for SrcPort -> SrcAS -> In Interface at home

On this connection, I get most of Google traffic from Google’s AS directly; however, for a few days, I was on Airtel as primary due to an outage on the underlying ISP and thus could not reach Esto over the tunnel. For those days, traffic patterns were very different. Both Google and Akamai terminated most of the traffic on caching nodes within Airtel.

Traffic profile when running home traffic via Airtel:

Akvorado has support for Grafana, and while their dashboard seems good for usual lookups, I miss a pie chart. Let’s see what the pie chart of the last 10 days’ traffic looks like:

Another fun thing is to look for traffic by Etype for IPv4 Vs IPv6 comparison. Let’s see how much IPv4 Vs IPv6 traffic there is in the last week:

The same logic can be used to have a pie chart, but it has to be in Grafana.

Thus for now 89% of traffic is IPv6 at home. On that note, time to end this post. 😀

Who can light fibre in India?

Sun, 22 Feb 2026 02:19:05 +0530

Friend and his two homes…

Recently, a friend of mine mentioned his plan to procure dark fibre to connect two of his homes. His idea was straightforward - get LCO to give a dark strand between both homes and run Bidi optics over it. I ended up telling him that while his idea is technically fine, as modern optics can easily cover that sort of distance, it would be illegal as per Indian telecom laws!

It wasn’t just him, but numerous content players, hosting companies, and Cloud players find it a bit odd when they find this out. In this post, I will document what is allowed and what is not allowed.

Advance warning: I am a technical person & do not know much about regulatory. Whatever I am sharing here is what I have learned from my friends who have expertise in the subject. Indian telecom rules are overly complex and can have a different set of interpretations, and often go into the grey area. I welcome readers to share feedback if they are aware of something which isn’t correct as per the regulation. With that warning out of the way, I will proceed with the post.

Understanding licensing

In India, when fibre is going outside of a given building, one needs to have a license to light it. In fact, not just “lighting up a dark fibre” but to deploy and resell dark fibre, one needs a license (IP-1). Infrastructure Provider Category 1 (or IP-1) is considered an easy license which can be used to sell passive assets such as dark fibre, ducts, towers, etc. Active infra i.e lighting up fibre, is not allowed under this license, and it’s expected that an IP-1 holder leases their dark fibre only to a license holder who is authorised to light the fibre. Within a building, campus, datacenter, home etc one can deploy a light fibre but as soon as it goes outside of that it becomes a bit of a grey area depending on the use case.

An Internet Service Provider (ISP) holding an ISP license or Unified License with ISP authorisation can light fibre but only for two specific purposes:

Offering internet drop to the end user (quite obvious)
Their own backhaul connectivity between routers, switches, OLTs, etc located anywhere within their authorisation area

While this looks quite straightforward, it specifically excludes point-to-point connectivity. An ISP cannot offer point-to-point connectivity under its license. So for my friend’s use case - he cannot buy a point-to-point link using any technology (MPLS, layer 2 VLAN, DWDM wavelength or dark fibre strands) from the ISP as the ISP under the ISP license is not authorised to sell those. But funny enough if the same ISP sells him “internet drop” under ISP license and two routers (outside of core ISP business) and “helps” in setting VPN over the public internet, it’s largely legal because from ISP and DoT’s point of view this is an internet product. About #2 - For a long time ISPs could not procure dark fibre and make use of it because they could not make full use of the capacity. They could not light it with say DWDM & resell wavelengths. However recently it was clarified that an ISP can sell capacity to other licensed ISPs only (not end non-licensed customers). When researching I found some friends not being aware of it as well as some calling it a grey area rule.

Access License & NLD

An Access license in India for selling circuits. It allows many things including selling p2p circuits, voice, mobile, etc. Many networks use an access license along with NLD (National Long Distance) which allows these services across India. It’s a vast license due to involvement of voice and mobile permissions but for the context of this post - it’s the ultimate license needed by players to do things without going into the grey area.

Debate on captive

I have heard of different opinions around captive traffic. An ISP friend of mine who has expertise in regulation (and doesn’t want to be named in this post) recently bid for a government contract. A project where the goal was to offer dark fibre to one department of the government. He had an ISP license while the government department that was taking fibre had no license. Their end goal was to connect to their own infra. This raised some concerns from other bidders. For now it’s unclear and I guess subject to interpretation on whether someone can light fibre for their captive use or not when the end product itself does not resell the capacity. Most large projects end up taking NLD and access licenses anyway because they intend to sell excess capacity. Thus networks like Railtel (by Indian Railway), Oil India, Powergrid Corp, etc all take NLD license to have legal ability to light fibre for their own use as well as to resell excess capacity. Industry-wide I hear that private players taking dark fibre and lighting it themselves is absolutely not allowed while for the government. bodies it remains a grey area.

Cloud and content players

Modern cloud and content players are major users of dark fibres, especially within metros. Hyperscalars like Amazon AWS, Google Cloud and Microsoft Azure deploy their infra in three datacenters in a given region & these have to be technically connected with dark fibre. There’s massive East-West traffic due to replication, high availability & many other reasons. In the majority of markets these companies can procure dark fibre between their datacenters and light it themselves but in India they cannot. In India, it’s a known case that they get external license-holding players to light fibre for them. In most of these cases hyperscalars work very closely with those players, they define network topology, hardware, configs etc but ultimately someone else executes it for them. Most of the Cloud players will never take a telecom license (ISP, access, NLD, ILD etc) in India simply because then they would be subject to many of the telecom-related rules and regulations. That brings business, security and many concerns. Take e.g the long over-discussed AGR case. If any of the Cloud players had a license, under older terms their compute, storage (and just anything) revenue would be subject to AGR / license fees. In a way, it’s good that Cloud players work well without a license as that keeps things simple. The fact that they cannot light fibre themselves is a drawback of our Indian licensing system and should be fixed. In most other markets including the US, UK, Germany, Singapore etc - lighting of fibre does not need a license.

Status of datacenters

Over time datacenters have taken an NLD license. List includes but is not limited to NTT Netmagic, Yotta, Ctrls etc. This is mostly because these players have multiple datacenters in a given area, and they want to connect and resell lit services. They can in theory do without NLD by using IP-1 and then reselling dark fibre strands only but that does not scale up, plus their customers who are buying would need to have a license to light the fibre.

Old and outdated ideas around regulation

Most of these ideas around licensing in the fixed-line market do not make sense anymore, simply because everything runs on fibre and fibre is not a scarce resource. ISP license somewhat makes sense (excluding the AGR part) because there is a requirement of URL filtering, CGNAT logging, etc., but outside of that, it’s largely an inflated set of rules. On the access side, the requirement of a license to light fibre simply promotes more rent-seeking behaviour instead of cleaner, competitive and optimised deployments. AGR on fixedline (and probably mobile telephony) also doesn’t make sense. In case of fixed line, the network runs on optical fibre which is laid by vendors who pay Rights of Way (RoW) to municipalities, State and central government (whichever road they use). There is taxation in the form of GST in the system on top of RoW. So the AGR on top doesn’t make sense as bits are not flowing over any scarce resource. It’s largely over privately built infrastructure, for which RoW and GST are already being paid.

With those thoughts I think my friend needs an access license to connect his homes and his LCO (or ISP who actually runs it) needs a IP-1 license to legally re-sell him dark fibre. 😀

Disclaimer: This is my personal blog, and hence, posts made here are in my personal capacity. These do not represent the views of my employer.

Update - 18 Mar 2026

I cannot count how many people by now have told me that new rules are just about to be notified and changes are inevitable. New rules will allow ISP license holders (or folks with UL license with ISP authorization) to be able to sell point to point services. That won’t fix the problem completely but greatly reduces it.

Another point to mention here is that in Nov 2022 DoT tweaked IP-1 license rules to allow them to give dark fibre to non-licence holders which are specified by DoT.

Airtel India - Europe routing issues

Thu, 29 Jan 2026 03:45:04 +0530

Since 20:08 IST / 14:38 GMT on 28 Jan 2026, the latency between Airtel India and Europe has gone up along with significant packet loss.

Here’s an example of shortlist 300+ endpoints from Airtel India with European endpoints:

Traceroutes

Airtel Haryana -> Contabo Europe

Start: 2026-01-29T03:52:33+0530
HOST: desktop                                                                  Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. AS???    _gateway (172.16.0.1)                                             0.0%    10    0.2   0.2   0.2   0.2   0.0
 2. AS???    10.240.9.204                                                      0.0%    10    6.2   7.7   4.6  17.7   3.7
 3. AS???    172.31.0.155                                                      0.0%    10   12.1   8.4   5.0  12.8   3.2
 4. AS9498   173.168.18.125.dhcp.anaronline.net (125.18.168.173)               0.0%    10    5.1   6.5   5.0   8.0   1.2
 5. AS???    182.79.243.34                                                     0.0%    10  266.6 262.7 259.1 276.3   5.3
 6. AS1299   lax-b22-link.ip.twelve99.net (62.115.162.62)                      0.0%    10  259.4 258.1 256.6 259.9   1.1
 7. AS1299   lax-bb2-link.ip.twelve99.net (62.115.140.156)                    10.0%    10  256.7 257.9 256.6 259.6   1.2
 8. AS1299   dls-b7-link.ip.twelve99.net (62.115.140.247)                     20.0%    10  323.4 324.2 322.6 326.2   1.2
 9. AS1299   atl-b24-link.ip.twelve99.net (62.115.140.32)                      0.0%    10  336.4 336.5 335.4 338.0   0.9
 10. AS1299   atl-bb2-link.ip.twelve99.net (62.115.143.236)                     0.0%    10  323.1 321.7 320.2 323.4   1.0
 11. AS1299   ash-bb2-link.ip.twelve99.net (62.115.137.132)                    40.0%    10  320.9 321.1 320.6 321.7   0.4
 12. AS1299   prs-bb2-link.ip.twelve99.net (62.115.140.106)                     0.0%    10  318.8 318.3 317.2 319.5   0.9
 13. AS1299   laut-b2-link.ip.twelve99.net (62.115.136.185)                     0.0%    10  325.7 326.0 324.3 327.7   1.3
 14. AS1299   contabohubeurope-ic-385701.ip.twelve99-cust.net (213.248.67.11)   0.0%    10  325.8 324.4 322.8 326.4   1.2
 15. AS???    ???                                                              100.0    10    0.0   0.0   0.0   0.0   0.0
 16. AS???    ???                                                              100.0    10    0.0   0.0   0.0   0.0   0.0
 17. AS51167  eu01.anuragbhatia.com (213.199.54.67)                             0.0%    10  323.1 323.5 322.4 324.9   0.9

Reading this trace: This is Ambala » Airtel MPLS » Los Angeles Arelion AS1299 handover > Arelion Dallas > Arelion Atlanta > Arelion Ashburn > Arelion Paris > Europe

Contabo Europe -> Airtel Haryana (CGNAT IP terminating in Saha (Ambala) Haryana):

Start: 2026-01-29T03:53:51+0530
HOST: eu01.anuragbhatia.com                                                Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. AS???    10.243.1.101                                                  0.0%    10    0.2   0.3   0.2   0.4   0.1
 2. AS???    10.0.66.2                                                     0.0%    10    0.3   0.3   0.2   0.3   0.0
 3. AS1299   laut-b2-link.ip.twelve99.net (213.248.70.176)                 0.0%    10    2.7   2.8   1.4   4.7   1.0
 4. AS1299   laut-b1-link.ip.twelve99.net (62.115.136.194)                 0.0%    10    3.6   2.7   1.9   3.6   0.6
 5. AS1299   ffm-bb1-link.ip.twelve99.net (62.115.136.146)                 0.0%    10    2.9   3.0   2.9   3.5   0.2
 6. AS1299   mei-b5-link.ip.twelve99.net (62.115.124.59)                   0.0%    10   18.4  18.3  18.2  18.7   0.1
 7. AS1299   sng-b7-link.ip.twelve99.net (62.115.134.229)                  0.0%    10  155.1 155.2 154.9 155.6   0.3
 8. AS1299   sng-b5-link.ip.twelve99.net (62.115.139.1)                   80.0%    10  162.5 162.5 162.5 162.6   0.1
 9. AS1299   bhartiairtel-ic-375944.ip.twelve99-cust.net (62.115.49.157)   0.0%    10  238.1 236.2 233.5 252.4   5.9
 10. AS9498   116.119.44.230                                                0.0%    10  320.7 321.0 320.5 323.2   0.8
 11. AS9498   170.168.18.125.dhcp.anaronline.net (125.18.168.170)           0.0%    10  327.1 328.9 319.4 355.2  13.6
 12. AS???    ???

This one is handover to Arelion AS1299 Europe locally > Arelion Frankfurt > Arelion Marseille » Arelion Singapore > Bharti Airtel AS9498 Singapore handover »> Airtel MPLS > Haryana

How to know that it’s not just me on both ends - my home connection in Haryana and my server in EU being impacted?

Let’s look for average latency to a few dozen Airtel endpoints in India from Contabo, Hetzner, hosting.de:

Impact from BGP routing table

Let’s compare Airtel AS9498 routes inside Arelion AS1299 table with BGP community tags: 1299:30000 (EU Customers) Vs 1299:37000 (APAC customers) based on routes visible from various downstreams of AS1299 across public route collectors (since Arelion’s own direct feed does not include the BGP community tag):

Within a few hours Arelion’s learning of Airtel AS9498 routes in Europe (1299:30000) went down by 7000 routes and APAC (1299:37000) went up by over 6000 routes.

What about other Indian telcos?

There seems some minor impact on Tata Comm but it is largely contained. Average latency from EU - Tata Comm Indian endpoint went up from 164ms to 180ms. In the case of Jio, it also shows the same behaviour as usual. However, on Jio, there’s a visible jump in latency every late evening for months, and this jump seems to follow the same pattern.

Trace from EU to Tata Comm AS4755 Delhi confirms largely normal routing.

Start: 2026-01-29T04:16:29+0530
HOST: eu01.anuragbhatia.com                                                Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. AS???    10.243.1.101                                                  0.0%    10    0.2   0.2   0.2   0.3   0.0
 2. AS???    10.0.66.2                                                     0.0%    10    0.2   0.2   0.2   0.3   0.0
 3. AS1299   laut-b1-link.ip.twelve99.net (62.115.45.148)                  0.0%    10    2.5   3.1   1.6   4.7   1.0
 4. AS1299   ffm-bb1-link.ip.twelve99.net (62.115.136.146)                 0.0%    10    3.2   3.1   2.9   3.4   0.2
 5. AS1299   ffm-b5-link.ip.twelve99.net (62.115.136.213)                  0.0%    10    3.5   3.2   3.0   3.5   0.1
 6. AS1299   tata-ic-378325.ip.twelve99-cust.net (213.248.78.41)          60.0%    10    4.3   4.3   3.9   4.7   0.3
 7. AS6453   if-bundle-56-2.qcore2.pvu-paris.as6453.net (80.231.245.40)   60.0%    10   16.1  16.0  15.6  16.4   0.4
 8. AS6453   if-bundle-12-2.qcore1.pvu-paris.as6453.net (80.231.245.12)    0.0%    10   15.4  15.8  15.4  16.8   0.4
 9. AS6453   if-bundle-22-3.qcore1.pye-paris.as6453.net (80.231.154.200)  50.0%    10   15.7  15.9  15.6  16.3   0.3
 10. AS6453   if-bundle-2-2.qcore2.pye-paris.as6453.net (80.231.154.27)     0.0%    10   15.5  15.9  15.4  16.4   0.4
 11. AS6453   if-bundle-13-2.qcore1.ldn-london.as6453.net (80.231.196.37)  90.0%    10   20.5  20.5  20.5  20.5   0.0
 12. AS6453   195.219.213.129                                              70.0%    10   15.4  15.5  15.4  15.7   0.2
 13. AS6453   80.231.131.77                                                 0.0%    10  123.0 125.2 123.0 144.3   6.7
 14. AS???    ???                                                          100.0    10    0.0   0.0   0.0   0.0   0.0
 15. AS4755   115.114.73.212.static-delhi.vsnl.net.in (115.114.73.212)      0.0%    10  158.5 158.6 158.5 158.7   0.1

With hope that your packets won’t be spread at the bottom of the ocean, it’s time for me to end this post and get back to work. 😀

Update - 29 Jan 2026 / 17:14 IST

A friend from the industry has confirmed that MENA cable has went down around the same time.

Update - 04 Mar 2026

Significant traffic routing between India and EU seems direct now.

Update - 13 Mar 2026

MENA Cable has been fixed and connectivity between India and EU is back online. Since evening there’s a visible further drop in average latency between India and EU over the Airtel.

When BGP lies, the internet believes!

Wed, 21 Jan 2026 00:02:42 +0530

Earlier in the day, I came across Liberty Global (AS6830) seemingly originating several Vodafone Romania (AS12302) prefixes.

Source: https://bgp.he.net/AS6830#_prefixes

This is highly unusual because AS6830 is a large transit-free network. I have seen some transit-free networks leaking routes, but originating a large number of prefixes is not common. It has massive stakes from Belgium-based Telenet to Virgin Media, etc. To verify if they are actually originating these or not, let’s check from their looking glass for one of the prefixes here: 46.97.104.0/24 via their PoP at Interxion FRA6 Frankfurt:

This clearly shows that their own router is learning it from Vodafone C&W AS1273 and not holding the fake route. The origin is AS12302 (Vodafone Romania). So why does that prefix appear in bgp.he.net?

Let’s look at real-time lookup from super-lg:

Reading this AS_PATH: 35505 44682 8751 6830

So RIPE RIS RRC22 in Bucharest, Romania “learns” this from AS35505 (Pronet Solutii IT SRL) which learns it from AS44682 (SIL-MIRO COM SRL) which learns it from AS8751 (MEDIA SAT SRL) which “claims” to have learnt it from AS6830. This very much smells like a “fake route” generated by someone here likely from a route optimiser. From the Liberty Global looking glass, it’s clear that AS6830 does not have it. So it has to be either AS8751 or AS44682 or AS35505 having this route in their table. It’s hard to verify and be 100% sure who since those three ASNs don’t have a looking glass or a RIPE Atlas probe for me to see their routing. Thus when BGP lies, the internet believes!

Disclaimer: This is my personal blog, and hence, posts made here are in my personal capacity. These do not represent the views of my employer.

Opentofu + Tailscale = Bootstrapped DevOps environment with VPN

Sun, 18 Jan 2026 02:58:46 +0530

For the last few days, I have been playing with OpenTofu. For those who may not know, it’s a fork of last Terraform as Terraform’s license was changed due to IBM’s acquisition of Hashicorp and is a Cloud Native Foundation project. It can be used to quickly deploy (and remove) resources from various cloud players.

Yesterday came across this tweet from Tailscale about tailscale’s module for deployment.

Cloud-init can be tough, and we've heard all about it. So we built an open-source Terraform module, one that helps provide a more consistent Tailscale experience across AWS, Azure, GCP, and everywhere. No more surprises, OS quirks, or other mysteries: https://t.co/vn1ITU7Fgz pic.twitter.com/stcF6WuiD7
— Tailscale (@Tailscale) January 15, 2026

It’s cool and useful, though I find Cloud init way more useful since for me these tools are more useful for quick testing rather than a permanent production server. The idea of tailscale on devops VMs as they come up is pretty powerful, as it takes care of the private network between these machines, plus underlay doesn’t matter, and hence one can use (cheaper) IPv6-only VMs.

An example of OpenTofu config (which is similar to Terraform) to deploy three IPv6 only servers on players like Hetzner across Nuremberg, Helsinki, and Ashburn:

main.tf

terraform {
  required_providers {
    hcloud = {
      source  = "hetznercloud/hcloud"
      version = "~> 1.59.0"
    }
  }
}

provider "hcloud" {
  token = var.hcloud_token
}


data "hcloud_ssh_key" "desktop" {
  name = "desktop"
}

data "hcloud_image" "ubuntu" {
  name   = "debian-13"
  most_recent = true
}

resource "hcloud_server" "vms" {
  for_each = var.vms

  name        = each.key
  image       = data.hcloud_image.ubuntu.id
  server_type = each.value.server_type
  location    = each.value.location
  ssh_keys    = [data.hcloud_ssh_key.desktop.id]

  user_data = <<-EOF
    #cloud-config
    package_upgrade: true
    packages:
      - curl
    runcmd:
      - curl -fsSL https://tailscale.com/install.sh | sh
      - tailscale up --authkey=${var.tailscale_authkey} --accept-routes
  EOF

  public_net {
    ipv4_enabled = false
    ipv6_enabled = true
  }

  labels = {
    managed-by = "opentofu"
  }

  lifecycle {
    ignore_changes = [user_data]
  }  
}

output "vm_ips" {
  value = { for k, vm in hcloud_server.vms : k => vm.ipv6_address }
}

variables.tf (to hold non-secret variables)

variable "hcloud_token" {
  description = "Hetzner Cloud API Token"
  type        = string
  sensitive   = true
}

variable "vms" {
  description = "Map of VMs with locations"
  type = map(object({
    location    = string
    server_type = optional(string, "cx23")
  }))
  default = {
    devops01 = { location = "nbg1" }
    devops02 = { location = "ash", server_type = "cpx11" }
    devops03 = { location = "hel1" }
  }
}

variable "tailscale_authkey" {
  description = "Tailscale Auth key"
  type        = string
  sensitive   = true
}

terraform.tfvars (to hold secret variables)

hcloud_token = "XXX"
tailscale_authkey = "tskey-auth-XXXXXX"

Triggering machines (with VPN built in)

Initiating (to install providers, initiate backend etc)

anurag@desktop ~/R/P/c/a/test (main)> tofu init

...

anurag@desktop ~/R/P/c/a/test (main)>

And the deployment!

anurag@desktop ~/R/P/c/a/test (main)> tofu apply
data.hcloud_image.ubuntu: Reading...
data.hcloud_ssh_key.desktop: Reading...
data.hcloud_ssh_key.desktop: Read complete after 0s [name=desktop]
data.hcloud_image.ubuntu: Read complete after 0s [name=debian-13]

OpenTofu used the selected providers to generate the following execution plan. Resource actions are indicated with
the following symbols:
  + create

OpenTofu will perform the following actions:

  # hcloud_server.vms["devops01"] will be created
  + resource "hcloud_server" "vms" {
      + allow_deprecated_images    = false
      + backup_window              = (known after apply)
      + backups                    = false
      + datacenter                 = (known after apply)
      + delete_protection          = false
      + firewall_ids               = (known after apply)
      + id                         = (known after apply)
      + ignore_remote_firewall_ids = false
      + image                      = "310554929"
...
...

Plan: 3 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + vm_ips = {
      + devops01 = (known after apply)
      + devops02 = (known after apply)
      + devops03 = (known after apply)
    }

Do you want to perform these actions?
  OpenTofu will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

hcloud_server.vms["devops02"]: Creating...
hcloud_server.vms["devops01"]: Creating...
hcloud_server.vms["devops03"]: Creating...
hcloud_server.vms["devops01"]: Still creating... [10s elapsed]
hcloud_server.vms["devops03"]: Still creating... [10s elapsed]
hcloud_server.vms["devops02"]: Still creating... [10s elapsed]
hcloud_server.vms["devops01"]: Creation complete after 19s [id=117722922]
hcloud_server.vms["devops02"]: Still creating... [20s elapsed]
hcloud_server.vms["devops03"]: Still creating... [20s elapsed]
hcloud_server.vms["devops02"]: Creation complete after 26s [id=117722920]
hcloud_server.vms["devops03"]: Still creating... [30s elapsed]
hcloud_server.vms["devops03"]: Creation complete after 35s [id=117722921]

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

Outputs:

vm_ips = {
  "devops01" = "2a01:4f8:1c1f:88df::1"
  "devops02" = "2a01:4ff:f0:7df7::1"
  "devops03" = "2a01:4f9:c013:f6e0::1"
}
anurag@desktop ~/R/P/c/a/test (main)>

In the Hetzner console for this project, three VMs appear:

Reaching devops01 -> devops02

root@devops01:~# tailscale ping devops02
pong from devops02 (100.68.146.42) via [2a01:4ff:f0:7df7::1]:41641 in 111ms
root@devops01:~# 

root@devops01:~# ping -c 5 devops02
PING devops02.tail362a2.ts.net (100.116.50.72) 56(84) bytes of data.
64 bytes from devops02.tail362a2.ts.net (100.116.50.72): icmp_seq=1 ttl=64 time=106 ms
64 bytes from devops02.tail362a2.ts.net (100.116.50.72): icmp_seq=2 ttl=64 time=106 ms
64 bytes from devops02.tail362a2.ts.net (100.116.50.72): icmp_seq=3 ttl=64 time=106 ms
64 bytes from devops02.tail362a2.ts.net (100.116.50.72): icmp_seq=4 ttl=64 time=106 ms
64 bytes from devops02.tail362a2.ts.net (100.116.50.72): icmp_seq=5 ttl=64 time=106 ms

--- devops02.tail362a2.ts.net ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4006ms
rtt min/avg/max/mdev = 105.735/106.012/106.243/0.197 ms
root@devops01:~#

It can reach devops02 over IPv6 transport.

NCMC design and technical limitations

Sun, 04 Jan 2026 00:16:00 +0530

I was in Mumbai a few months ago for the Equinix India Peering forum. The event happened to be very near the airport & I found that the airport, conference venue, hotel, and a few other locations I was visiting were all near the newly built Mumbai Line 3 (Aqua Line). To save on hassle with regular tickets, I went for an NCMC (National Common Mobility Card). Most people travelling in the Delhi metro would be aware of it by now, thanks to a bit of marketing push inside the metro train with announcements. Mumbai line 3 has a tie-up with SBI for the card, and thus, I got the SBI card. Ended up using it in Mumbai, Delhi and Chennai over the last few months.

Lately, I have been reading about way NCMC works, the limitations of the Indian NCMC design compared to other mass transit systems. In many countries, mass transit systems simply use MasterCard/Visa/Apple Pay tap and charge directly from the respective card (credit/debit card, etc.), like the London tube, Stockholm metro, HK airport express, Singapore, etc. Indian deployment is bit different.

Indian NCMC offline wallet concept

The whole idea of NCMC is to have a single card which can make payment for public transport, including metro, buses etc across the country digitally. It has to be fast and reliable, and this is where the whole tech differs from systems like UPI or Visa/MasterCard/Rupay POS payments. In case of UPI as well as credit/debit card payments on POS machines - entire transaction runs online. Money goes from the person paying the amount to the receiver as part of a settlement facilitated by NPCI. I don’t need to go in detailed steps involved there, but the key idea is that there’s an active involvement of 1) Payer’s APP 2) Payer’s bank 3) Receiver’s PSP 4) Receiver’s bank 5) NPCI UPI switch, etc. All these have to work and most important - internet has to work. 😀

All this largely works well for casual payments across the vendors, but not for transit payments, especially at metro stations. It easily takes 2-3 seconds in best case scenario for these online payments, which can result in a massive queue at the metro station gate, besides the risk of it not working at all if “the internet goes down” or the app freezes and whatnot. To deal with these issues, most of the mass transit systems work in hybrid mode, where the money is collected locally offline, and details of such transactions are uploaded in bulk transactions (which, if delayed, do not cause any major issue). A regular Delhi (or any other) metro card follows that. When money is added to the card, it is literally added to the card via an NFC operation with cryptography in place. One can add money to that card online but it won’t reflect unless a sync operation is performed at the metro station where online balance is literally written to the card. While there is of course UPI support where tickets can be bought online using UPI & processed by the gate via QR code but anyone who has used that would know it’s very slow. There has been some development at the UPI front with UPI Lite X supporting local on-device wallet with the option of offline transaction, but yet to see it in action. NCMC is much ahead compared to UPI Lite X for now.

NPCI seems to have taken these same concepts of offline wallet into a rupay card. So one can have a prepaid/debit/credit Rupay card from any (supported) bank and can request the creation of a “service area” in the card, and then an offline wallet balance can be stored in that service area. So while this ensures there remains a single card for transit across India (which I personally find very useful) but it’s not a simple “you can use your Rupay card to tap and pay” concept that exists in some countries, etc. To pay for transit, existing debit/credit card one must have support on the card itself, the bank, metro station (to create a service area). That’s quite a lot of moving parts and feels a bit fragile. Adding insulting to injury - NCMC is often markted as card which can pay outside of transit systems as well on the POS machines. Credit/Debit could anyway do that but this projection is often made for the prepaid cards & all this adds to chaos KYC & extra steps.

Direct use of Visa/MasterCard/Apple Pay

While the Indian system doesn’t support Visa/MasterCard/Apple Pay for transit payments and rightfully so as the early days of Russia-Ukriane conflict have taught us, with Visa/Mastercard suspending their operations in Russia, besides the high MDR rates. But concept-wise, these implementation are interesting. No special service account, no special permissions - just directly tap the card & go. And ofcourse they are not processing these payments in real time either (same latency/uptime/performance challenge). What they do instead is that their system records each tap, and at the end of the day they do a batch settlement with the card issuer. The fun part implementation comes in how they detect fraud. Imagine someone tapping a debit card with no balance in a bank account or a credit card with no credit limit - they would let such a card pass initially, but once the system detects it during the batch settlement, they will simply block it and upload the blocked card data in batch operation across all terminals. Thus frauds can happen, but is largely contained. Unsure why NPCI didn’t go for a similar design with Rupay cards.

NCMC is the default new card

Due to the “service account creation” process, use of Rupay debit/credit didn’t really picked up at metro stations, but prepaid NCMC with pre-created service area has become de-facto for the newly issued cards. Logically, that makes sense as that removes the metro operator from payment handling & puts an RBI authorized full fledged bank or payments bank to deal with the balance money. Delhi Metro went with Paytm initially and later moved to Airtel NCMC as RBI imposed restrictions on Paytm. Mumbai metro went for SBI, and many state transport systems did their own partership for issuing these cards.

Offline balance limitations

Since these NCMCs are holding an offline balance, there are some interesting and weird limitations in the way they work.

Similar to older non-NCMC metro cards, these can be recharged online but a “sync” operation has to be performed either by metro station staff or newly installed machines or with the limited support in the Android app (Apple’s NFC is still closed & yet not supported). This adds a bottleneck at busy stations.
If a card is lost, the bank either cannot refund the balance at all or can only do so after the card actually expires. This is because the card has an offline balance & there’s no way for the bank to claim & credit it back unless the card cryptographically expires.
For a credit/debit card, this is an entirely different account unrelated to their primary “online” balance. Money can be moved from an online balance to an offline balance, but again would require a sync operation by an NFC terminal.
The apps/websites connected to an NCMC gets updated with a delay of days as the payments are processed.
While the app/websites take time to update, card themselves hold recent transactions locally. I noticed this when I paid via NCMC in Chennai for a bus to the airport, that same transaction was visible at the DMRC terminal in Delhi metro, but not one the bank portal for the next few days.

This talk from Sandesh Kunder (NCPI) at Global Fintech Fest 23 covers the concept as well as ideas from NPCI’s point of view.

Fragile system

While there is a conceptual idea of full compatibility across NCMC - cards, terminals, etc., it’s not smooth. E.g initially I got “ECH_0007” error on DMRC terminals when doing a sync operation on the Mumbai metro-issued SBI card. It’s hard to know what went wrong since so many people were involved. Thus I could not move “online balance” which I paid via an UPI app to the card’s offline balance.

I think at some point, NPCI will ditch this overly complicated system and will bring NCMC version 2 with a direct charge option for the Rupay card. That, along with a mix of tech like UPI Lite X with NFC, will act as a reliable, fast option for transit payments. Remember there are over 600 million rupay debit cards alone. Making them directly compatible while keeping single “online” balance which can be settled in batch operations.

Understanding multiple routes from same ASN

Tue, 02 Dec 2025 00:24:58 +0530

A while back bgp.he.net added the feature of a live route propagation graph for a given prefix. Besides being near real-time, it is also specific to a prefix, e.g let’s look up for 2401:4900:87f0::/44 (Airtel mobility - 5G prefix):

https://bgp.he.net/net/2401:4900:87f0::/44#_graph

What sometimes confuses people is why there are multiple paths from a given ASN towards the originator ASN. Many people assume multiple paths exist for different prefixes only, and that is not true. Take e.g in the above route propagation graph AS3356 has a direct route to AS9498 as well as via AS2914, AS6939, AS6453, etc. And this kind of multiple paths is not limited just to AS3356 (which is run by three companies as of now - Lumen in North America, Colt in the EU and Cirion in South America). Notice GTT AS3257 is also learning it via Cogent AS174 as well as Tata Comm AS6453. Which is actually “best path” as determined by the BGP? Quick short answer: Both!

Why does this happen?

This kind of routing often happens when the other side is a large network with multiple routers doing eBGP with multiple other larger networks & not anywhere in the upstream path. Let’s take the GTT AS3257 example first before coming to AS3356 case (as AS3356 has something more to it as well, which I will cover after this one).

Let’s query 2401:4900:87f0::/44 on Hurricane Electric’s super-lg and restrict output with AS3257 in the path - output here

Path 2 shows GTT AS3257 learning it from Cogent AS174 but path 4 shows it’s learning it from Tata Comm AS6453. Both of these are “best paths” as determined by BGP in the respective routers that feed the routes. Path 2 via Cogent is being fed into route-views.chicago while path 4 is being fed into RIPE RIS RRC12 (DE-CIX, Frankfurt).

Let’s look at the GTT looking glass and trace to this route in the Chicago and Frankfurt router:

GTT Chicago > 2401:4900:87f0::1

IPv6 traceroute to 2401:4900:87f0::1
HOST: cr1-chi1-re1                Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 2001:668:0:2:ffff:0:d5c8:7ff  0.0%     5    0.5   1.2   0.5   1.9   0.6 <- GTT AS3257
 2. 2001:550:3::18d              40.0%     5    2.7   1.9   1.3   2.7   0.7 <- Cogent AS174
 3. 2001:550:0:1000::9a36:2eb1   40.0%     5    1.2   1.3   1.2   1.6   0.3
 4. ???                          100.0     5    0.0   0.0   0.0   0.0   0.0
 5. 2001:550:0:1000::9a36:5f6d   80.0%     5   31.9  31.9  31.9  31.9   0.0
 6. 2001:550:0:1000::9a36:5a15   80.0%     5   29.1  29.1  29.1  29.1   0.0
 7. 2001:550:0:1000::9a36:a3fa   60.0%     5   38.5  50.8  38.5  63.0  17.3
 8. 2001:550:0:1000::9a36:59a     0.0%     5  181.7 181.5 180.7 183.1   1.0
 9. 2001:550:0:1000::9a36:a8fd    0.0%     5  188.2 188.4 187.9 189.8   0.8
 10. 2001:550:0:1000::9a36:a902    0.0%     4  188.0 188.3 187.7 189.6   0.8
 11. 2001:550:0:1000::9a36:1996    0.0%     4  222.2 204.6 187.3 222.2  19.0
 12. 2001:550:0:1000::9a36:5e71   75.0%     4  182.2 182.2 182.2 182.2   0.0
 13. static-36-2-68-128.xxxxx.svi  0.0%     4  182.7 182.7 181.9 184.2   1.0
 14. 2404:a800::106                0.0%     4  280.8 270.9 266.9 280.8   6.6
 15. 2404:a800:1a00:800::72        0.0%     4  262.1 263.2 262.0 265.1   1.4
 16. ???                          100.0     4    0.0   0.0   0.0   0.0   0.0

GTT Frankfurt > 2401:4900:87f0::1

IPv6 traceroute to 2401:4900:87f0::1
HOST: cr10-fra2-re0               Loss%   Snt   Last   Avg  Best  Wrst StDev
1. 2001:668:0:2:ffff:0:8d88:6be  0.0%     5    1.3  20.2   1.3  89.5  38.8 <- GTT AS3257
2. 2001:668:0:3:ffff:0:9a0e:9aa  0.0%     5    1.1   1.1   1.0   1.2   0.1 <- GTT AS3257
3. 2a01:3e0:ff20::110           80.0%     5    7.3   7.3   7.3   7.3   0.0 <- Tata Comm AS6453
4. 2a01:3e0:ff20:110::3         80.0%     5    6.8   6.8   6.8   6.8   0.0
5. ???                          100.0     5    0.0   0.0   0.0   0.0   0.0
6. ???                          100.0     5    0.0   0.0   0.0   0.0   0.0
7. 2a01:3e0:3900::48             0.0%     5   15.6  15.7  15.5  16.0   0.3
8. 2a01:3e0:3900::17            80.0%     5  230.8 230.8 230.8 230.8   0.0
9. 2405:2000:d00:100::14         0.0%     5  231.1 231.1 230.9 231.3   0.1
10. 2001:5a0:2300:200::d2         0.0%     5  129.6 129.4 129.2 129.6   0.2
11. 2404:a800::106                0.0%     4  160.6 162.8 160.3 169.7   4.6
12. 2404:a800:1a00:800::72        0.0%     4  162.3 162.3 162.2 162.4   0.1
13. ???                          100.0     4    0.0   0.0   0.0   0.0   0.0

Essentially, what is happening here is that for the GTT Chicago route via Cogent AS174 became the best path, while for GTT’s router in Frankfurt, the route via Tata Comm AS6453 became the best path.

Revisiting the BGP route selection algorithm:

Weight - Cisco specific, higher weight wins
Local preference, higher localpref wins
Locally Originated wins (over externally originated)
AS_Path - Shorter AS_PATH wins
Origin code preference i>e>? - Not common these days
MED, low MED wins (used in multiple sessions across the same set of ASNs)
eBGP wins over iBGP (enables hot potato)
Lowest IGP metric to next-hop
Oldest route
Lowest router-id

So if weight is different, the higher weight wins; if weight is the same, localpref is matched etc. In this case #1, #2, #3, #4, #5, #6 and #7 are likely all the same as far as I can think. So, deal breaker becomes #8 which is the lowest IGP metric to the next-hop. If e.g for this specific router in Chicago, the IGP metric is lower to the router connected to Cogent vs. Tata Comm it will prefer Cogent and vice versa in Frankfurt. If the IGP metric is the same and if both Cogent & Tata Comm are on the same set of devices, then the next one #9, the oldest route will take over. It could be a case that the Chicago router first learnt this route from Cogent via the Frankfurt router learnt it from Tata Comm and hence the case.

What about multiple paths to AS3356?

AS3356 is a rather interesting case because a direct path exists in this case as visible in the route propagation chart. So based on BGP route selection #4, the shortest AS_PATH wins should take over if everything else is the same.

Let’s query 2401:4900:87f0::/44 again and filter all routes with 3356 in the path - output here:

Let’s compare a direct path vs an indirect path:

2401:4900:87f0::/44 - 38008 3356 9498 45609 fed via rrc25.ripe.net (RIPE RRC in Amsterdam)

This has BGP communities:

3356:4 - APAC
3356:666 - Peer route
3356:703 - Singapore
3356:2172 - SNG3 - Singapore3

So this is a “peer route” in Singapore.

2401:4900:87f0::/44 - 209 3356 174 9498 45609

This has the following communities:

3356:3 - North America
3356:575 - USA
3356:666 - Peer route
3356:2003 - LAX1 - Los Angeles

So both are “peer routes” - one learnt from a peer in Singapore (short AS_PATH) while the other learnt from a peer in the US (longer AS_PATH). The localpref has to be the same in these cases because if localpref is higher on any route, AS_PATH would not be compared at all and that path would become the best path. To find what is happening, let’s query the Los Angeles router of AS3356 via their looking glass.

Output is here. Basically, the Singapore route is not there at all. Hence it seems like a special setup where AS3356 peers with AS9498 in Singapore, keeps the route local for APAC and does not export them in US/EU PoPs, etc. The same is reflected by Asia Vs non-Asia downstreams of AS3356 when they feed routes to public collectors and hence multiple paths. Since there isn’t a peering in the US/EU with AS9498, they would pick one of the few upstreams of AS9498, depending on their lowest IGP metric cost (or the oldest route if IGP metric cost is the same).

BGP is fascinating!

Disclaimer: This is my personal blog, and hence, posts made here are in my personal capacity. These do not represent the views of my employer.

Mapping Starlink's global IP transit providers

Fri, 24 Oct 2025 00:48:35 +0530

A while back, I posted about Starlink’s Indian upstream. It’s interesting that now their prefixes are not visible behind routed via Telstra - AS4637 anymore, but it seems like they are testing Mumbai-based Microscan - AS55352.

I have always been curious about how they interconnect globally, particularly with which IP transit providers. To find out, I have to establish a relation between their GeoIP data and their BGP announcements globally. They have 2161 IP pools in the GeoIP sheet while originating an aggregate of 1109 prefixes from Starlink - AS14593.

Hurricane Electric’s Graph_v4 for AS14593 shows a rather complicated graph because they have different upstreams at different locations.

Here’s the logic I can follow to map:

Their GeoIP is either /24 or smaller than /24. I created a unique pair for the aggregate /24-city pair and did a lookup only for that. E.g for Luxembourg they have 4 x /27 (87.251.24.0/27, 87.251.24.32/27, 87.251.24.192/27 and 87.251.24.224/27) coming from same /24 (87.251.24.0/24). That way, I match this with the BGP table only once and not 4 times to get the same result.
Extract all routes from the global routing table dump I carry in my ClickHouse table for AS14593.
Get a subset of routes which are visible behind known transit-free tier 1 networks (to filter out announcements via peering)
Break BGP table prefix,as_path mappings to /24 ASNs - AS_PATH mappings to easily map the data.
Look for ASN on the left side of AS_PATH as visible from various transit-free tier 1 networks. E.g. a few lines of output for 87.251.24.0/24, I get:

"87.251.24.0/24","[28624,61568,2914,14593]"
"87.251.24.0/24","[262462,12956,1299,14593]"
"87.251.24.0/24","[20253,6762,2914,14593]"
"87.251.24.0/24","[54309,1299,14593]"
"87.251.24.0/24","[398465,3257,1299,14593]"
"87.251.24.0/24","[49544,2914,14593]"
"87.251.24.0/24","[13786,2914,14593]"
"87.251.24.0/24","[199524,3356,2914,14593]"

This means AS2914 and AS1299 are the upstreams here. This is just an example. Actual output has 1005 routes, and I cannot post that long list here.

Result

Since the number of transits is smaller than the number of locations, I mapped each location to a given transit provider. Here’s what it looks like:

ASN	AS Name	Upstream for following locations
AS10075	Fiber@Home Global Limited	Dhaka (Bangladesh)
AS12956	TELXIUS Cable	Buenos Aires (Argentina), Chicago (US), Dallas (US), Doha (Qatar), Guatemala City (Guatemala), Kuujjuaq (Canada), Lima (Peru), Managua (Nicaragua), Mexico City (Mexico), Montevideo (Uruguay), Panama City (Panama), Paris (France), Quito (Ecuador), San Jose (Costa Rica), San Salvador (El Salvador), Santiago (Chile), Sao Paulo (Brazil), Seattle (US), Stanley (Falkland Islands), Tegucigalpa (Honduras)
AS1299	Arelion/Telia Carrier	Aden (Yemen), Amsterdam (Netherlands), Anchorage (US), Andorra la Vella (Andorra), Ashburn (US), Astana (Kazakhstan), Athens (Greece), Atlanta (US), Baku (Azerbaijan), Bamako (Mali), Basse-Terre (Guadeloupe), Beirut (Lebanon), Belgrade (Serbia), Berlin (Germany), Bratislava (Slovakia), Brucejack (Canada), Brussels (Belgium), Bucharest (Romania), Budapest (Hungary), Calgary (Canada), Cape Canaveral (US), Charlotte Amalie (U.S. Virgin Islands), Chicago (US), Chisinau (Moldova), Colombo (Sri Lanka), Copenhagen (Denmark), Dallas (US), Denver (US), Dhaka (Bangladesh), Doha (Qatar), Dublin (Ireland), Dushanbe (Tajikistan), Dutch Harbor (US), Fort-de-France (Martinique), Guatemala City (Guatemala), Gustavia (Saint Barthélemy), Helsinki (Finland), Jerusalem (Israel), Kansas City (US), Khartoum (Sudan), Kingston (Jamaica), Kuala Lumpur (Malaysia), Kukes (Albania), Kuujjuaq (Canada), Kyiv (Ukraine), Lisbon (Portugal), Ljubljana (Slovenia), London (United Kingdom), Longyearben (Svalbard, Norway), Luxembourg (Luxembourg), Madrid (Spain), Malé (Maldives), Managua (Nicaragua), Manila (Philippines), Mariehamn (Åland Islands, Finland), Marigot (Saint Martin), Mexico City (Mexico), Miami (US), Minneapolis (US), N’Djamena (Chad), Nassau (Bahamas), Nicosia (Cyprus), Nome (US), Oslo (Norway), Panama City (Panama), Paris (France), Philipsburg (Sint Maarten), Phoenix (US), Podgorica (Montenegro), Port-au-Prince (Haiti), Prague (Czech Republic), Praia (Cape Verde), Regina (Canada), Reykjavik (Iceland), Riga (Latvia), Rome (Italy), Roseau (Dominica), Saint John’s (Antigua and Barbuda), Saint Peter Port (Guernsey), Salt Lake City (US), San Jose (Costa Rica), San Juan (Puerto Rico), San Salvador (El Salvador), Santo Domingo (Dominican Republic), Sarajevo (Bosnia and Herzegovina), Seattle (US), Singapore (Singapore), Skopje (North Macedonia), Sofia (Bulgaria), Stockholm (Sweden), Tallinn (Estonia), Tbilisi (Georgia), Tegucigalpa (Honduras), Tempe (US), Thimphu (Bhutan), Tirana (Albania), Toronto (Canada), Valletta (Malta), Vancouver (Canada), Vienna (Austria), Vilnius (Lithuania), Warsaw (Poland), Winnipeg (Canada), Yangon (Myanmar), Yerevan (Armenia), Zagreb (Croatia), Zurich (Switzerland)
AS137409	GSL Networks Pty LTD	Adamstown (Pitcairn Islands), Apia (Samoa), Auckland (New Zealand), Avarua District (Cook Islands), Brisbane (Australia), Chicago (US), Christchurch (New Zealand), Colombo (Sri Lanka), Dallas (US), Dhaka (Bangladesh), Dili (Timor-Leste), Doha (Qatar), Funafuti (Tuvalu), Hobart (Australia), Honiara (Solomon Islands), Honolulu (US), Kuala Lumpur (Malaysia), Malé (Maldives), Melbourne (Australia), Nuku’alofa (Tonga), Pago Pago (American Samoa), Paris (France), Perth (Australia), Port-Vila (Vanuatu), Seattle (US), Singapore (Singapore), Suva (Fiji), Sydney (Australia), Tarawa (Kiribati), Thimphu (Bhutan), Yangon (Myanmar), Yaren (Nauru)
AS13786	SEABORN, US	Asuncion (Paraguay), Buenos Aires (Argentina), Cayenne (French Guiana), Fortaleza (Brazil), Georgetown (Guyana), Paramaribo (Suriname), Santiago (Chile), Sao Paulo (Brazil)
AS21859	ZEN-ECN, US	Bandar Seri Begawan (Brunei), Hagatna (Guam), Kuala Lumpur (Malaysia), Manila (Philippines), Palikir (Federated States of Micronesia), Saipan (Northern Mariana Islands)
AS2516	KDDI	Chicago (US), Dallas (US), Doha (Qatar), Honolulu (US), Majuro (Marshall Islands), Paris (France), Seattle (US), Seoul (South Korea), Tokyo (Japan), Ulaanbaatar (Mongolia), Yangon (Myanmar)
AS267613	ELETRONET S.A., BR	Brasilia (Brazil), Kuala Lumpur (Malaysia)
AS2914	NTT	London (United Kingdom), Longyearben (Svalbard, Norway), Luxembourg (Luxembourg), Madrid (Spain), Mariehamn (Åland Islands, Finland), Montreal (Canada), New York (US), Oslo (Norway), Paris (France), Prague (Czech Republic), Praia (Cape Verde), Reykjavik (Iceland), Saint Peter Port (Guernsey), Saint-Pierre (Saint Pierre and Miquelon), Seattle (US), Stockholm (Sweden), Toronto (Canada), Vienna (Austria), Warsaw (Poland), Zurich (Switzerland)
AS30844	LIQUID-AS, GB	Aden (Yemen), Antananarivo (Madagascar), Chicago (US), Dallas (US), Doha (Qatar), Gaborone (Botswana), Gitega (Burundi), Harare (Zimbabwe), Johannesburg (South Africa), Juba (South Sudan), Khartoum (Sudan), Kigali (Rwanda), Kinshasa (Democratic Republic of the Congo), Lilongwe (Malawi), Lusaka (Zambia), Maputo (Mozambique), Mayotte (France), Mbabane (Eswatini), Mogadishu (Somalia), Nairobi (Kenya), Paris (France), Reunion (France), Seattle (US)
AS3257	GTT	Ashburn (US), Astana (Kazakhstan), Athens (Greece), Baku (Azerbaijan), Basse-Terre (Guadeloupe), Beirut (Lebanon), Belgrade (Serbia), Berlin (Germany), Bratislava (Slovakia), Bucharest (Romania), Budapest (Hungary), Charlotte Amalie (U.S. Virgin Islands), Chicago (US), Chisinau (Moldova), Dallas (US), Doha (Qatar), Dushanbe (Tajikistan), Fort-de-France (Martinique), Fredericton (Canada), Gustavia (Saint Barthélemy), Halifax (Canada), Hawthorne (US), Honolulu (US), Iqaluit (Canada), Kansas City (US), Khartoum (Sudan), Kingston (Jamaica), Kukes (Albania), Kuujjuaq (Canada), Kyiv (Ukraine), Ljubljana (Slovenia), Los Angeles (US), Manila (Philippines), Marigot (Saint Martin), Mexico City (Mexico), Miami (US), Milan (Italy), Minneapolis (US), Montreal (Canada), N’Djamena (Chad), Nassau (Bahamas), New York (US), Nicosia (Cyprus), Paris (France), Philipsburg (Sint Maarten), Podgorica (Montenegro), Port-au-Prince (Haiti), Portland (US), Prague (Czech Republic), Regina (Canada), Riga (Latvia), Rome (Italy), Roseau (Dominica), Saigon (Vietnam), Saint John’s (Antigua and Barbuda), Saint-Pierre (Saint Pierre and Miquelon), San Jose (Costa Rica), San Juan (Puerto Rico), Santo Domingo (Dominican Republic), Sarajevo (Bosnia and Herzegovina), Seattle (US), Skopje (North Macedonia), Sofia (Bulgaria), Tallinn (Estonia), Tbilisi (Georgia), Tirana (Albania), Toronto (Canada), Vaduz (Liechtenstein), Valletta (Malta), Vienna (Austria), Vilnius (Lithuania), Warsaw (Poland), Winnipeg (Canada), Yerevan (Armenia), Zagreb (Croatia), Zurich (Switzerland)
AS3356	Lumen/Colt/Level3	Anchorage (US), Asuncion (Paraguay), Atlanta (US), Bogota (Colombia), Brasilia (Brazil), Brasília (Brazil), Bridgetown (Barbados), Brucejack (Canada), Buenos Aires (Argentina), Calgary (Canada), Caracas (Venezuela), Castries (Saint Lucia), Chicago (US), Dallas (US), Denver (US), Doha (Qatar), Hawthorne (US), Honolulu (US), Kingstown (Saint Vincent and the Grenadines), Kralendijk (Bonaire), Kuala Lumpur (Malaysia), Lima (Peru), Los Angeles (US), Mexico City (Mexico), Montevideo (Uruguay), Nome (US), Panama City (Panama), Paris (France), Port of Spain (Trinidad and Tobago), Quito (Ecuador), Saint George’s (Grenada), San Jose (Costa Rica), Santiago (Chile), Sao Paulo (Brazil), Seattle (US), Stanley (Falkland Islands), Vancouver (Canada)
AS37282	MAINONE, NG	Abdijan (Côte d’Ivoire), Accra (Ghana), Bissau (Guinea-Bissau), Freetown (Sierra Leone), Lagos (Nigeria), Monrovia (Liberia), N’Djamena (Chad), Niamey (Niger), Porto-Novo (Benin), Yaoundé (Cameroon)
AS37662	WIOCC-AS, MU	Abdijan (Côte d’Ivoire), Accra (Ghana), Bissau (Guinea-Bissau), Dakar (Senegal), Freetown (Sierra Leone), Harare (Zimbabwe), Juba (South Sudan), Kinshasa (Democratic Republic of the Congo), Lagos (Nigeria), Maputo (Mozambique), Maseru (Lesotho), Monrovia (Liberia), N’Djamena (Chad), Niamey (Niger), Ouagadougou (Burkina Faso), Porto-Novo (Benin), Sao Tome (São Tomé and Príncipe), Yaoundé (Cameroon)
AS4826	Vocus Connect	Brisbane (Australia), Dili (Timor-Leste), Hobart (Australia), Honiara (Solomon Islands), Honolulu (US), Melbourne (Australia), Nuku’alofa (Tonga), Perth (Australia), Suva (Fiji), Sydney (Australia), Yaren (Nauru)
AS48728	VODAFONEQATAR, QA	No mapping to PoP but visible upstream for non-GeoIP tagged prefixes / might be up for testing
AS52468	UFINET PANAMA S.A., PA	Guatemala City (Guatemala), Kuujjuaq (Canada), Mexico City (Mexico), Panama City (Panama), San Jose (Costa Rica), San Salvador (El Salvador), Tegucigalpa (Honduras)
AS5405	INTERDOTLINK powered by Inter.link, DE	Berlin (Germany), Chicago (US), Dallas (US), Doha (Qatar), Ljubljana (Slovenia), Milan (Italy), Paris (France), Seattle (US), Vaduz (Liechtenstein), Vienna (Austria), Zagreb (Croatia), Zurich (Switzerland)
AS5416	Beyon	Manama (Bahrain)
AS55352	Microscan	No mapping to PoP but visible upstream for non-GeoIP tagged prefixes / might be up for testing
AS55836	Reliance Jio	Mumbai (India)
AS55850	Mercury NZ Limited	Adamstown (Pitcairn Islands), Apia (Samoa), Auckland (New Zealand), Avarua District (Cook Islands), Christchurch (New Zealand), Funafuti (Tuvalu), Honolulu (US), Pago Pago (American Samoa), Port-Vila (Vanuatu), Tarawa (Kiribati)
AS58717	Summit Communications	Dhaka (Bangladesh)
AS60849	ILEVANT-AS	Amman (Jordan)
AS6327	SHAW	Billings (US), Calgary (Canada), Regina (Canada), Vancouver (Canada)
AS63927	RISE-HK RISE, HK	Hagatna (Guam), Kuala Lumpur (Malaysia), Manila (Philippines), Palikir (Federated States of Micronesia), Saipan (Northern Mariana Islands)
AS6453	Tata Comm	No mapping to PoP but visible upstream for non-GeoIP tagged prefixes
AS6461	Zayo	Chicago (US), Dallas (US), Doha (Qatar), Fredericton (Canada), Halifax (Canada), Iqaluit (Canada), Kansas City (US), Kuujjuaq (Canada), Mexico City (Mexico), Montreal (Canada), Paris (France), Phoenix (US), Portland (US), Regina (Canada), Saint John’s (Canada), Salt Lake City (US), Seattle (US), Tempe (US), Toronto (Canada), Vancouver (Canada)
AS6762	TELECOM ITALIA SPARKLE	Asuncion (Paraguay), Buenos Aires (Argentina), Cayenne (French Guiana), Chicago (US), Dallas (US), Doha (Qatar), Fortaleza (Brazil), Georgetown (Guyana), Paramaribo (Suriname), Paris (France), Santiago (Chile), Sao Paulo (Brazil), Seattle (US)
AS7195	EDGEUNO	Bogota (Colombia), Bridgetown (Barbados), Caracas (Venezuela), Castries (Saint Lucia), Kingstown (Saint Vincent and the Grenadines), Kralendijk (Bonaire), Kuujjuaq (Canada), Mexico City (Mexico), Panama City (Panama), Port of Spain (Trinidad and Tobago), Quito (Ecuador), Saint George’s (Grenada), San Jose (Costa Rica), Sao Paulo (Brazil)
AS8529	Zain Omantel International	Muscat (Oman)
AS8551	Bezeqint Internet Backbone	Jerusalem (Israel), Ramallah (Palestine)
AS8781	Ooredoo	Chicago (US), Dallas (US), Doha (Qatar), Dubai (United Arab Emirates), Paris (France), Seattle (US)

The above data has been mapped from raw data which is posted here

Misc notes:

Some providers are visible in the list as transit players like Tata Comm (AS6453), Vodafone Qatar (AS48728) and Microscan (AS55352), but no PoP is mapped to them. This is either because they are just testing and these are indeed their upstreams, but no mapping found in GeoIP data, OR they are upstream for a large number of prefixes as well as downstreams, which are again not in the GeoIP mapping.
This table above is simply showing upstream for a PoP. In many cases, the drop is local, while in many cases, the drop is not local. E.g. Arelion (AS1299) seems to be upstream in Starlink Sri Lanka IPs, while actually that’s a drop for Arelion in Singapore.
This post is going live on 24 Oct 2025, while I worked on most of this data on 14 Oct 2025. Routing might have changed a little bit in the last 10 days.

Disclaimer: This is my personal blog, and hence, posts made here are in my personal capacity. These do not represent the views of my employer.

BSNL drops IP transit from BSCCL Bangladesh

Wed, 22 Oct 2025 01:23:54 +0530

Multiple news articles came earlier in the day in Bangladeshi media, suggesting that BSNL (AS9829) has dropped its 10G link from BSCCL (AS132602) at 00:00 on 21st Oct. This was in use for connectivity in North Eastern states of India.

Checking routing table for routes with AS_PATH “132602 9829” from a combined view of various route-collectors.

Date	BSNL prefixes behind BSSCL
2025-10-06	57
2025-10-07	57
2025-10-08	57
2025-10-09	57
2025-10-10	57
2025-10-11	57
2025-10-12	57
2025-10-13	57
2025-10-14	57
2025-10-15	57
2025-10-16	57
2025-10-17	57
2025-10-18	57
2025-10-19	57
2025-10-20	57
2025-10-21	0
2025-10-22	0

So indeed seems to have gone for now after 9 years of the agreement between India and Bangladesh.

Large scale optical switching at Google - AS15169

Mon, 20 Oct 2025 03:55:52 +0530

It’s the day of Diwali. Happy Diwali 🪔 to everyone reading this post. 😀

I am in holiday mode from couple of days and mostly reading (and binge watching). While reading paper on Apollo - Google’s optical circuit switching, I looked around and came across a fascinating Google’s talk from last year. This talk covers how they made their own optical switches.

Quick notes

(Advanced warning: These are from eyes of network engineer working mostly at IP/ethernet layer with poor idea about optical layer)

Architecture design

Google’s Jupiter datacenter architecture had typical Spine, aggregation and TOR switches. The idea here is to have big fat switches on top forming the “spine”. Next, aggregation switches, which are meshed with spine switches and these aggregation switches serve top of the rack (TOR) switches where servers connect. This helps to aggregate all servers in a given rack within the ToR switch(es) within that rack. Because of the mesh between aggregation & spine, the actual connections here are very high. Around 8 years ago, Google started replacing these Spine switches with optical switches. This essentially gives them a way to mesh all switches within the aggregation layer and control that optical connectivity from software. If you are watching the above embedded video from YouTube, those bits are coming from one of these optically switched systems.

Optical Switches

(Image source here)

Oftentimes in the IP world, to actually call an Ethernet switch with SFP optical ports an “optical switch”, but this is a different case. Imagine having a “hardware” where 100s of fibres land, say from Input 1, Input 2,… Input 100 and next 100s of fibre going out - Output 1, output 2…output 100. Next, this hardware can optically route a given input (say, e.g input 1) to any given output (e.g output 10). To do this, there are a few methods. Google’s one is based on MEMS mirrors. These are tiny (less than 1mm) mirror arrays which can steer a given optical input at a given angle to a pre-defined path. They do this within milli-second range which is crazy fast.

(Image source here)

East-West traffic is huge

Google started deploying this 7 years ago, and all their datacenters are using it. A considerable part of it comes from modern workloads as datacenters have a lot of East-West traffic (within DC) these days, compared to North-South (outside of DC). Google Cloud largely defines Google’s network decisions these days, and besides internal traffic for analytics, logs, backups, replications, there would be a major traffic between GPUs, compute, storage, etc. Thus, they need a mesh of aggregation switches to ensure traffic goes directly between them instead of another layer on top, which would get quite hot and become the congestion point. To deal with that, they need to connect big fat switches with lots of ports in a mesh. It’s quite interesting how, back in 2015-2016, when traffic was shooting up due to streaming, a lot of them ended up going to / aggregating on CDNs. Now, in the next bandwidth jump, it’s all local traffic to AI-based workloads and not even hitting internet but staying mostly within a DC.

Remembering Shashank Dubey AKA Delhi Gateway

Sat, 11 Oct 2025 20:23:53 +0530

On Wednesday, I received extremely sad news of the passing away of my good friend Shashank Dubey. We worked together in my previous job at Spectra (AS10029) and had both good and bad times as we navigated various challenges. Shashank was part of the NOC team when I joined the Network Planning & Ops team. We would often chat about networking whenever I worked out of Spectra’s Delhi office. This was my first job of working on JunOS, and I remember how I used to be afraid of changes and Shashank would chill out as we would go around making bulk changes. He used to work so much on one of the core routers in Delhi (Delhi gateway) that’s why we used to call him “Delhi Gateway”.

Later, I left for Hurricane Electric, and Shashank stayed for a few more years and left for Extreme IX. We used to have long calls in the evening once in a few months, and also tried meeting once in a while. He told me a few times that he wanted to start an ISP back at home but eventually gave up due to market conditions as Airtel/Jio were selling super cheap FTTH and it didn’t made any financial sense for him to enter the market at that stage. Earlier this year, after spending some time at Extreme IX and some consulting, he joined Spectra. Last month he wanted to talk, but I was busy and thought to had a call after returning to India from APNIC 60. I missed calling him back and lost the opportunity to ever speak to him. 😟

Will always be in my thoughts as a good friend. Om Shanti.

Meta using XGS-PON in the datacenter

Tue, 07 Oct 2025 02:10:25 +0530

Earlier this year, Mike Joseph from Meta gave a very interesting talk at NANOG 94 in Denver. The title of his talk was “PON in the Datacenter: Hyperscale for Management and Console” (slides here). Here’s a 41-minute video of this talk, and it is very much worth watching if you are in the datacenter / ISP space.

Notes from Meta’s talk:

XGS PON seems good enough to replace copper for management traffic. Again, this is for management traffic only and not the actual Meta content traffic, which likely would be over 10G/40G/100G links within the datacenter.
By using PON (Passive Optical Network), Meta has designed to terminate copper coming from devices within the rack so that outside of the rack is just (lighter) optical fibre (only). Copper, just for management alone at their scale, was quite a lot of cables. An end-of-row device mounted a “goalpost rack” on top. They had a rigid design to plan for Ethernet & serial drops in advance for each rack. Changes in that were harder due to the fixed design of copper cables.

(Image source for all pictures: Meta’s slides)

With PON, they were able to get (20+20 - ethernet & console drops) or any other combination like (32+8) per rack. On a row basis, that’s up to 1280 Ethernet drops + 800 console connections - where a pair of them in a rack would aggregate within the rack.
They didn’t say the vendor name, but from the pictures, it’s quite clear they are using Cienna’s solution. The picture below shows 6 x 4 port ONUs stacked, giving 24 copper ports. This can be expanded to 12 x 4 ports = 48 ports easily, as visible on the chassis.

Meta also has a high availability on the PON network by making use of 2 x OLTs feeding a 2:8 or 2:4 splitter.
They also have an interesting headend. Instead of using traditional OLT, they are using “micro-OLTs”. These are OLTs implemented on an SFP+ module and plug into a layer 3 switch. I think from the picture (plus the fact they are using Cienna ONUs), this is Cienna’s Tibit pluggables.

Image source: Cienna Tibit.
These devices basically speak 10G Ethernet at one end and XGS PON on the other end. In the backend, they do Ethernet to PON MAC bridge.

Thoughts on PON outside of the last mile

I found this presentation fascinating because I have always wondered why PON is not being deployed outside of the FTTH last mile. I have asked this to a few friends working on enterprise networks and felt as if the product line is not developed yet, while technically it’s very much feasible, as we know from the FTTH world. Go to any corporate office with 200-250 people sitting on each floor. These typically deploy Cat6 copper from each desk to a centralised switch. That’s a lot of copper, and the aggregation of that takes quite a bit of expensive real estate. A 4-port OLT with a 64-user per port can easily do 4 x 64 = 256 users. Likely much higher than that because optical power (unlike FTTH networks) won’t be an issue here. A mix of active Ethernet (for high traffic) + PON + DC power (to power ONUs) running over the same physical cable + pre-terminated fibre can do quite a bit. Corning displayed these concepts at the Courtyard hotel 6 years ago (video here).

Once, I was at a hotel in Singapore, and at night, it was crazy loud right outside of my room. One can easily tell it was a switch placed in a telco closet. All these can be quite nice applications for PON going forward. Would save a ton of expensive, heavy copper Cat6 cable, along with real estate, electricity and more when offering drops. An XGS 10G symmetric uplink/downlink capacity can easily do 150Mbps per drop for a 1:64 split or 78Mbps for a 1:128 split. Most of the hotel’s Wi-Fi is inbound heavy (streaming effect) while cameras are outbound heavy. Even at 10Mbps per stream, it can do many hundreds of cameras on the same PON port, where copper aggregates within a few meters on fibre and fibre aggregates per floor on the splitters.

I think going forward, we will see wifi APs as well as IP cameras, printers, and IP phones with PON ONU (or even PON based ATAs like this one!) tech built in, and it will essentially become a plug-and-play solution with pre-terminated hybrid copper-fibre, where copper will carry DC power and fibre will carry traffic.

Can we detect submarine cable cuts from BGP routing table?

Sat, 27 Sep 2025 02:41:11 +0530

Background

I wrote most of this post over a week ago, but I couldn’t complete it as I was down with a viral fever, which has disrupted my schedule a bit. Anyway, to the post now. A University researcher recently sent me an email about my post on Submarine cable cuts at Jeddah. Their question was about how we can detect submarine cable cuts in detail from the BGP routing table. There was some hint to “how” in the last part of my earlier post, where a graph shows how the route announcement from Bharti Airtel (AS9498) went to zero at DE-CIX Frankfurt. Let’s find out if we can actually detect a large outage using just the BGP as the control plane. Using the data plane is always useful as one can spray packets around & find out latency/packet loss, but one can ultimately have limited source points. But for the BGP table, there are great projects collecting full routing table & publishing it for researchers like RIPE RIS, Oregon Routeviews, besides quite nice looking glass API from DE-CIX, AMS-IX, etc.

Who is expected to pull announcements?

Typically, tier 1 / transit-free networks, as well as other networks which may technically be tier 1 but have large downstream table & peerings, would not pull announcements in case of submarine cable failure. This is by design; they are expected to ensure they have enough transport capacity available to carry the traffic. If they cannot, they would arrange for that, but won’t pull off BGP announcement because pulling announcements in one region (like Europe in this case) may result in their peering partner using more capacity. This is typically not expected as per peering policies.

Some examples of peering policy enforcing consistent announcements:

1. GTT AS3257 peering policy

2.7. Each Internet Network must announce consistent routes across all interconnection points, unless mutually agreed in writing by both parties.

2. Orange AS5511 peering policy

provide consistent routing announcements, i.e., the same set of routes announced with the same autonomous system (“AS”) path length at all peering locations,

3. Arelion AS1299 peering policy

Routes: Interconnection Candidate must carry full customer routes in interconnect routers, and announce consistent routes using BGP4 at all peering locations.

Thus, it’s clear that one should not look for announcement changes by any of the free networks from a viewpoint. A notable exception here will be tier 1 transit-free networks which peer at IXPs. :)

However, tier 1 transit-free networks can be an interesting viewpoint to see changes in announcements coming from their downstream.

DE-CIX FRA announcements

This shows a major downfall on 5th Sep. Here’s the raw table:

Timestamp	Routes
2025-09-05 19:30:44	294706
2025-09-05 20:30:45	294755
2025-09-05 21:31:02	294782
2025-09-05 22:31:12	294627
2025-09-05 23:30:44	282089
2025-09-06 00:30:43	282422
2025-09-06 01:31:05	282563
2025-09-06 02:31:07	282532

This shows a downfall of 12538 routes between 22:31 and 23:30. I am collecting this data from the Looking Glass API of DE-CIX, but I think the same should be visible (with more compute) from MRT dumps as RIPE RIS RRC12 carries routes fed from DE-CIX route servers on AS6695. There is a similar visible drop in routes at LINX London around the same period.

Here’s a summary of ASNs I can find from DE-CIX and LINX data which seem to be impacted:

IXP	IXP Code	ASN	AS Name	address	Drop in routes_imported at IXP
DE-CIX	rs1_dxb_ipv4	3356	Century Link	185.1.8.41	21
DE-CIX	rs1_dxb_ipv4	8220	COLT Telecom Group plc	185.1.8.60	6823
DE-CIX	rs1_dxb_ipv4	9498	Bharti Airtel Limited	185.1.8.51	3037
DE-CIX	rs1_dxb_ipv4	9583	Sify Technologies Ltd	185.1.8.57	165
DE-CIX	rs1_dxb_ipv6	8220	COLT Telecom Group plc	2001:7f8:73::201c:0:1	771
DE-CIX	rs1_fra_ipv4	9498	Bharti Airtel Limited	80.81.196.112	9681
DE-CIX	rs1_fra_ipv4	9498	Bharti Airtel Limited	80.81.194.250	9144
DE-CIX	rs1_fra_ipv4	18001	Dialog Axiata PLC	80.81.192.114	241
DE-CIX	rs1_fra_ipv4	35753	Integrated Telecom Company (ITC) / Salam.sa	80.81.192.26	281
DE-CIX	rs1_fra_ipv4	35753	Integrated Telecom Company (ITC) / Salam.sa	80.81.194.9	105
DE-CIX	rs1_fra_ipv4	58717	Summit Communications Limited	80.81.192.208	345
DE-CIX	rs1_fra_ipv6	9498	Bharti Airtel Limited	2001:7f8::251a:0:2	5656
DE-CIX	rs1_fra_ipv6	9498	Bharti Airtel Limited	2001:7f8::251a:0:1	3765
DE-CIX	rs1_fra_ipv6	35753	Integrated Telecom Company (ITC) / Salam.sa	2001:7f8::8ba9:0:3	64
DE-CIX	rs1_fra_ipv6	35753	Integrated Telecom Company (ITC) / Salam.sa	2001:7f8::8ba9:0:4	16
DE-CIX	rs1_fra_ipv6	58717	Summit Communications Limited	2001:7f8::e55d:0:1	86
DE-CIX	rs1_ist_ipv4	6939	Hurricane Electric	185.1.48.16	89077
DE-CIX	rs1_ist_ipv6	6939	Hurricane Electric	2001:7f8:3f::1b1b:0:1	60337
DE-CIX	rs1_mrs_ipv4	9498	Bharti Airtel Limited	185.1.47.7	9149
DE-CIX	rs1_mrs_ipv4	10075	Fiber Home Global Ltd	185.1.47.145	663
DE-CIX	rs1_mrs_ipv6	9498	Bharti Airtel Limited	2001:7f8:36::251a:0:1	3765
DE-CIX	rs1_mrs_ipv6	10075	Fiber Home Global Ltd	2001:7f8:36::275b:0:1	349
DE-CIX	rs1_mrs_ipv6	58717	Summit Communications Limited	2001:7f8:36::e55d:0:1	1193
DE-CIX	rs1_nyc_ipv4	9498	Bharti Airtel Limited	206.82.104.130	17062
DE-CIX	rs1_nyc_ipv6	9498	Bharti Airtel Limited	2001:504:36::251a:0:1	2495
DE-CIX	rs2_dxb_ipv4	3356	Century Link	185.1.8.41	21
DE-CIX	rs2_dxb_ipv4	8220	COLT Telecom Group plc	185.1.8.60	6823
DE-CIX	rs2_dxb_ipv4	9498	Bharti Airtel Limited	185.1.8.51	3037
DE-CIX	rs2_dxb_ipv6	8220	COLT Telecom Group plc	2001:7f8:73::201c:0:1	771
DE-CIX	rs2_fra_ipv4	9498	Bharti Airtel Limited	80.81.196.112	9681
DE-CIX	rs2_fra_ipv4	9498	Bharti Airtel Limited	80.81.194.250	9144
DE-CIX	rs2_fra_ipv4	18001	Dialog Axiata PLC	80.81.192.114	241
DE-CIX	rs2_fra_ipv4	35753	Integrated Telecom Company (ITC) / Salam.sa	80.81.192.26	281
DE-CIX	rs2_fra_ipv4	35753	Integrated Telecom Company (ITC) / Salam.sa	80.81.194.9	105
DE-CIX	rs2_fra_ipv4	58717	Summit Communications Limited	80.81.192.208	345
DE-CIX	rs2_fra_ipv6	9498	Bharti Airtel Limited	2001:7f8::251a:0:2	5656
DE-CIX	rs2_fra_ipv6	9498	Bharti Airtel Limited	2001:7f8::251a:0:1	3765
DE-CIX	rs2_fra_ipv6	35753	Integrated Telecom Company (ITC) / Salam.sa	2001:7f8::8ba9:0:3	64
DE-CIX	rs2_fra_ipv6	35753	Integrated Telecom Company (ITC) / Salam.sa	2001:7f8::8ba9:0:4	16
DE-CIX	rs2_fra_ipv6	58717	Summit Communications Limited	2001:7f8::e55d:0:1	86
DE-CIX	rs2_ist_ipv4	6939	Hurricane Electric	185.1.48.16	89066
DE-CIX	rs2_ist_ipv6	6939	Hurricane Electric	2001:7f8:3f::1b1b:0:1	60337
DE-CIX	rs2_mrs_ipv4	9498	Bharti Airtel Limited	185.1.47.7	9148
DE-CIX	rs2_mrs_ipv4	10075	Fiber Home Global Ltd	185.1.47.145	663
DE-CIX	rs2_mrs_ipv4	30990	DJIBOUTI TELECOM S.A.	185.1.47.25	709
DE-CIX	rs2_mrs_ipv6	9498	Bharti Airtel Limited	2001:7f8:36::251a:0:1	3765
DE-CIX	rs2_mrs_ipv6	10075	Fiber Home Global Ltd	2001:7f8:36::275b:0:1	349
DE-CIX	rs2_mrs_ipv6	58717	Summit Communications Limited	2001:7f8:36::e55d:0:1	1193
DE-CIX	rs2_nyc_ipv4	9498	Bharti Airtel Limited	206.82.104.130	17067
DE-CIX	rs2_nyc_ipv6	9498	Bharti Airtel Limited	2001:504:36::251a:0:1	2495
DE-CIX	rs2-bom-v4	3303	AS3303 - Swisscom (Switzerland) Ltd	103.27.170.87	2490
DE-CIX	rs2-bom-v6	3303	AS3303 - Swisscom (Switzerland) Ltd	2401:7500:fff6::87	452
LINX	rs1-in2-lon1-linx-net-v4	9498	Bharti Airtel Ltd	195.66.226.204	12071
LINX	rs1-in2-lon1-linx-net-v6	9498	Bharti Airtel Ltd	2001:7f8:4::251a:2	5905
LINX	rs2-in2-lon2-linx-net-v4	16637	MTN GlobalConnect Solutions LTD	195.66.236.18	1962
LINX	rs1-in2-lon1-linx-net-v4	9583	Sify Technologies Ltd	195.66.226.6	1632
LINX	rs1-in2-lon1-linx-net-v4	17494	Bangladesh Telecommunications Company Limited(BTCL)	195.66.226.78	157

Note:

The impact here varies from a few hours to weeks. The above table does not reflect that.
Routes going off from IXP aren’t necessarily a bad thing. Reduction in routes does not mean loss of connectivity because there will always be alternate (less direct) paths between the networks.

MRT dumps

Let’s look at updates from RIPE RIS RRC12 in Frankfurt. Just the size of the MRT update files gives the hint:

4.5M    updates.20250905.2200.gz
3.6M    updates.20250905.2205.gz
3.4M    updates.20250905.2210.gz
3.7M    updates.20250905.2215.gz
3.7M    updates.20250905.2220.gz
3.5M    updates.20250905.2225.gz
4.8M    updates.20250905.2230.gz
4.9M    updates.20250905.2235.gz
4.3M    updates.20250905.2240.gz
9.0M    updates.20250905.2245.gz
14M     updates.20250905.2250.gz
8.3M    updates.20250905.2255.gz
4.1M    updates.20250905.2300.gz
3.7M    updates.20250905.2305.gz
3.6M    updates.20250905.2310.gz
3.7M    updates.20250905.2315.gz
3.6M    updates.20250905.2320.gz
3.8M    updates.20250905.2325.gz
4.0M    updates.20250905.2330.gz
3.7M    updates.20250905.2335.gz
3.4M    updates.20250905.2340.gz
3.5M    updates.20250905.2345.gz
3.4M    updates.20250905.2350.gz
3.4M    updates.20250905.2355.gz

A detailed analysis of prefixes having withdrawal and re-announcement gives a clue on who is impacted.

What about transit-free tier 1 networks?

As stated earlier, we cannot use these methods on transit-free tier 1 networks. Their route announcement is mostly consistent unless they suffer a massive failure, taking down routers or entire transport in a given region. For detecting issues within them, I think only hints can be announced from their downstream side. E.g. looking at routes tagged with EU or APAC customer BGP community besides just spraying measurement traffic to destinations on those networks.

Submarine cable cuts at Jeddah impact connectivity in the Indian subcontinent

Sun, 07 Sep 2025 05:48:53 +0530

I was away from my system since Friday evening, and as I look at various alerts, latency, packet loss and routing changes, it seems like quite a lot has happened since Saturday early morning. Latency from India to Europe has increased across all known large telcos - Airtel, Tata Comm, Jio, etc.

One sample trace from my server in the EU on Contabo -> Airtel India:

HOST: server7.anuragbhatia.com                                                 Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. AS51167  ip-11-191-88-45.static.contabo.net (45.88.191.11)                 0.0%    10    0.3   0.3   0.3   0.5   0.1
 2. AS???    10.0.66.2                                                         0.0%    10    0.4   0.3   0.2   0.4   0.1
 3. AS3356   212.133.82.97                                                     0.0%    10    7.1  10.2   7.1  21.5   4.7
 4. AS???    ???                                                              100.0    10    0.0   0.0   0.0   0.0   0.0
 5. AS174    port-channel2813.ccr92.dca04.atlas.cogentco.com (154.54.82.2)     0.0%    10   90.0  90.0  89.9  90.1   0.1
 6. AS174    be3483.ccr42.atl01.atlas.cogentco.com (154.54.172.170)            0.0%    10  106.9 107.0 106.7 107.3   0.1
 7. AS174    port-channel3704.ccr92.jan02.atlas.cogentco.com (154.54.40.110)   0.0%    10  113.7 123.0 113.6 206.3  29.3
 8. AS174    be8122.ccr32.dfw01.atlas.cogentco.com (154.54.41.53)              0.0%    10  121.6 121.6 121.4 122.1   0.2
 9. AS174    be3846.ccr22.elp02.atlas.cogentco.com (154.54.165.30)             0.0%    10  134.8 134.6 134.4 134.8   0.1
 10. AS174    be5473.ccr32.phx01.atlas.cogentco.com (154.54.166.70)             0.0%    10  141.6 142.9 141.5 153.2   3.6
 11. AS174    be2932.ccr42.lax01.atlas.cogentco.com (154.54.45.162)             0.0%    10  152.1 152.0 151.8 152.2   0.1
 12. AS174    be3360.ccr41.lax04.atlas.cogentco.com (154.54.25.150)             0.0%    10  147.3 149.4 147.1 168.6   6.7
 13. AS174    38.122.147.122                                                    0.0%    10  146.6 147.5 146.5 150.5   1.3
 14. AS9498   116.119.112.26                                                    0.0%    10  320.9 321.0 320.6 323.2   0.8
 15. AS9498   152.52.190.90                                                     0.0%    10  335.3 335.3 335.2 335.4   0.1

Lumen EU > Cogent EU (likely on hop 4 - remember hot potato routing) > Cogent Washington > Cogent Los Angeles > Bharti Airtel handover in Los Angeles -> India (via Pacific)

Here’s an average latency graph of 3 EU sources -> 9 Airtel endpoints in India for the last 48 hours:

As per this issue, it started at 04:00 IST (GMT+5.5) on 06 Sept 2025. Jio as well as Tata Comm endpoints show similar latency. The same also matches the route drop across

Here’s a combined graph of Airtel, Jio, ACT broadband and Excitel:

What caused it?

The time of this latency jump seems to be related to news about IMEWE and SMW4 fibre cut near Jeddah, Saudi Arabia. While there’s not much reported in India, PTCL Pakistan issued a statement on x about these outages. The cable cut point impacts Indian traffic as well. IMEWE lands in Mumbai, and both Airtel & Tata Comm are the consortium partners. SMW4 is an older cable and lands in both Mumbai & Chennai as it crosses over from the West to the East. As far as I can find, there’s no official report by Indian telcos about this, but time as well as latency/packet loss very much show that it is related.

Another metric outside of latency/packet loss to view is the route announcement. Here’s the charge of AS9498 route announcement at DECIX Frankfurt:

Impact on traffic?

While only respective networks can tell how much of their capacity depends on it. Do keep in mind that over 90% of traffic for Indian retail-facing operators is local. That’s content to eyeball traffic. So whether there’s an impact or not would depend more on whether Google/Facebook/Microsoft/AWS store a lot of Indian accessed content in the EU or not. If not, it would be a small % of overall traffic, even though it may impact a higher percentage of international traffic. For anyone in businesses of connecting Indian endpoints with the EU (CRM, VoIP), etc, it may be 100% of their traffic. So it would depend on the kind of traffic profile.

Disclaimer: This is my personal blog, and hence, posts made here are in my personal capacity. These do not represent the views of my employer.

Update: 15 Sep 2025

Seems like Bharti Airtel AS9498 capacity is getting restored. This is visible in reduced latency from EU to Airtel endpoints in India as well as their route announcements at DE-CIX FRA.

Analysing ccTLD anycast

Sat, 09 Aug 2025 01:43:56 +0530

In June, I attended NPNOG in Nepal. In the evening, I had a discussion with some friends about infra hosting ccTLD. We compared how some countries in the region host their ccTLD on their own while some outsource it to larger ccTLD operators. I have a feeling that it’s pretty hard to host a ccTLD these days, not because of the tech stack, but because of DDoS concerns. The only way to take care of DDoS at large-scale is to a) Over provision on the bandwidth and b) Spread the attack surface using anycast

So to host a ccTLD reliably in 2025, one would need at least eight‑ten global locations near major data sources to take care of the attack. Something like Ashburn, Dallas, Palo Alto, Frankfurt, Amsterdam, Singapore, Hong Kong, etc. While we had the discussion - I wanted to check how many ccTLDs are actually anycasted and if my guess was correct. Seems like I was partially correct.

Understanding anycast

Anycast basically means having more than one node and using the same address. It is assumed/expected that BGP will take us to the nearest announcement. But in reality, BGP path selection is influenced by neighbor relationships. Slide 26-33 of my presentation at INNOG covers this in detail. Essentially, network operators have the highest localpref on routes they learn from customers, the second highest on routes from peers, and the lowest on routes from upstream. Thus, if one does anycast with a mix and match of upstreams, it won’t work well. To deal with this, anycast is usually done with the same set of global upstream ISPs everywhere or with a very aggressive use of BGP action communities or a mix of global + local nodes. But even with all this, there can still be cases where some networks hit a local node, some hit far off anycast node.

Thus, to deal with this specific part, I have to test from multiple providers in a given location. Anycast cannot be easily tested by just looking at the routing table. In theory, with location communities, one can get an idea, but it’s much easier to find it from latency/trace from different locations.

So I ended up putting a script to look up for IPv4 and IPv6 latency to all ccTLD nameservers from the following providers/locations:

Contabo - New York and Düsseldorf
Hetzner - Falkenstein, Ashburn, and Singapore
Vultr - Amsterdam, Los Angeles, and Singapore

These nodes are in key cities where peering/interconnection happens. Latency from these to ccTLD auth ns can determine whether they are anycasted or not. Finding the exact number of anycast nodes is hard and need a lot more distributed measurements but one can safely assume that if an IP is anycasted across the US, EU, and Asia, one should be able to reach it in less than 80ms from these locations (more like within 40-50ms actually). If I get low latency from one provider and high from other, I consider that it proves that an anycast node exists and it’s just bad routing (which I am not measuring here).

Result of ccTLD anycast check

The table below reflects the results. I have categorized into three simple categories:

All NS anycasted - All nameservers for the given ccTLD are anycasted and reachable with low latency.
At least one NS doing anycast - This is a case where I see some nameservers giving consistently low latency from the US, EU & Singapore, but some with high. In most cases, it’s basically non-anycasted own servers and anycasted servers of commercial/non-profit providers.
None of the NS is doing anycast - All nameservers have above 80ms latency when checked from the US, EU, and Singapore (at least two of these regions, usually lower latency where closer to the ccTLD origin country)

ccTLD	Category
aw	All NS anycasted
bh	All NS anycasted
fm	All NS anycasted
fo	All NS anycasted
gd	All NS anycasted
in	All NS anycasted
la	All NS anycasted
nl	All NS anycasted
pw	All NS anycasted
to	All NS anycasted
vg	All NS anycasted
ac	Atleast one NS doing anycast
ad	Atleast one NS doing anycast
ae	Atleast one NS doing anycast
af	Atleast one NS doing anycast
ag	Atleast one NS doing anycast
ai	Atleast one NS doing anycast
al	Atleast one NS doing anycast
am	Atleast one NS doing anycast
ao	Atleast one NS doing anycast
aq	Atleast one NS doing anycast
ar	Atleast one NS doing anycast
as	Atleast one NS doing anycast
at	Atleast one NS doing anycast
au	Atleast one NS doing anycast
ax	Atleast one NS doing anycast
az	Atleast one NS doing anycast
ba	Atleast one NS doing anycast
bb	Atleast one NS doing anycast
bd	Atleast one NS doing anycast
be	Atleast one NS doing anycast
bf	Atleast one NS doing anycast
bg	Atleast one NS doing anycast
bi	Atleast one NS doing anycast
bj	Atleast one NS doing anycast
bl	Atleast one NS doing anycast
bm	Atleast one NS doing anycast
bn	Atleast one NS doing anycast
bo	Atleast one NS doing anycast
br	Atleast one NS doing anycast
bs	Atleast one NS doing anycast
bt	Atleast one NS doing anycast
bw	Atleast one NS doing anycast
bz	Atleast one NS doing anycast
ca	Atleast one NS doing anycast
cc	Atleast one NS doing anycast
cd	Atleast one NS doing anycast
cf	Atleast one NS doing anycast
cg	Atleast one NS doing anycast
ch	Atleast one NS doing anycast
ci	Atleast one NS doing anycast
cl	Atleast one NS doing anycast
cm	Atleast one NS doing anycast
cn	Atleast one NS doing anycast
co	Atleast one NS doing anycast
cr	Atleast one NS doing anycast
cu	Atleast one NS doing anycast
cv	Atleast one NS doing anycast
cw	Atleast one NS doing anycast
cx	Atleast one NS doing anycast
cy	Atleast one NS doing anycast
cz	Atleast one NS doing anycast
de	Atleast one NS doing anycast
dj	Atleast one NS doing anycast
dk	Atleast one NS doing anycast
dm	Atleast one NS doing anycast
do	Atleast one NS doing anycast
dz	Atleast one NS doing anycast
ec	Atleast one NS doing anycast
ee	Atleast one NS doing anycast
eg	Atleast one NS doing anycast
er	Atleast one NS doing anycast
es	Atleast one NS doing anycast
eu	Atleast one NS doing anycast
fi	Atleast one NS doing anycast
fk	Atleast one NS doing anycast
fr	Atleast one NS doing anycast
ga	Atleast one NS doing anycast
gb	Atleast one NS doing anycast
gf	Atleast one NS doing anycast
gg	Atleast one NS doing anycast
gi	Atleast one NS doing anycast
gl	Atleast one NS doing anycast
gm	Atleast one NS doing anycast
gn	Atleast one NS doing anycast
gp	Atleast one NS doing anycast
gq	Atleast one NS doing anycast
gr	Atleast one NS doing anycast
gt	Atleast one NS doing anycast
gu	Atleast one NS doing anycast
gw	Atleast one NS doing anycast
gy	Atleast one NS doing anycast
hk	Atleast one NS doing anycast
hm	Atleast one NS doing anycast
hn	Atleast one NS doing anycast
hr	Atleast one NS doing anycast
ht	Atleast one NS doing anycast
hu	Atleast one NS doing anycast
id	Atleast one NS doing anycast
ie	Atleast one NS doing anycast
il	Atleast one NS doing anycast
im	Atleast one NS doing anycast
io	Atleast one NS doing anycast
iq	Atleast one NS doing anycast
ir	Atleast one NS doing anycast
is	Atleast one NS doing anycast
it	Atleast one NS doing anycast
je	Atleast one NS doing anycast
jm	Atleast one NS doing anycast
jo	Atleast one NS doing anycast
jp	Atleast one NS doing anycast
ke	Atleast one NS doing anycast
kg	Atleast one NS doing anycast
kh	Atleast one NS doing anycast
ki	Atleast one NS doing anycast
kn	Atleast one NS doing anycast
kw	Atleast one NS doing anycast
ky	Atleast one NS doing anycast
kz	Atleast one NS doing anycast
lb	Atleast one NS doing anycast
lc	Atleast one NS doing anycast
li	Atleast one NS doing anycast
lk	Atleast one NS doing anycast
lr	Atleast one NS doing anycast
ls	Atleast one NS doing anycast
lt	Atleast one NS doing anycast
lu	Atleast one NS doing anycast
lv	Atleast one NS doing anycast
ly	Atleast one NS doing anycast
ma	Atleast one NS doing anycast
mc	Atleast one NS doing anycast
md	Atleast one NS doing anycast
me	Atleast one NS doing anycast
mg	Atleast one NS doing anycast
mh	Atleast one NS doing anycast
mk	Atleast one NS doing anycast
ml	Atleast one NS doing anycast
mm	Atleast one NS doing anycast
mn	Atleast one NS doing anycast
mo	Atleast one NS doing anycast
mq	Atleast one NS doing anycast
mr	Atleast one NS doing anycast
ms	Atleast one NS doing anycast
mt	Atleast one NS doing anycast
mu	Atleast one NS doing anycast
mv	Atleast one NS doing anycast
mw	Atleast one NS doing anycast
mx	Atleast one NS doing anycast
my	Atleast one NS doing anycast
mz	Atleast one NS doing anycast
na	Atleast one NS doing anycast
nc	Atleast one NS doing anycast
ne	Atleast one NS doing anycast
nf	Atleast one NS doing anycast
ng	Atleast one NS doing anycast
ni	Atleast one NS doing anycast
no	Atleast one NS doing anycast
np	Atleast one NS doing anycast
nr	Atleast one NS doing anycast
nu	Atleast one NS doing anycast
nz	Atleast one NS doing anycast
om	Atleast one NS doing anycast
pa	Atleast one NS doing anycast
pe	Atleast one NS doing anycast
pg	Atleast one NS doing anycast
ph	Atleast one NS doing anycast
pk	Atleast one NS doing anycast
pl	Atleast one NS doing anycast
pm	Atleast one NS doing anycast
pn	Atleast one NS doing anycast
pr	Atleast one NS doing anycast
ps	Atleast one NS doing anycast
pt	Atleast one NS doing anycast
py	Atleast one NS doing anycast
qa	Atleast one NS doing anycast
re	Atleast one NS doing anycast
ro	Atleast one NS doing anycast
rs	Atleast one NS doing anycast
ru	Atleast one NS doing anycast
rw	Atleast one NS doing anycast
sa	Atleast one NS doing anycast
sb	Atleast one NS doing anycast
sc	Atleast one NS doing anycast
sd	Atleast one NS doing anycast
se	Atleast one NS doing anycast
sg	Atleast one NS doing anycast
sh	Atleast one NS doing anycast
si	Atleast one NS doing anycast
sj	Atleast one NS doing anycast
sk	Atleast one NS doing anycast
sm	Atleast one NS doing anycast
sn	Atleast one NS doing anycast
so	Atleast one NS doing anycast
st	Atleast one NS doing anycast
sv	Atleast one NS doing anycast
sx	Atleast one NS doing anycast
sy	Atleast one NS doing anycast
sz	Atleast one NS doing anycast
tc	Atleast one NS doing anycast
td	Atleast one NS doing anycast
tg	Atleast one NS doing anycast
th	Atleast one NS doing anycast
tj	Atleast one NS doing anycast
tk	Atleast one NS doing anycast
tl	Atleast one NS doing anycast
tm	Atleast one NS doing anycast
tn	Atleast one NS doing anycast
tr	Atleast one NS doing anycast
tt	Atleast one NS doing anycast
tv	Atleast one NS doing anycast
tw	Atleast one NS doing anycast
tz	Atleast one NS doing anycast
ua	Atleast one NS doing anycast
ug	Atleast one NS doing anycast
uk	Atleast one NS doing anycast
us	Atleast one NS doing anycast
uy	Atleast one NS doing anycast
uz	Atleast one NS doing anycast
va	Atleast one NS doing anycast
vc	Atleast one NS doing anycast
ve	Atleast one NS doing anycast
vi	Atleast one NS doing anycast
vn	Atleast one NS doing anycast
vu	Atleast one NS doing anycast
wf	Atleast one NS doing anycast
ws	Atleast one NS doing anycast
ye	Atleast one NS doing anycast
yt	Atleast one NS doing anycast
za	Atleast one NS doing anycast
zm	Atleast one NS doing anycast
zw	Atleast one NS doing anycast
by	None of NS doing anycast
ck	None of NS doing anycast
et	None of NS doing anycast
ge	None of NS doing anycast
gh	None of NS doing anycast
km	None of NS doing anycast
kr	None of NS doing anycast
pf	None of NS doing anycast
sr	None of NS doing anycast

Conclusion

I was partially correct (and partially incorrect). The majority of ccTLDs are not using anycast on all their nameservers but are using anycast on atleast one of their nameserver. As per this list, nameservers of 11 ccTLDs have full anycast, 9 have no anycast, and the remaining 219 have partial anycast, with some nameservers doing anycast, some not. Also, some ccTLDs are skipped from the list where all nameservers have ICMP closed. I should have probably tested with DNS latency instead of ICMP latency only.

The raw latency checks are posted here.

Analysing Starlink's upstreams in India

Thu, 07 Aug 2025 03:54:35 +0530

On 10th July, Starlink was approved by the Indian Govt as reported in various news articles.

In past I was tracking their GeoIP allocation sheet, but lost track of it over time as my tracking system was broken. Yesterday saw their IP allocations for Mumbai:

anurag@desktop ~> curl -s https://geoip.starlinkisp.net/feed.csv | grep Mumbai
150.228.128.0/24,IN,IN-MH,Mumbai,
150.228.129.0/24,IN,IN-MH,Mumbai,
150.228.130.0/24,IN,IN-MH,Mumbai,
150.228.131.0/24,IN,IN-MH,Mumbai,
150.228.132.0/24,IN,IN-MH,Mumbai,
150.228.133.0/24,IN,IN-MH,Mumbai,
150.228.164.0/24,IN,IN-MH,Mumbai,
150.228.165.0/24,IN,IN-MH,Mumbai,
150.228.166.0/24,IN,IN-MH,Mumbai,
150.228.167.0/24,IN,IN-MH,Mumbai,
2406:2d40:2800::/40,IN,IN-MH,Mumbai,
2406:2d40:2900::/40,IN,IN-MH,Mumbai,
2406:2d40:2a00::/40,IN,IN-MH,Mumbai,
2406:2d40:2b00::/40,IN,IN-MH,Mumbai,
anurag@desktop ~>

As of now, they have 10 x /24s IPv4 and 4 x /40 IPv6. Let’s explore the BGP announcement of these prefixes:

The first 4 are visible in the aggregate announcement of 150.228.128.0/22, the next four in 150.228.132.0/23 & 150.228.166.0/23. All these are visible in the global routing table. Due to Indian norms, these must be routed from within India. In simpler terms Starlink cannot beam internet in India while using ground stations from outside of India.

Upstream for Starlink gateway in India

Source: https://bgp.he.net/net/150.228.128.0/22#_graph

Source: https://bgp.he.net/net/150.228.132.0/23#_graph

Source: https://bgp.he.net/net/150.228.166.0/23#_graph

Thus they seem to be having Reliance Jio (AS55836) and Telstra (AS4637) as upstreams in India.

Quick points on their transit

Starlink seems to be using Jio & Telstra both for global reach. For domestic routing with Jio, it is via their Jio transit.
For domestic routing with other major networks - Airtel & Tata Comm - it seems to be via Telstra. I see announcements of Starlink behind Tata Comm AS4755 as well as to Airtel directly within India. All these are on the downstream side of Airtel & Tata Comm. So Startlink is an indirect customer of Airtel & Tata Comm in India for now.
Jio + Airtel + Tata Comm largely form the Indian routing table, as the majority of Indian routes are in the downstream of these networks. With these, they should be able to reach the majority of local networks. May still get some odd issues with Vodafone IDEA, Sify and BSNL, though, depending on whether the network of those prefixes is announced in domestic BGP sessions with Jio/Airtel/Tata Comm mix or not.

Telstra transit location

How to know if Telstra’s drop is within India? Well, their routing table as viewed from their looking glass gives hints.

Notice the tag 4637:30310. This matches 4637:3xx10,India as listed in the Telstra BGP community reference here.

Trace from Airtel to Starlink

Here’s a trace from Airtel FTTH in Haryana to Starlink Mumbai:

  1. AS???    172.16.36.1 (172.16.36.1)                             0.0%    10    0.2   0.3   0.2   0.4   0.1
  2. AS???    10.240.9.204 (10.240.9.204)                           0.0%    10    4.5   9.2   4.5  25.1   6.7
  3. AS???    172.31.0.195 (172.31.0.195)                           0.0%    10    5.2  16.8   5.2  56.7  15.0
  4. AS9498   173.168.18.125.dhcp.anaronline.net (125.18.168.173)   0.0%    10    5.3   6.4   4.7   8.4   1.3
  5. AS9498   116.119.33.34 (116.119.33.34)                         0.0%    10   31.1  35.4  29.4  47.8   6.9
  6. AS9498   nsg-static-042.114.72.182.airtel.in (182.72.114.42)   0.0%    10   29.8  30.6  29.3  32.0   0.9
  7. AS4637   194.50.57.210.adispace.co.jp (210.57.50.194)          0.0%    10   30.3  29.7  28.4  32.0   1.0
  8. AS14593  undefined.hostname.localhost (206.224.75.121)         0.0%    10   29.6  30.0  28.4  31.9   1.0
  9. AS14593  undefined.hostname.localhost (206.224.75.165)         0.0%    10   30.4  30.4  29.2  32.4   1.0
 10. AS???    ???                                                  100.0    10    0.0   0.0   0.0   0.0   0.0

Connectivity with Tata Comm

Telstra AS4637 outside of India usually seems to be a peer of Tata Comm AS6453; however, in this setup, they seem to be a customer of Tata Comm domestic AS4755 to route these prefixes.

SELECT prefix, as_path
FROM bgp.table
WHERE toDate(timestamp) = today() AND hasAll(as_path, [14593, 4755]) AND as_path[1] = 6453

Query id: 10b94036-1549-4111-817e-2bd59a6454b8

   ┌─prefix──────────────┬─as_path────────────────┐
1. │ 2406:2d40:2800::/40 │ [6453,4755,4637,14593] │
2. │ 2406:2d40:2900::/40 │ [6453,4755,4637,14593] │
3. │ 2406:2d40:2a00::/40 │ [6453,4755,4637,14593] │
4. │ 2406:2d40:2b00::/40 │ [6453,4755,4637,14593] │
   └─────────────────────┴────────────────────────┘
   ┌─prefix──────────────┬─as_path────────────────┐
5. │ 2605:59c0:5035::/48 │ [6453,4755,4637,14593] │
   └─────────────────────┴────────────────────────┘
   ┌─prefix───────────┬─as_path────────────────┐
6. │ 144.126.110.0/24 │ [6453,4755,4637,14593] │
7. │ 144.126.77.0/24  │ [6453,4755,4637,14593] │
8. │ 144.126.78.0/24  │ [6453,4755,4637,14593] │
   └──────────────────┴────────────────────────┘
    ┌─prefix───────────┬─as_path────────────────┐
 9. │ 150.228.128.0/22 │ [6453,4755,4637,14593] │
10. │ 150.228.132.0/23 │ [6453,4755,4637,14593] │
    └──────────────────┴────────────────────────┘
    ┌─prefix──────────────┬─as_path────────────────┐
11. │ 2406:2d40:2800::/40 │ [6453,4755,4637,14593] │
12. │ 2406:2d40:2900::/40 │ [6453,4755,4637,14593] │
13. │ 2406:2d40:2a00::/40 │ [6453,4755,4637,14593] │
14. │ 2406:2d40:2b00::/40 │ [6453,4755,4637,14593] │
    └─────────────────────┴────────────────────────┘
    ┌─prefix───────────┬─as_path────────────────┐
15. │ 144.126.110.0/24 │ [6453,4755,4637,14593] │
16. │ 144.126.77.0/24  │ [6453,4755,4637,14593] │
17. │ 144.126.78.0/24  │ [6453,4755,4637,14593] │
    └──────────────────┴────────────────────────┘
    ┌─prefix──────────────┬─as_path────────────────┐
18. │ 2605:59c0:5035::/48 │ [6453,4755,4637,14593] │
    └─────────────────────┴────────────────────────┘
    ┌─prefix───────────┬─as_path────────────────┐
19. │ 150.228.128.0/22 │ [6453,4755,4637,14593] │
20. │ 150.228.132.0/23 │ [6453,4755,4637,14593] │
    └──────────────────┴────────────────────────┘

Disclaimer: This is my personal blog and hence posts made here are in my personal capacity. These do not represent the views of my employer.

Understanding erasure coding

Sun, 27 Jul 2025 14:38:17 +0530

Over time, I went down the rabbit hole of reading about erasure coding on cloud platforms. It’s quite interesting how these players promise 99.999999999% (that’s eleven 9s) of durability for the stored data.

While Backblaze documents this to great detail on their 17+3 (or 15+5) storage design, AWS S3 so far has been a mystery. They provide extensive documentation on their user-facing APIs and interfaces, but offer little insight into the limitations behind their design choices. Their presentation at AWS re:Invent 2024 talks about some of it in detail. Must say that their re:Invent is way more interesting than their documentation. 😀

Erasure coding

The basic idea behind erasure coding is that drives can fail at any time, and hence, redundancy is needed. One way to have redundancy is to have an exact copy on another disk.

Imagine disk01 and disk02 - both holding the exact same set of bits. Now, if disk01 fails, disk02 will be the only one holding the data. Single disk failures are quite common at scale, and it’s too risky to be holding critical data on disk02 alone for now and that too when it has to be completely read to rebuild the copy. This brings the idea to three copies - store data in disk01, disk02 and disk03.

This works in theory, but has two major challenges:

If two disks go down, data will be at major risk
Storage cost has gone 3x.

To address the first challenge, one can add another disk, and now we have disk01, disk02, disk03 and disk04 - each holding the exact same data, and one can lose up to three of them, which is quite reasonable, but this worsens the other issue as now to store x, we are using 4x capacity. To solve this, most storage systems at scale use erasure coding. The idea is to break data into chunks & pass through a pre-defined algorithm and mathematically calculate parity data (often called shards).

In the case of Backblaze design, they break files into 17 pieces (17 data shards) and calculate 3 parity bits (3 data shards). Each of these shards (whether data or parity) is of the same size as the others, and any 17 of these in any mix can reproduce the original data. Thus, storing 20 (17+3) shards across 20 different disks gives the ability to lose up to three drives without losing data. This provides the same fault tolerance as storing 4 identical copies, but instead of 4x storage, it’s just 1.18x storage, which is the real magic. Thus, for around 17.65% overhead on storage, this gives the ability to lose up to three drives without losing data.

In case of AWS S3 (as given in the presentation), they do a 5+4 design (5 data shards, 4 parity shards), giving them 9 shards. They likely went for this because of a number of factors, with one being a number which can be evenly divided by 3. That way, they can store 3+3+3 shards across three different data centres and need only any 5 shards to rebuild the original data.

Tradeoff between shard count and storage overhead

This 5+4 = 9 shards may make one wonder - why not 5+1 = 6 and store 2 + 2 + 2 in each failure zone and store 6 in each zone. Each design: More shards vs Fewer shards — has its own advantages.

Let’s compare a 5+4 Vs 5+1. The number of shards needed to recover data in 5+4 is 5, while in 5+1 it is also 5. This means if all shards are spread around different hard disks, one can lose 4 HDDs in a 5+4 design but only 1 in a 5+1 design. More than 1 HDD loss will cause data loss. Thus, the number of shards fewer than 3 - 4 can be hard (as that’s what defines acceptable HDD failure).

What about high shard count? Why not 11+7 = 18 shards, with 6 going to each failure zone? In case of 11+7, they can lose up to 7 disks with no data loss; however, it will take much more CPU. Also, spreading data to more disks will risk overloading I/O across many disks without much return. Another factor is storage overhead.

Here’s a table to calculate overhead in these configs:

Scheme	Data Shards	Parity Shards	Disk Failure Tolerance	Storage Overhead
17+3	17	3	3	17.65% (3/17)
15+5	15	5	5	33.33% (5/15)
5+4	5	4	4	80% (4/5)
11+7	11	7	7	63.64% (7/11)
11+10	11	10	10	90.91% (10/11)

Backblaze once in their post in 2024 mentioned:

The ratios of data shards to parity shards we currently use are 17/3, 16/4, and 15/5, depending primarily on the size of the drives being used to store the data—the larger the drive, the higher the parity.

This had to do with restore times. For higher capacity drives (16TB), restore time is much higher than a 4TB drive. Thus, when a drive fails in a 17+3 design, they are effectively on 17+2 for a few days. Doing a 15+5 allows them to have tolerance up to 5 disk failures, but for 33.3% overhead (instead of 17.6%).

Playing with erasure coding

Most of these systems are built on Reed-Solomon codes. There are many open source implementations available for it, including from Backblaze - JavaReedSolomon, Go implementation and Python library - pyeclib.

Product-wise, these concepts are available in open source Minio if one is planning to self-host with erasure coding.

I picked up pyeclib and tried with tools from openstack, which provide scripts for simple encode/decode etc.

> python3 pyeclib_encode.py --help
usage: pyeclib_encode.py [-h] k m ec_type file_dir filename fragment_dir

Encoder for PyECLib.

positional arguments:
  k             number of data elements
  m             number of parity elements
  ec_type       EC algorithm used
  file_dir      directory with the file
  filename      file to encode
  fragment_dir  directory to drop encoded fragments

options:
  -h, --help    show this help message and exit

Let’s generate a dummy 1GB file, calculate its checksum, break it into 17+3, restore and do checksum again.

Create a test file

> dd if=/dev/zero of=output_1GB_file.bin bs=1G count=1
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.1953 s, 898 MB/s

> sha256sum output_1GB_file.bin 
49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14  output_1GB_file.bin

Now let’s create shards for it based on liberasurecode:

> mkdir shards_17_3

> python3 pyeclib_encode.py 17 3 liberasurecode_rs_vand . output_1GB_file.bin shards_17_3
k = 17, m = 3
ec_type = liberasurecode_rs_vand
filename = output_1GB_file.bin

> ls shards_17_3
output_1GB_file.bin.0   output_1GB_file.bin.11  output_1GB_file.bin.14  output_1GB_file.bin.17  output_1GB_file.bin.2  output_1GB_file.bin.5  output_1GB_file.bin.8
output_1GB_file.bin.1   output_1GB_file.bin.12  output_1GB_file.bin.15  output_1GB_file.bin.18  output_1GB_file.bin.3  output_1GB_file.bin.6  output_1GB_file.bin.9
output_1GB_file.bin.10  output_1GB_file.bin.13  output_1GB_file.bin.16  output_1GB_file.bin.19  output_1GB_file.bin.4  output_1GB_file.bin.7

> du -s output_1GB_file.bin
1048580	output_1GB_file.bin

> du -s shards_17_3
1233684	shards_17_3

Thus, size went up by 185104 bytes (17.65% overhead).

Let’s randomly delete three shards and recover data:

> rm shards_17_3/output_1GB_file.bin.0 
> rm shards_17_3/output_1GB_file.bin.10
> rm shards_17_3/output_1GB_file.bin.19 

> python3 pyeclib_decode.py 17 3 liberasurecode_rs_vand shards_17_3/output_1GB_file.bin.1 shards_17_3/output_1GB_file.bin.2 shards_17_3/output_1GB_file.bin.3 shards_17_3/output_1GB_file.bin.4 shards_17_3/output_1GB_file.bin.5 shards_17_3/output_1GB_file.bin.6 shards_17_3/output_1GB_file.bin.7 shards_17_3/output_1GB_file.bin.8 shards_17_3/output_1GB_file.bin.9 shards_17_3/output_1GB_file.bin.11 shards_17_3/output_1GB_file.bin.12 shards_17_3/output_1GB_file.bin.13 shards_17_3/output_1GB_file.bin.14 shards_17_3/output_1GB_file.bin.15 shards_17_3/output_1GB_file.bin.16 shards_17_3/output_1GB_file.bin.17 shards_17_3/output_1GB_file.bin.18 recovered-output_1GB_file.bin
k = 17, m = 3
ec_type = liberasurecode_rs_vand
fragments = ['shards_17_3/output_1GB_file.bin.1', 'shards_17_3/output_1GB_file.bin.2', 'shards_17_3/output_1GB_file.bin.3', 'shards_17_3/output_1GB_file.bin.4', 'shards_17_3/output_1GB_file.bin.5', 'shards_17_3/output_1GB_file.bin.6', 'shards_17_3/output_1GB_file.bin.7', 'shards_17_3/output_1GB_file.bin.8', 'shards_17_3/output_1GB_file.bin.9', 'shards_17_3/output_1GB_file.bin.11', 'shards_17_3/output_1GB_file.bin.12', 'shards_17_3/output_1GB_file.bin.13', 'shards_17_3/output_1GB_file.bin.14', 'shards_17_3/output_1GB_file.bin.15', 'shards_17_3/output_1GB_file.bin.16', 'shards_17_3/output_1GB_file.bin.17', 'shards_17_3/output_1GB_file.bin.18']
filename = recovered-output_1GB_file.bin

> sha256sum recovered-output_1GB_file.bin.decoded 
49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14  recovered-output_1GB_file.bin.decoded

And here’s one similar to what AWS S3 uses:

> mkdir shards_5_4

> python3 pyeclib_encode.py 5 4 liberasurecode_rs_vand . output_1GB_file.bin shards_5_4/
k = 5, m = 4
ec_type = liberasurecode_rs_vand
filename = output_1GB_file.bin

> du -s shards_5_4/
1887448	shards_5_4/

1887448 bytes (80% overhead), which may seem high, but once we consider it’s better than 4x storage and with distribution of 3+3+3 shards in each failure zone, AWS can bear the loss of a full zone and even an additional loss of one shard in the remaining DC and still recover the data.

Storage is fascinating. 😀

Cloudflare DNS outage & BGP route hijack

Wed, 16 Jul 2025 01:40:00 +0530

Yesterday was quite an eventful day on the internet with an outage on Cloudflare’s public DNS - 1.1.1.1. One can say impact is high when even Indian media starts reporting it (like this and this). 😀

Source: Cloudflare community

Status doesn’t tell much and there has been speculation on BGP hijack by Tata Comm Indian network AS4755 causing it. Cloudflare’s own tool (unrelated to their DNS) reported that Tata Comm AS4755 originated 1.1.1.0/24 for a period of 7 minutes. The timeline here is interesting. Tata Comm’s hijack ended at 22:01 while Cloudflare reports issues with 1.1.1.1 at 22:13 but routing-wise it doesn’t make any sense. Hijack of Cloudflare prefix by Tata Comm cannot cause such a widespread outage.

For prefix hijack it always makes sense to look at the AS_PATH before coming to a conclusion as in theory one can hijack a prefix by using AS4755 in origin in the AS_PATH and if BGP adjacencies do not filter, they will pick it up. Let’s look for 1.1.1.0/24 via RIPE BGPPlay for this duration. This link should load BGPplay with an exact time filter.

RIPE RIS Collector RRC03 - Amsterdam shows 1.1.1.0/24 with path 6453 4755. This is a real route as AS6453 directly feeds RRC03.

Limited route propagation

While the announcement was bad but the impact would be extremely low and cannot cause a global outage. Here’s why:

Cloudflare AS13335 is directly connected to all major backbones and announcing 1.1.1.0/24 directly. The only exception to this is German Deutsche Telekom - DTAG AS3320. Most of them consider Cloudflare as a customer (some as a peer). All of them consider Tata Comm a peer. Thus they would either have high local pref on Cloudflare (where Cloudflare is downstream) compared to Tata Comm (a peer) or the same localpref on Cloudflare (a peer) vs Tata Comm (also a peer).
RPKI validation would drop invalid hijacked announcement by many large networks.
The majority of small to mid-sized networks are peered to Cloudflare directly & hence would pick that route over the route of Cloudflare coming via AS6453 > AS4755 path except a) Non-Cloudflare peer downstream networks of Tata Comm AS4755 in India and b) DTAG AS3320.
Tata Comm would normally treat both - their domestic Indian network AS4755 and Cloudflare AS13335 as downstream of AS6453 and hence with the same local preference. Thus if AS13335 announcement is up, likely AS4755 route will not show up outside of India.

Domestic Indian routing with AS4755

This is a classic case of hijack of the route of a well-peered network (Cloudflare) by a transit-free network (selective peering) and hence quite tricky to detect. As long as the Cloudflare announcement is up, the majority of public collectors outside of India will not get it.

Since it’s from AS4755, route announcement at NIXI Chennai by AS4755 as picked by PCH collector on 14 July 2025 - route-collector.maa.pch.net:

Timestamp (GMT)	Type	BGP Adjacency	Prefix	AS_PATH	BGP Communities
9:50:07 PM	Announcement	24029	1.1.1.0/24	24029 4755 13335	4755:25 4755:44 4755:199 4755:888 4755:2000 4755:40018 4755:47556 6453:9636
9:53:06 PM	Withdrawal	24029	1.1.1.0/24
9:53:38 PM	Announcement	24029	1.1.1.0/24	24029 4755	4755:44 4755:99
10:19:53 PM	Announcement	24029	1.1.1.0/24	24029 4755 13335	4755:25 4755:44 4755:199 4755:888 4755:2000 4755:40018 4755:47556 6453:9636
10:19:55 PM	Announcement	24029	1.1.1.0/24	24029 4755 13335	4755:25 4755:44 4755:199 4755:888 4755:2000 4755:40018 4755:47556 6453:9636

BGP community 4755:44 - 44 here is likely Chennai (STD code of Chennai - Tata Comm & Airtel both use STD codes in their BGP communities). This is a bit strange as there is clearly visible withdrawal of Cloudflare’s announcement at 9:53:06 PM and within 22 seconds there was origination by AS4755. This cannot be manual and as already guessed my friend Doug Madory in his tweet - I also think that AS4755 announcement was already there and was normally suppressed. It showed up when AS13335 pulled its announcement for whatever internal issue they had. Possibly Tata Comm AS4755 has high localpref on eBGP routes compared to locally originated routes. The majority of networks set high localpref on downstream routes but sometimes do not match localpref on locally originated routes.

Likely that announcement is still there inside AS4755 but just suppressed for now. In past, I have seen 1.1.1.1 Anycast broken on Tata Comm. It always routes it via Chennai. Wonder if it’s related. Hard to know for sure since the AS4755 looking glass has been broken for years.

Run your own DNS resolvers!

This is also a reminder to various network operators, datacenter and Cloud players - run your own DNS resolvers. There’s no good reason to use 1.1.1.1 or 8.8.8.8 for your customers since there’s no SLA, no contact point involved with Cloudflare or Google. They may go down and worst - they can rate limit you anytime. A couple of unbound instances with anycast within your network can take you very far.

Disclaimer: This is my personal blog and hence posts made here are in my personal capacity. These do not represent the views of my employer.

Update - 15 July 2025

Cloudflare has shared some details of outage in this post. This confirms that their outage was not related to route hijack by AS4755.

On June 6, during a release to prepare a service topology for a future DLS service, a configuration error was introduced: the prefixes associated with the 1.1.1.1 Resolver service were inadvertently included alongside the prefixes that were intended for the new DLS service. This configuration error sat dormant in the production network as the new DLS service was not yet in use, but it set the stage for the outage on July 14.

Change in Google's peering policy

Wed, 02 Jul 2025 03:49:10 +0530

In the last few months, there’s been a constant discussion around Google’s change of peering policy. This has been across various ISP groups, NOG meetings etc. For those who may not know - Google has gone from “Open peering policy” into a “Selective peering policy” sometimes between 14 Feb 2025 - 06 Mar 2025.

This initially started as Google stop peering over the IXPs and that actually made technical sense. Later more & more ISPs, content players started complaining about Google being closed on the peering front by either not responding, delaying and in some rare cases even rejecting peering requests. Today Dr. Peering on X pointed to Google’s presentation at NANOG 94 about the change in the peering policy.

Dave Schwartz from Google explaining Google’s peering policy and Verified Peering Provider (VPP) program at #NANOG94. Found via LinkedIn, video here … https://t.co/EGEVAR6MJc pic.twitter.com/z0suA2wlFJ
— Dr. Peering (@DrPeering) July 1, 2025

This is interesting as finally there’s an official presentation by Dave Schwartz on some of the challenges they had and what triggered them in this direction.

What has changed

Google is not going to do bilateral peering over an IXP with any new network.
They prefer 100G PNIs with networks with at least 10G traffic (Google -> Other network direction, peak 10G flow).
For everyone who doesn’t fit into the 100G port model, they suggest coming via VPP which is simply a certification from Google that the given network is peered to them with different grades of redundancy (as defined by silver Vs gold status) or respective upstream IP transit provider.
Google will not deploy (expensive) YouTube edge at smaller peering locations anymore.

Some quick points around why Google is not openly anymore

Ten years ago (2015) they were expanding, traffic flows were unpredictable and they needed a massive amount of bandwidth. In that mix, peering made sense (besides many other things). Now in 2025, traffic growth is predictable, they already have a massive amount of peering with many players and (my guess) the peering situation of AS15169 is reaching a diminishing return.
Google Cloud is now at the centre of everything - from investment to growth, from product to AI adoption. Essentially more peering with lower capacities is causing them planning issues as explained in slides 6 & 7. Running a small 10G port at an IX or even doing remote peering with a small transport wavelength can get parts of their edge congested at times.
Their network expansion is driven by Google Cloud & not by YouTube anymore.
Many of their Cloud customers are running VPN concentrators in Google Cloud and expect stable latency. They seem to be having transport challenges within the metro during (planned/unplanned) outages as well as random traffic spikes.
They are pushing layer 2 IXPs to become layer 3 partial transit providers by enrolling in VPP.

Does it make technical sense?

Things which he said technically make sense if we see them in isolation. But if we see the internet at large, many of those things are questionable. Take e.g his latency spike case from NSW-IX (New South Wales Internet Exchange):

Surely Google can get edge port congested if they run a 10G port at an IX with at least 24 participants with over 100G capacity, 7 players with 40G and many players with 10G ports. It’s unclear from Dave’s presentation when this issue occurred but even if I check now their port size is just 20G at this IXP. If they simply do not add any new peers anymore - a congestion issue can still come from existing peers, especially from the internal transport network of their large peers which maintain a backbone.

Why layer 2 peering was better than transit or even partial transit/layer 3 peering in the first place? That is because if we take a 100G wavelength or run 100G over dark fibre to Google, we know for sure that capacity exists and can be used. If I try to do the same via another layer 3 network in between (or even an MPLS circuit), there is no guarantee of the capacities involved. The link might be over an oversold transport.

Technically VPP is simply “offloading congestion problem” to someone else (large to mid-sized networks).

Commercial angle

Ultimately there’s a clear commercial angle to these decisions. Dave hints at that in his talk multiple times. From hints as well as from what we know from public data:

Cloud and AI growth needs massive networking resources in terms of engineering manpower, hardware, datacenter capacity etc and Google is trying to divert some of that from its peering ecosystem. Remember AI needs massive bandwidth within DC and super tiny with the internet-facing edge.
The cost of borrowing money in the US has gone up significantly. 2015 was the time when everyone from Google to CDN players like Akamai/Fastly and many more were expanding fast. Money was “cheap” and hence many things were possible back then but not anymore.
While Google may not be technically a tier 1 transit-free network, for all practical reasons it is. Their traffic volume numbers are not known but I would guess a super-tiny part of their traffic goes via their transit. Thus whether they peer with a smaller network or not, it has virtually no impact on their transit traffic. Only their traffic volume with the dominant incumbent would increase in that market.
The cost of IP transit, peering, and hardware has declined over time but the cost of manpower has gone crazy. I am writing this post when just 5 days ago Meta hired three OpenAI researchers with bonuses of as high as $100 million. For Google spending $300k - $500k on engineers, the manpower cost is just huge compared to the money going on the other aspects of the network including peering.
More and more traffic is getting concentrated over the years. Take e.g my home country India - Jio & Airtel have massive eyeballs. Google’s traffic once we exclude Airtel/Jio and a few other PNIs, is likely a low single-digit percentage. I can imagine it won’t be very different in many countries around the world with a very high concentration of eyeballs on a limited set of players. Imagine traffic in the US once you exclude AT&T, Verizon, T-Mobile. Comcast & a dozen more networks which already have the PNI.

So ultimately it will degrade things for a small set of networks (and their users) and it’s a calculated loss Google has decided to take with this strategic shift in peering. Many of us have seen a vibrant IXP ecosystem develop because of Google’s aggressive peering push. An interesting note 2015-2016 was such a crazy growth time that Mr Geoff Huston (Scientist at APNIC and my good friend) even declared “death of transit”.

Disclaimer: This is my personal blog and hence posts made here are in my personal capacity.
These do not represent the views of my employer.

Common misconceptions about peering ecosystem in India

Wed, 11 Jun 2025 03:58:29 +0530

Last week I attended ISOC-Medianama peering roadshow in New Delhi. The core idea for the discussion was about TRAI’s recommendation to bring authorization to IXPs in India.

Somehow during the discussion, the point about incumbents not peering in India kept on coming up. One of the presenters casually commented on how incumbents are not peering in India with smaller networks and thus smaller networks have to buy transit. This is technically incorrect and confuses peering with transit. During the panel discussion things again went in the direction of how the peering ecosystem is failing because incumbents do not peer at the local IXPs. In this post, I am going to explain these misconceptions in detail.

Misconception #1: Small ISPs need transit because large players do not peer

It’s important to first understand the difference between peering and transit. Let’s say we have two networks: A and B. Here’s what happens if A peers with B Vs A buys IP transit from B:

Peering	IP Transit
A can only reach B (and customers of B). B can only reach A and customers A.	A can reach any network in the world via B
A would get B’s own routes + customer routes only	A would get B’s own routes + B’s customer routes + B’s peer routes + B’s upstream routes

I covered this concept long back in Oct 2020 at the INNOG Tech session. Slides 6-11 here cover the difference in detail. By just peering a couple of Indian operators one would not get access to the full routing table and transit would still be needed. The only situation where peering will get access to the full routing table would be to peer with all known transit-free tier 1 networks and all (except Tata Comm) in the list are not present in India let alone the closed peering policies. So the idea of someone serving within a city, state or even a single country would not be practical.

Misconception #2: Indian peering ecosystem is not good because telcos do not peer

Again, this point keeps coming up. It is often suggested that peering is bad because large telcos do not peer with smaller ISPs. In reality, the majority of traffic goes from content to eyeball ISP networks. The majority of content players present in India have an open peering policy & peer with a mix of PNI (Private Network Interconnect) and IX peering. Interestingly majority are peers of large incumbent networks. Thus their routes won’t even be visible to an ISP peer technically even if say Airtel/Jio agree to peer with a small ISP who is expecting content traffic.

I am a supporter of an open peering policy, I prefer to buy my own home, and server connectivity from players with an open peering policy. But with that being said whether incumbent peers openly or not should not impact the peering ecosystem. Pick any large IXP in the world and see if incumbents in those countries are peering openly.

Let’s pick some example IXPs from Terabit club page:

IXP	Dominant Incumbent	Peering policy of incumbent
DECIX Frankfurt	Deutsche Telekom	Restrictive (peeringdb)
Equinix Singapore	Singtel	Selective (peeringdb)
LINX London	British Telecom	Restrictive (peeringdb)
France IX	Orange	Restrictive (peeringdb)
Namex IXP Italy	Telecom Italia	Restrictive (peeringdb

I can go on and on here. Whether incumbents have open or restrictive peering policies - the impact of that is on outbound heavy content players and not the eyeball ISPs in the ecosystem. This is specially true for hosting companies, smaller cloud players, VPS providers, datacenters etc.

Decisions to peer or not should be left to free market decsion making. The only situation where it makes sense to force peering should be if ISPs have an exclusive last mile/termination monopoly (like some of the countries in the EU). In India, I guess BSNL at one point had it (much before content peering even came into existence) but for now no one - BSNL, Airtel, Jio, ACT etc has termination monopoly rights in an area (as far as I know).

Indian IX peering ecosystem has low traffic (not the peering system outside of IXP) for the scale and size of the country because:

Low numbers of end users outside of PNI capacity - There are close to 900 million mobile broadband users compared to just 41 million fixed-line users. And even in the fixed-line userbase of 41 million, 27.8% is Jio, 20.7% is Airtel, 10.5% BSNL, and 5.6% ACT and hence these top 4 players carry 67.6% of the broadband market. So what is left is just 32.4% of fixed-line or barely 13.2 million users on smaller players where IX peering would be useful. For the top 4 it’s mostly 100G PNIs with large content networks. It would probably extend to some players in that 32.4% bracket like my ex-employer Spectra, Excitel, Ishaan, Sify, Microscan etc. In the overall scheme of things ISPs serving 13.2 million need IX peering which accounts for just 1.4% of the overall user base (wireless+fixedline).
Availability of circuits & regulation - More and more options should be encouraged to promote peering. One major roadblock to it is active infra-sharing. Take e.g if an ISP serving me in Haryana is building its dark fibre to Delhi to peer, they can light fibre for their own usage and peer but they cannot run DWDM and sell capacity without an expensive NLD / access license. Govt. should try to design a way to make 10G/100G wavelengths available in a radius of say 500-600Km from existing large peering hubs like Delhi, Mumbai, and Chenna over a mix of NOFN, Railtel, BSNL, Powergrid etc. The rest private sector would immediately put that sort of capacity to use.
Lack of India international peering hub - Find out who peers at IXPs I put the above list - non-local networks! Take e.g. DECIX Frankfurt where the majority of networks from Asia, the Middle East and Africa peer with content hosted in the EU. We should promote peering of ISPs from Nepal, Bhutan, Sri Lanka, UAE, Singapore (and Bangladesh once it’s stable again) at Indian IXPs. As of now some of it happens, some of it is grey area in Indian regulation, some of it is grey area in those respective countries regulation.

Misconception #3: There is no peering outside of the IXP ecosystem in India

There is a massive amount of peering & caching outside of the IXP ecosystem in India. I often post about the growth of Facebook FNA caches. Last post from May 2024 points out that Facebook FNA caches exist at 79 unique airport codes. The real number city-wise is much higher because many cities do not have airport codes. Take e.g. Airtel. They recently set up BNG in Ambala to serve users from Haryana moving off from the Mohali / Chandigarh tri-city area. Both the Mohali node and Ambala node carry the airport code of IXC (Chandigarh). As of now Airtel + Jio have Google/Facebook caches in almost all state capitals / large cities with BNG, along with quite a lot of Akamai nodes followed by PNIs for peering with large content players. The same is true for players like ACT Broadband. Thus it’s safe to say majority (80%+) of bit delivered in India come from a mix local caches within ISPs as well as their content peerings.

Remember: Someone up the transit path there is always peering!
With hope that you reach one of my CDN edge caches hosted on VMs via peering to read this blog post, time for me to end this post.

Disclaimer: The views expressed on this blog are my own and do not reflect those of my employer.