Isp-Column

Earlier in March I visited Andaman & Nicobar Islands. The trip was purely personal as my wife happened to have been born there. These are Indian islands in the Bay of Bengal located in the South East of West Bengal and geographically quite near Myanmar and Thailand. The nearest large Indian cities on the mainland are Kolkata and Chennai.

In the initial part of the trip, we stayed in Swaraj Dweep (old name Havelock islands) and later in Port Blair. The place is isolated and has amazing natural beauty. It has one of the most beautiful beaches in the world (Radhanagar Beach). Tourism has grown nicely in Andaman & Nicobar islands in recent years and besides many other factors, one of that is a submarine cable!

I visited Delhi earlier today and noticed latency from my phone on Jio 5G to my home (on an ISP behind Airtel) was just 20ms. It varied a bit (as one would expect on a wireless radio network) but 20ms is special because until now it was at least 80-90ms. After all, Jio and Airtel were not connected in Delhi NCR until now. There were sometimes jokes about them being connected at NIXI Noida but that never pushed any traffic because NIXI injects its route server AS24029 in the AS_PATH and while their direct PNIs (i.e Private Network Interconnect) in Mumbai & Chennai would have a direct (short) AS_PATH.

A rather long title but the post is about self-hosted open-source mesh VPN with IPv6 support and works with nodes behind CGNAT!
This will be a long post documenting the concept of mesh VPN, the problem it is solving as well as a working demo. If you are not planning to deploy it right away, you can skip the post after the “Configs and setup” section.

Problem

I am running a site-to-site VPN for a long time between various servers located far away from each other. Originally these used to be on OpenVPN and later I moved to wireguard. These were not mesh but rather in a linear topology. I would have a home node here in Rohtak connected to two different servers in Mumbai over two different ISPs (via policy-based routing), those two Mumbai nodes would maintain the site-to-site VPNs with a few servers in Europe & those servers further connect to a few servers in the US. This setup ensured private network connectivity with encryption so that I can have GitLab runners spread around based on available CPU load and those runners would speak to database/storage servers securely without having to deal with encryption on per project/app basis. This also gave me basic features like running cameras are home which feeds into the Frigate instance in Mumbai for motion detection-based recording, monitoring these cameras & other device uptime using the uptime-kuma instance in Ashburn etc.

Last month I got access to Jio 5G like everyone else around in Haryana. They are running a beta program with uncapped data for now. Overall it works fine for usual stuff (web surfing on popular sites, YouTube videos, music streaming etc) but 464XLAT seems to be a little buggy in IPv4 hardcoded destinations. Initially it was giving quite a few issues but many of them seem to be fixed in last few days.

Last year had many interesting developments and one of that has been object storage. For those unaware, object storage is de-facto cloud storage which stores data as objects instead of file system architecture. This gives the option of simple plug-and-play horizontal scalability. It became popular when Amazon Web Services (AWS) launched S3. The idea was straightforward - pay-as-go storage with a few cents/GB/month charge to store data and a few cents/GB to egress data. No need to plan storage, no need to plan hard disk, storage servers, or rack capacity but a simple pay-as-you-go opex cost. Plus top tier cloud players do offer redundancy of data. The API replies with “success” on uploads only when data is replicated to multiple datacenters.