Isp-Column

Last month after the mnNOG 5 event in Mongolia, I got a chance to visit Skytel’s IPTV headend. Skytel is one of the large Mongolian operators doing mobile and fixed-line networks. Tour of network infrastructure is always interesting and this time it was not just IP but broadcast network over the IP which excited me. In past, I have visited datacenters, IXPs etc and those are mostly IP (layer 3) or ethernet (layer 2) networks. This was the first time a tour of the broadcast infrastructure. Special thanks to friends from Skytel especially my friend Tuvshuu from the IP broadcast network team for arranging this tour of this infrastructure.

I have been in Ulaanbaatar, Mongolia for the last few days. This is my first travel to Mongolia and this far up in the North (except for previous travel to Russia in 2016 and some parts of Nordic areas in Europe). Geographically Mongolia is located between Russia (on the North side) and China (on the South side).

I am here for mnNOG 5 event. mnNOG is the Mongolian Network Operators Group. On Monday mnNOG conducted it’s 5th annual conference and it followed a five-day workshop. I am doing a workshop here on Network automation along with engineers from local networks. Mongolia is a landlocked country with no access to sea and hence no sub-sea cable. For the internet Mongolia relies on in-land fibre optic cables to connect to Russia and China. Though cables connect physically via Russia and China, I do not see Mongolian networks doing L3 termination in Russia or Mainland China. Instead, there is connectivity to Hong Kong, Singapore, Frankfurt etc. for the L3 connectivity. Interestingly due to it’s geographic location, a bit of China-Russia internet traffic exchange happens via Mongolia.

Earlier in March I visited Andaman & Nicobar Islands. The trip was purely personal as my wife happened to have been born there. These are Indian islands in the Bay of Bengal located in the South East of West Bengal and geographically quite near Myanmar and Thailand. The nearest large Indian cities on the mainland are Kolkata and Chennai.

In the initial part of the trip, we stayed in Swaraj Dweep (old name Havelock islands) and later in Port Blair. The place is isolated and has amazing natural beauty. It has one of the most beautiful beaches in the world (Radhanagar Beach). Tourism has grown nicely in Andaman & Nicobar islands in recent years and besides many other factors, one of that is a submarine cable!

I visited Delhi earlier today and noticed latency from my phone on Jio 5G to my home (on an ISP behind Airtel) was just 20ms. It varied a bit (as one would expect on a wireless radio network) but 20ms is special because until now it was at least 80-90ms. After all, Jio and Airtel were not connected in Delhi NCR until now. There were sometimes jokes about them being connected at NIXI Noida but that never pushed any traffic because NIXI injects its route server AS24029 in the AS_PATH and while their direct PNIs (i.e Private Network Interconnect) in Mumbai & Chennai would have a direct (short) AS_PATH.

A rather long title but the post is about self-hosted open-source mesh VPN with IPv6 support and works with nodes behind CGNAT!
This will be a long post documenting the concept of mesh VPN, the problem it is solving as well as a working demo. If you are not planning to deploy it right away, you can skip the post after the “Configs and setup” section.

Problem

I am running a site-to-site VPN for a long time between various servers located far away from each other. Originally these used to be on OpenVPN and later I moved to wireguard. These were not mesh but rather in a linear topology. I would have a home node here in Rohtak connected to two different servers in Mumbai over two different ISPs (via policy-based routing), those two Mumbai nodes would maintain the site-to-site VPNs with a few servers in Europe & those servers further connect to a few servers in the US. This setup ensured private network connectivity with encryption so that I can have GitLab runners spread around based on available CPU load and those runners would speak to database/storage servers securely without having to deal with encryption on per project/app basis. This also gave me basic features like running cameras are home which feeds into the Frigate instance in Mumbai for motion detection-based recording, monitoring these cameras & other device uptime using the uptime-kuma instance in Ashburn etc.