Background

I am running my own authoritative DNS servers for the last few years. In earlier stages I just used registrar-provided DNS, later moved to “Cloud provider” provided DNS and ultimately settled for running my own auth DNS.

Two major requirements pushed me to self-host auth DNS:

1. Requirement of REST API for DNS needed by the web servers to resolve Letsencrypt certbot DNS-based challenge. This allows me to have internally hosted tools with Letsencrypt issues TLS certificates instead of self-signed ones. The API access is mostly missing in the registrars hosted DNS.
2. Occasional DDoS on my blog. There have been occasional DDoS on my blog (probably from random people who enjoy doing volumetric attacks). This always worried me about DNS bills during DDoS, especially for low TTL records. The last hosted DNS option I had over a year ago was Google Cloud DNS and they charge $0.40 per million queries per month. This can add a significant amount to the bill if under massive DDoS resulting in many millions of DNS queries. Plus per zone 20 cents charge gets expensive at a scale with a half a dozen domains.

After exploring a few options I settled for running PowerDNS with MySQL backend. This is kind of a comfort zone since I ran similar systems for my employers in past and it worked well. PowerDNS is a great option for authoritative DNS as it has nice documented REST API, CLI utility pdnsutil for easy high-level scripting, supports a bunch of backends to store DNS records from BIND like text files to MySQL. It is also good (automated) support to handle DNSSEC for signing the zone.

(Note: Unusual post, not about network routing or Linux systems. For purely technical audience of this blog, feel free to pass unless you have a young kid at home!)

In Sept 2021 - my wife and I were blessed with a baby boy (Avyukt). He is now around 1.6 years old. Since Oct 2022 he is wearing specs on his eyes. It’s not common (so far) to see babies that young with eye specs, this often brings up curiosity and questions across friends, family members and even strangers we meet outside on the road, on public transport etc. I am going to document in this blog post about his eyes, our interactions with the doctor and more.

Next month is SANOG 39 in Dhaka, Bangladesh. SANOG is a South Asian Network Operator Group event and a good place for meeting a number of ISPs, telecom players, Ops team members of content networks, internet exchanges etc. Besides attending the conference, I will be doing a workshop on Network Automation. It will be a four-day workshop covering Containers, Ansible, Gitlab CI/CD pipeline and REST APIs for automation in the workflow.

I visited Delhi earlier today and noticed latency from my phone on Jio 5G to my home (on an ISP behind Airtel) was just 20ms. It varied a bit (as one would expect on a wireless radio network) but 20ms is special because until now it was at least 80-90ms. After all, Jio and Airtel were not connected in Delhi NCR until now. There were sometimes jokes about them being connected at NIXI Noida but that never pushed any traffic because NIXI injects its route server AS24029 in the AS_PATH and while their direct PNIs (i.e Private Network Interconnect) in Mumbai & Chennai would have a direct (short) AS_PATH.

A rather long title but the post is about self-hosted open-source mesh VPN with IPv6 support and works with nodes behind CGNAT!
This will be a long post documenting the concept of mesh VPN, the problem it is solving as well as a working demo. If you are not planning to deploy it right away, you can skip the post after the “Configs and setup” section.

Problem

I am running a site-to-site VPN for a long time between various servers located far away from each other. Originally these used to be on OpenVPN and later I moved to wireguard. These were not mesh but rather in a linear topology. I would have a home node here in Rohtak connected to two different servers in Mumbai over two different ISPs (via policy-based routing), those two Mumbai nodes would maintain the site-to-site VPNs with a few servers in Europe & those servers further connect to a few servers in the US. This setup ensured private network connectivity with encryption so that I can have GitLab runners spread around based on available CPU load and those runners would speak to database/storage servers securely without having to deal with encryption on per project/app basis. This also gave me basic features like running cameras are home which feeds into the Frigate instance in Mumbai for motion detection-based recording, monitoring these cameras & other device uptime using the uptime-kuma instance in Ashburn etc.