Isp-Column

I have been in Ulaanbaatar, Mongolia for the last few days. This is my first travel to Mongolia and this far up in the North (except for previous travel to Russia in 2016 and some parts of Nordic areas in Europe). Geographically Mongolia is located between Russia (on the North side) and China (on the South side).

I am here for mnNOG 5 event. mnNOG is the Mongolian Network Operators Group. On Monday mnNOG conducted it’s 5th annual conference and it followed a five-day workshop. I am doing a workshop here on Network automation along with engineers from local networks. Mongolia is a landlocked country with no access to sea and hence no sub-sea cable. For the internet Mongolia relies on in-land fibre optic cables to connect to Russia and China. Though cables connect physically via Russia and China, I do not see Mongolian networks doing L3 termination in Russia or Mainland China. Instead, there is connectivity to Hong Kong, Singapore, Frankfurt etc. for the L3 connectivity. Interestingly due to it’s geographic location, a bit of China-Russia internet traffic exchange happens via Mongolia.

Earlier in March I visited Andaman & Nicobar Islands. The trip was purely personal as my wife happened to have been born there. These are Indian islands in the Bay of Bengal located in the South East of West Bengal and geographically quite near Myanmar and Thailand. The nearest large Indian cities on the mainland are Kolkata and Chennai.

In the initial part of the trip, we stayed in Swaraj Dweep (old name Havelock islands) and later in Port Blair. The place is isolated and has amazing natural beauty. It has one of the most beautiful beaches in the world (Radhanagar Beach). Tourism has grown nicely in Andaman & Nicobar islands in recent years and besides many other factors, one of that is a submarine cable!

I visited Delhi earlier today and noticed latency from my phone on Jio 5G to my home (on an ISP behind Airtel) was just 20ms. It varied a bit (as one would expect on a wireless radio network) but 20ms is special because until now it was at least 80-90ms. After all, Jio and Airtel were not connected in Delhi NCR until now. There were sometimes jokes about them being connected at NIXI Noida but that never pushed any traffic because NIXI injects its route server AS24029 in the AS_PATH and while their direct PNIs (i.e Private Network Interconnect) in Mumbai & Chennai would have a direct (short) AS_PATH.

A rather long title but the post is about self-hosted open-source mesh VPN with IPv6 support and works with nodes behind CGNAT!
This will be a long post documenting the concept of mesh VPN, the problem it is solving as well as a working demo. If you are not planning to deploy it right away, you can skip the post after the “Configs and setup” section.

Problem

I am running a site-to-site VPN for a long time between various servers located far away from each other. Originally these used to be on OpenVPN and later I moved to wireguard. These were not mesh but rather in a linear topology. I would have a home node here in Rohtak connected to two different servers in Mumbai over two different ISPs (via policy-based routing), those two Mumbai nodes would maintain the site-to-site VPNs with a few servers in Europe & those servers further connect to a few servers in the US. This setup ensured private network connectivity with encryption so that I can have GitLab runners spread around based on available CPU load and those runners would speak to database/storage servers securely without having to deal with encryption on per project/app basis. This also gave me basic features like running cameras are home which feeds into the Frigate instance in Mumbai for motion detection-based recording, monitoring these cameras & other device uptime using the uptime-kuma instance in Ashburn etc.

Last month I got access to Jio 5G like everyone else around in Haryana. They are running a beta program with uncapped data for now. Overall it works fine for usual stuff (web surfing on popular sites, YouTube videos, music streaming etc) but 464XLAT seems to be a little buggy in IPv4 hardcoded destinations. Initially it was giving quite a few issues but many of them seem to be fixed in last few days.

Last year had many interesting developments and one of that has been object storage. For those unaware, object storage is de-facto cloud storage which stores data as objects instead of file system architecture. This gives the option of simple plug-and-play horizontal scalability. It became popular when Amazon Web Services (AWS) launched S3. The idea was straightforward - pay-as-go storage with a few cents/GB/month charge to store data and a few cents/GB to egress data. No need to plan storage, no need to plan hard disk, storage servers, or rack capacity but a simple pay-as-you-go opex cost. Plus top tier cloud players do offer redundancy of data. The API replies with “success” on uploads only when data is replicated to multiple datacenters.

I was recently discussing with a friend Jio’s Fifa streaming issues. Considering PNI capacity challenges with other telcos, I wonder if they were serving FIFA streams out of their network or if it would be on some CDN like Akamai. As I was testing, I noticed a couple of megs of flow data with my provider’s local IP. Turns out that was a local Google GGC node in Rohtak and as I try to connect to it, it replies on HTTP port 80 and 443. The port 443 response is rather more interesting because while connecting to IP throws an error, it does give me the SSL certificate out of handshake and now I know it’s indeed Google! :)

Yesterday there was an article in the Indian paper Financial Express with the title “OTTs may have to pay access charge to telcos”.

Quoting a few points from the article:

• Social media intermediaries like WhatsApp, Facebook and Twitter, and over-the-top (OTT) players like Netflix, Prime Video and Disney+Hotstar may have to pay a carriage charge to telecom service providers
• Data, particularly video, comprises 70% of the overall traffic flow on telecom networks, and this would grow further with the rollout of 5G services
• Upon reference from the DoT, Trai is currently studying various possible models under which OTTs can be brought within the purview of some form of regulation
• According to sources, an interconnect regime is a must between OTTs and telcos because as 5G services grow, there would be immense data/ video load on networks, which may lead to them getting clogged or even crashing at times.

This concept of “OTTs must pay” is not new. This has been argued a few times in past. Exactly ten years ago in 2012 I wrote a blog post about Bharti Airtel expecting Google/YouTube to pay. At that time they could not convince OTTs to pay. Why is this renewed interest now? Well, that has to do with the first SK Telecom (South Kore telecom) Vs Netflix court case in South Korea where SK Telecom claimed that a large part of bandwidth utilization was because of Netflix and hence they should pay a “fair share” of their traffic which they lost. Soon around this multiple of large telecom monopolies in Europe started this discussion in their respective geography. Four of the top EU players - Deutsche Telekom, Orange, Vodafone and Telefonica are of opinion that OTTs should share the burden (news here). And hence Indian telcos possibly looking to renew this debate.

I was having this discussion with someone recently on possible software to manage an IXP. Lately, IXP Manager has become the de-facto choice for managing IX. It’s a good tool. Nick and INEX team has built a fantastic open-source tool. But I still feel it’s a bit overloaded for a small 1-2 DC IX operation.

If I have to set up a small to mid-size IX, I would rather do that with arouteserver instead of IXP Manager as I did in case of BharatIX in Mumbai (until it shutdown!). One of the problems with arouteserver is that it can be script intensive and one may need something around it to manage it for things like build config on clients.yml update, regularly update filters etc.

Next week SANOG (South Asia Network Operator Group) event will start in Kathmandu, Nepal. I will be instructing on a 4-day workshop on Network Automation with two fellow instructors. The idea of this workshop is to make fellow Ops / Network engineers familiar with concepts of Docker, Ansible, and Gitlab CI/CD pipeline and ultimately to make use of REST APIs to bind these all together.

This is the first time I am doing such a workshop and the content here is built from scratch. On the positive side, it gives good flexibility on content but the challenge is to stick on time. Since content is not tested before, there will always be a risk of going “too slow” or “too fast”. The goal by the end of the workshop is to ensure that attendees can build up event-driven automation. They should be able to set up a system where “if x happens” then “action y is triggered”. This can fit a wide variety of use cases.