Isp-Column

I visited Delhi earlier today and noticed latency from my phone on Jio 5G to my home (on an ISP behind Airtel) was just 20ms. It varied a bit (as one would expect on a wireless radio network) but 20ms is special because until now it was at least 80-90ms. After all, Jio and Airtel were not connected in Delhi NCR until now. There were sometimes jokes about them being connected at NIXI Noida but that never pushed any traffic because NIXI injects its route server AS24029 in the AS_PATH and while their direct PNIs (i.e Private Network Interconnect) in Mumbai & Chennai would have a direct (short) AS_PATH.

A rather long title but the post is about self-hosted open-source mesh VPN with IPv6 support and works with nodes behind CGNAT!
This will be a long post documenting the concept of mesh VPN, the problem it is solving as well as a working demo. If you are not planning to deploy it right away, you can skip the post after the “Configs and setup” section.

Problem

I am running a site-to-site VPN for a long time between various servers located far away from each other. Originally these used to be on OpenVPN and later I moved to wireguard. These were not mesh but rather in a linear topology. I would have a home node here in Rohtak connected to two different servers in Mumbai over two different ISPs (via policy-based routing), those two Mumbai nodes would maintain the site-to-site VPNs with a few servers in Europe & those servers further connect to a few servers in the US. This setup ensured private network connectivity with encryption so that I can have GitLab runners spread around based on available CPU load and those runners would speak to database/storage servers securely without having to deal with encryption on per project/app basis. This also gave me basic features like running cameras are home which feeds into the Frigate instance in Mumbai for motion detection-based recording, monitoring these cameras & other device uptime using the uptime-kuma instance in Ashburn etc.

Last month I got access to Jio 5G like everyone else around in Haryana. They are running a beta program with uncapped data for now. Overall it works fine for usual stuff (web surfing on popular sites, YouTube videos, music streaming etc) but 464XLAT seems to be a little buggy in IPv4 hardcoded destinations. Initially it was giving quite a few issues but many of them seem to be fixed in last few days.

Last year had many interesting developments and one of that has been object storage. For those unaware, object storage is de-facto cloud storage which stores data as objects instead of file system architecture. This gives the option of simple plug-and-play horizontal scalability. It became popular when Amazon Web Services (AWS) launched S3. The idea was straightforward - pay-as-go storage with a few cents/GB/month charge to store data and a few cents/GB to egress data. No need to plan storage, no need to plan hard disk, storage servers, or rack capacity but a simple pay-as-you-go opex cost. Plus top tier cloud players do offer redundancy of data. The API replies with “success” on uploads only when data is replicated to multiple datacenters.

I was recently discussing with a friend Jio’s Fifa streaming issues. Considering PNI capacity challenges with other telcos, I wonder if they were serving FIFA streams out of their network or if it would be on some CDN like Akamai. As I was testing, I noticed a couple of megs of flow data with my provider’s local IP. Turns out that was a local Google GGC node in Rohtak and as I try to connect to it, it replies on HTTP port 80 and 443. The port 443 response is rather more interesting because while connecting to IP throws an error, it does give me the SSL certificate out of handshake and now I know it’s indeed Google! :)

Yesterday there was an article in the Indian paper Financial Express with the title “OTTs may have to pay access charge to telcos”.

Quoting a few points from the article:

• Social media intermediaries like WhatsApp, Facebook and Twitter, and over-the-top (OTT) players like Netflix, Prime Video and Disney+Hotstar may have to pay a carriage charge to telecom service providers
• Data, particularly video, comprises 70% of the overall traffic flow on telecom networks, and this would grow further with the rollout of 5G services
• Upon reference from the DoT, Trai is currently studying various possible models under which OTTs can be brought within the purview of some form of regulation
• According to sources, an interconnect regime is a must between OTTs and telcos because as 5G services grow, there would be immense data/ video load on networks, which may lead to them getting clogged or even crashing at times.

This concept of “OTTs must pay” is not new. This has been argued a few times in past. Exactly ten years ago in 2012 I wrote a blog post about Bharti Airtel expecting Google/YouTube to pay. At that time they could not convince OTTs to pay. Why is this renewed interest now? Well, that has to do with the first SK Telecom (South Kore telecom) Vs Netflix court case in South Korea where SK Telecom claimed that a large part of bandwidth utilization was because of Netflix and hence they should pay a “fair share” of their traffic which they lost. Soon around this multiple of large telecom monopolies in Europe started this discussion in their respective geography. Four of the top EU players - Deutsche Telekom, Orange, Vodafone and Telefonica are of opinion that OTTs should share the burden (news here). And hence Indian telcos possibly looking to renew this debate.

I was having this discussion with someone recently on possible software to manage an IXP. Lately, IXP Manager has become the de-facto choice for managing IX. It’s a good tool. Nick and INEX team has built a fantastic open-source tool. But I still feel it’s a bit overloaded for a small 1-2 DC IX operation.

If I have to set up a small to mid-size IX, I would rather do that with arouteserver instead of IXP Manager as I did in case of BharatIX in Mumbai (until it shutdown!). One of the problems with arouteserver is that it can be script intensive and one may need something around it to manage it for things like build config on clients.yml update, regularly update filters etc.

Next week SANOG (South Asia Network Operator Group) event will start in Kathmandu, Nepal. I will be instructing on a 4-day workshop on Network Automation with two fellow instructors. The idea of this workshop is to make fellow Ops / Network engineers familiar with concepts of Docker, Ansible, and Gitlab CI/CD pipeline and ultimately to make use of REST APIs to bind these all together.

This is the first time I am doing such a workshop and the content here is built from scratch. On the positive side, it gives good flexibility on content but the challenge is to stick on time. Since content is not tested before, there will always be a risk of going “too slow” or “too fast”. The goal by the end of the workshop is to ensure that attendees can build up event-driven automation. They should be able to set up a system where “if x happens” then “action y is triggered”. This can fit a wide variety of use cases.

As returning readers of this blog would be aware - I found a trick to find Facebook caching servers around the world during the APRICOT 2018 hackathon. Since then I am running my code again every year to see the changes and publish this report.

Previous reports

1. March 2018 here
2. Nov 2019 here
3. April 2021 here

Facebook knows!

Back in 2019, I was in San Francisco, California for NANOG 75. While roaming around in the lobby, someone read the NANOG card hanging around my neck and greeted me. His 2nd line after greeting was “Oh I know that name, you are the guy who mapped our caching nodes” and we both laughed. I must say this specific category of the post has brought some attention around.

In a recent Network AF podcast Avi Freedman (Kentik) joked with the guest about how he finds who is transit free / tier 1 network. He said, “I ask everyone who they think is a tier 1 network. Everyone includes their own name + other names”. Next, he ignores the self-nomination & looks at the common list to find who actually is a tier 1 network. This is funny, intuitive and gives some clue.