Isp-Column

Last year had many interesting developments and one of that has been object storage. For those unaware, object storage is de-facto cloud storage which stores data as objects instead of file system architecture. This gives the option of simple plug-and-play horizontal scalability. It became popular when Amazon Web Services (AWS) launched S3. The idea was straightforward - pay-as-go storage with a few cents/GB/month charge to store data and a few cents/GB to egress data. No need to plan storage, no need to plan hard disk, storage servers, or rack capacity but a simple pay-as-you-go opex cost. Plus top tier cloud players do offer redundancy of data. The API replies with “success” on uploads only when data is replicated to multiple datacenters.

I was recently discussing with a friend Jio’s Fifa streaming issues. Considering PNI capacity challenges with other telcos, I wonder if they were serving FIFA streams out of their network or if it would be on some CDN like Akamai. As I was testing, I noticed a couple of megs of flow data with my provider’s local IP. Turns out that was a local Google GGC node in Rohtak and as I try to connect to it, it replies on HTTP port 80 and 443. The port 443 response is rather more interesting because while connecting to IP throws an error, it does give me the SSL certificate out of handshake and now I know it’s indeed Google! :)

Yesterday there was an article in the Indian paper Financial Express with the title “OTTs may have to pay access charge to telcos”.

Quoting a few points from the article:

• Social media intermediaries like WhatsApp, Facebook and Twitter, and over-the-top (OTT) players like Netflix, Prime Video and Disney+Hotstar may have to pay a carriage charge to telecom service providers
• Data, particularly video, comprises 70% of the overall traffic flow on telecom networks, and this would grow further with the rollout of 5G services
• Upon reference from the DoT, Trai is currently studying various possible models under which OTTs can be brought within the purview of some form of regulation
• According to sources, an interconnect regime is a must between OTTs and telcos because as 5G services grow, there would be immense data/ video load on networks, which may lead to them getting clogged or even crashing at times.

This concept of “OTTs must pay” is not new. This has been argued a few times in past. Exactly ten years ago in 2012 I wrote a blog post about Bharti Airtel expecting Google/YouTube to pay. At that time they could not convince OTTs to pay. Why is this renewed interest now? Well, that has to do with the first SK Telecom (South Kore telecom) Vs Netflix court case in South Korea where SK Telecom claimed that a large part of bandwidth utilization was because of Netflix and hence they should pay a “fair share” of their traffic which they lost. Soon around this multiple of large telecom monopolies in Europe started this discussion in their respective geography. Four of the top EU players - Deutsche Telekom, Orange, Vodafone and Telefonica are of opinion that OTTs should share the burden (news here). And hence Indian telcos possibly looking to renew this debate.

I was having this discussion with someone recently on possible software to manage an IXP. Lately, IXP Manager has become the de-facto choice for managing IX. It’s a good tool. Nick and INEX team has built a fantastic open-source tool. But I still feel it’s a bit overloaded for a small 1-2 DC IX operation.

If I have to set up a small to mid-size IX, I would rather do that with arouteserver instead of IXP Manager as I did in case of BharatIX in Mumbai (until it shutdown!). One of the problems with arouteserver is that it can be script intensive and one may need something around it to manage it for things like build config on clients.yml update, regularly update filters etc.

Next week SANOG (South Asia Network Operator Group) event will start in Kathmandu, Nepal. I will be instructing on a 4-day workshop on Network Automation with two fellow instructors. The idea of this workshop is to make fellow Ops / Network engineers familiar with concepts of Docker, Ansible, and Gitlab CI/CD pipeline and ultimately to make use of REST APIs to bind these all together.

This is the first time I am doing such a workshop and the content here is built from scratch. On the positive side, it gives good flexibility on content but the challenge is to stick on time. Since content is not tested before, there will always be a risk of going “too slow” or “too fast”. The goal by the end of the workshop is to ensure that attendees can build up event-driven automation. They should be able to set up a system where “if x happens” then “action y is triggered”. This can fit a wide variety of use cases.

As returning readers of this blog would be aware - I found a trick to find Facebook caching servers around the world during the APRICOT 2018 hackathon. Since then I am running my code again every year to see the changes and publish this report.

Previous reports

1. March 2018 here
2. Nov 2019 here
3. April 2021 here

Facebook knows!

Back in 2019, I was in San Francisco, California for NANOG 75. While roaming around in the lobby, someone read the NANOG card hanging around my neck and greeted me. His 2nd line after greeting was “Oh I know that name, you are the guy who mapped our caching nodes” and we both laughed. I must say this specific category of the post has brought some attention around.

In a recent Network AF podcast Avi Freedman (Kentik) joked with the guest about how he finds who is transit free / tier 1 network. He said, “I ask everyone who they think is a tier 1 network. Everyone includes their own name + other names”. Next, he ignores the self-nomination & looks at the common list to find who actually is a tier 1 network. This is funny, intuitive and gives some clue.

CERT-IN i.e Computer Emergency Response Team, India issued new guidelines on 28th April. Guidelines essentially ask those VPN providers to keep a log of customer details, their IP addresses, emails, phone numbers etc and maintain that log for at least 5 years. The detailed notification is here.

This not only extends to VPN players but also to datacenters, VPS, cloud service providers etc. I can understand the problem they are trying to solve as most criminal activities are hidden behind VPN players and investigating agencies just hit a dead end as they see the WAN IP of a VPN player.

In the early phase of Russia - Ukraine war, Ukraine made a strange request to ICANN. They asked ICANN to remove .ru (Russian ccTLD) from the root DNS servers, revoke SSL certs for .ru and shut down root DNS servers hosted in Russia.

Here are the three requests they made:

Complete letter is here (and original source is here). This is going to be one of few notable cases where critical internet infrastructure is being weaponised. ICANN declined the request for good. Due to my limited understanding of Russia, Ukraine, US, EU, NATO etc I am not going to comment on the conflict itself. But coming to the critical infrastructure part - this reminds me of my earlier blog post on Doomsday and DNS resolution.

I am in Bangalore for two days. While there are many things packed into these two days short schedule, one of the most exciting ones is Google Global Network India Innovation Summit. While Google has presented across various events in past talking about their AS15169 backbone, this is the first summit where they are covering it in detail and that too with the Indian context!

Must say that I find AS15169 quite fascinating on the BGP side of things. A massive network which follows “cold potato” routing i.e keeping the majority of traffic over IGP over larger locations, terminating BGP sessions on the virtual appliance with SDN backing, a pretty robust failover design with BGP + DNS taking care of server(s) and even entire PoP failing. I blogged about them back in 2020 here. So this should be fun!