anuragbhatia.com

13 Feb

Indian IPv6 deployment

I had calls with a couple of friends over this week and somehow discussion IPv6 deployment came up. “How much has been IPv6 deployment in India now in 2020” is a very interesting question. It’s often added with – “how much of my traffic will flow over IPv6 once it is enabled“?

Game of numbers

There is a drastic difference in IPv6 deployment depending on which statistic we are looking at here in India. There can be a bunch of factors based on which we can try to judge IPv6 deployment:

How many operators are offering IPv6 to end users?
How many end-users are on IPv6?
How’s the content available on IPv6 in terms of a number of IPv6 enabled websites?
How’s the content available on IPv6 in terms of traffic volume over IPv6?

First two and last two points are related and point towards from vertically opposite ends. (Call it good or bad) the fact of high centralisation. There has been an ongoing centralisation of mobile operators and in-country like ours they connect a very large number of end-users. The number of fixed-line networks has increased considerably but at the same time in proportion to a number of mobile users they user base growth has been much lower.

On the content side like everywhere else in the world, there’s a lot more centralisation of content. Many of my Indian ISP friends tell me that Google + Akamai + Microsoft + Netflix + Facebook + Cloudflare is way over 75% of their traffic. Think about it, that’s just 6 AS number out of 68000+ odd networks in the world as per BGP routing table. Thus by traffic profile, we are looking at 0.0014% networks serving 75% of content traffic. The reason for such centralisation is actually beyond network and more around the success of products of these organisations followed by factors like the winner (or top 3) gets it all in most of these domains.

For eyeball traffic APNIC IPv6 stats for India (source here) as well as Hurricane Electric’s IPv6 progress report (source here) give us some numbers:

Very few fixed-line operators are offering IPv6 but on the mobile side – a large number of mobile networks are offering IPv6. Jio was on IPv6 since launch and as traffic increased, Airtel as well as Vodafone + IDEA also significantly increased their deployment. On fixed line, it’s just Jio + ACT broadband with any sizable IPv6 footprint. BSNL + Airtel have virtually no deployment. There might be some other network in the list but it’s off the radar.
There are a lot more end users on IPv6 than despite the small number of networks offering it because of the large mobile user base. On Jio 80%+ user base, on Airtel, it’s 45%, on Vodafone & IDEA (merged company but still separate ASNs) it’s close to 50%. That’s the number of users who are connecting over IPv6 when given option of IPv4 and IPv6 as tested by APNIC. That number is huge!
Around 98.5% of TLDs (top-level domain names like .com, .net etc) are IPv6 enabled. For .com & .net domains, only 7% of the domains have an AAAA record (compared to ones having an A record). Most of the numbers are much lower if we look at the content side of IPv6 (ignoring the traffic volume).
If we look at the content players with IPv6 and include the traffic volume numbers then it’s way higher. It is estimated that globally somewhere between 20-30 top ASNs carry 90%+ traffic and almost all of those top 20 are IPv6 enabled.

What do all these numbers actually mean?

If you are an eyeball network in India and you deploy IPv6, you can expect way over 70-80% traffic (by volume) on IPv6.
If you are a content network/datacenter in India and application is targetting to home fixed-line / enterprise network, expect a rather low amount of IPv6 traffic but would be rapidly increasing as more fixed-line networks deploy.
If you are a content network/datacenter in India and content hosted at your end attracts mobile traffic, you can expect way over 50%-60% of that mobile traffic over IPv6.

Some additional reasons to consider deploying IPv6

In India, ISPs need to maintain carrier-grade NAT logs of the translations. If one is doing dual-stack, a large part of traffic will flow over IPv6 saving on those logging requirements.
For ISPs, it will save you from significant strain on CGNAT device.
For content network/webmaster/datacenter – IPv6 will help in delivering your traffic outside of (often) congested CGNAT paths.

Happy IPv6ing! 🙂

11 Feb

Skipping Netconf 2020, Internet shutdowns, Kashmir issue and more

A post to dump mind views. Hasgeek folks (who run RootConf conference in Bangalore + some other places around) seem to be in expanding mode. Besides RootConf which is a conference primarily for DevOps community, they are doing events on Fintech etc and now expanding to networks. I have been to and presented at multiple RootConfs on RIPE Atlas probes and BGP routing security. Both of these topics were from networking domain but closely touch the Sysadmins and thus probably made sense.

Hasgeek is a private organisation (to best of my knowledge) and it was actually nice to have someone working hard in doing events, getting the operational community together and thus I supported them. They usually do a good job at organising part of the event (may not be true on content).
Now with their latest Netconf 2020 in Bangalore, I am surprised at the topics they are putting for a networking conference. While I got a glimpse of those in private emails, but now that part of the agenda is public here, I can probably talk about it.

Here’s how Call for proposal page looks like (on 10th Feb 2020):

It’s just crazy and pathetic that they picked Internet shutdowns and Internet censorship for a considerable part of the conference. Furthermore, they are expecting network engineers along with Civil society groups, Tech and law researchers and activists and what not. That’s a too diversified set of people and doesn’t make any sense.

Why Internet shutdowns discussion right now is a bad idea?

(Warning: Might seem unrelated in the first glance, but read on…)

A considerable part of shutdown or censorship talks link directly to Kashmir or a little bit now on CAA. As someone who spends considerable time “on the internet”, I do very well understand the pain of shutdown in Kashmir but internet shutdown & censorship cannot be seen in isolation.
A large part of the narrative here goes along projecting as if things were perfectly fine in Kashmir before removal of article 370 (technically making it ineffective) on 5th Aug 2019 and everything after that is a problem. That is simply not true at all. Abrogation of article 370, as well as 35A which was a temporary, transitional and special provision to grant State of Jammu & Kashmir special status, was an excellent step towards fixing this old issue of disintegration.
There’s too little Govt. can do in conflict areas in terms of monitoring these days especially due to end to end encryption in apps like WhatsApp, Telegram and even basic SSL based security on most of the email providers. WhatsApp groups are a very effective way to communicate and it’s usually fine in non-conflict areas but for conflict areas, it’s a pain. I am going to quote from some non-Indian sources here (which apparently world loves):

Some related articles which point towards this issue:
London bridge terror attack was planned on WhatsApp,

Telegraph article – WhatsApp accused of giving terrorists ‘a secret place to hide’ as it refuses to hand over London attacker’s messages

“Trucks, Knives, Bombs, Whatever” Exploring Pro-Islamic State Instructional Material on Telegram

Vox article – Terrorists’ love for Telegram
I am overall in favour of an end to end encryption, SSL use everywhere etc but subject to the right of the state to ensure law & order. I do not think the internet is more critical than ensuring law & order in conflict areas especially in the country of 1.3 billion people.
Specifically, on J&K – we cannot have a system where people from Kashmir had all the rights across India but non-J&K people had limited rights in the region. It has to be retrospective and not just with Jammu & Kashmir but across all the states of the Union of India.
Jammu & Kashmir was getting 10% of funds while having just 1% population (source). Uttar Pradesh with 13% Indian population got only 8% of the central govt. funds. That doesn’t make sense and should not go on forever.
Many progressive Indian laws were not applicable in Jammu & Kashmir. In fact, if we look at most of the laws & orders passed before 5th Aug 2019, they used to start with a statement – Applicable to entire India except J&K. In the US it’s much stronger federal structure due to different states getting freedom at different times but that’s not the case with India.
Article 370 exempted the State from the complete applicability of the Constitution of India. The State was conferred with the power to have its own Constitution.
On education & employment rights as per Wikipedia article: The state government officials of Jammu and Kashmir have issued “permanent resident certificates”. However, these certificates differ by gender.
The certificates issued to females are marked “valid only till marriage”, while certificates for males have no such markings. If a woman married to an Indian outside of Kashmir, she was denied a new certificate.
According to Article 35A, a Kashmiri woman loses property rights if she marries a non-Kashmiri. Furthermore, her children are not considered ‘permanent residents’ if their father is a non-Kashmiri.
The exodus of Kashmiri pandits from Kashmir back in 1990 which was sadly almost ignored by most of India. Anywhere from 200,000 to 800,000 left Kashmir due to militancy Wikipedia article on the exodus here.
Quoting again from Wikipedia article here: According to official figures released in Jammu and Kashmir assembly, there were 3,400 disappearance cases and the conflict has left more than 47,000 people dead which also includes 7,000 police personnel as of July 2009. All this pretty much due to extreme radicalization in Kashmir.

and a lot more.

So what I am trying to put here is that Kashmir already had some serious issues. Just because many people do not like talking about it doesn’t really take those issues away. The problem is complicated and needs way more personal bandwidth than an individual can put in such technical conference discussions.

Thus I feel it’s ethically wrong to ignore larger problem & look at internet shutdowns in isolation. Furthermore adding a bunch of people from outside India & projecting an image that somehow “things are at a very dangerous stage now” (which folks from Hasgeek are trying to do). I am willing to discuss Internet shutdowns with anyone (outside of my working hours) as long as the other person (whether from India or outside) is willing to discuss the entire problem without being selective about what they pick.

The number problem

Another problem these days is over obsession with the numbers.
Take for example this article from The Washingtonpost: You’re more likely to be fatally crushed by furniture than killed by a terrorist. It claims: somewhere around 100 Americans will have died throughout the day in vehicular accidents.

A good friend of mine once quoted someone (~~I miss a person’s name now~~ Kenneth Anderson is a professor at Washington College of Law) who countered it: If 100 Americans unfortunately loose life in vehicular accidents today, it would be similar tomorrow, next month, next year etc. Unless something drastically changes, the rate of growth would be relatively linear if not declining (due to better safety, technology, the reaction of the system to prevent those etc). What about terrorism? If 10 people die today, can we be sure it would be a similar number next year? What if all efforts against radicalised folks are stopped since after all more people are dying due to vehicular accidents, the number will rise exponentially. Remember it cost around $400,000 and $500,000 for 9/11 attacks (source here) where unfortunately close to 3000 people lost their life. In the people who died was also Daniel Lewin – American–Israeli mathematician who was a co-founder of the world’s largest CDN network Akamai (Wikipedia article has more details).

Another example here is the Medianama website – which used to be a good website on tech & policy reporting before it’s founder Nikhil Pahwa decided to reduce it to internet censorship and anti-aadhar portal.

In April 2018 they reported: Internet shutdowns in India caused a loss of $3.04 billion between 2012-2017 (article here). Taking numbers at face value – so yes $3.04 billion loss. What about the other side? If the internet wasn’t shutdown would business be conducted as usual? All those POS transactions, e-commerce delivery, e-whatever delivery would be just going in mid of tensions? I think such numbers are as stupid as the (old) claim of $43 billion loss in the 2G scam.

Did scam happen? Yes, because terms of allocation were changed to favour certain people.

Did govt. actually lose $43 billion, well very likely not at all.

As an example say an iPhone which costs $1000. I somehow steal 10 units from Apple and try to re-sell it for $600 each. So actual value of iPhones: 10 x 1000 = $10000, value for which I sell: 10 x 600 = $6000. Thus a notional loss of $4000 to Apple. What if the stealing didn’t happen and the phone was priced originally $1000 for everyone. Would 10 people who bought it for $600 have still bought it?

It’s easy to just put random numbers in support of any argument. I live in a city in Haryana where most of parks have statues of soldiers who gave up their life serving the country. Guess where most of them lost their life? Name starts with “K”.

For now, Hasgeek has reduced itself to a propaganda machine to support a specific narrative. It’s their right to organise an event in a free country and it’s equally my right to stay away from it in the same free country. 🙂

End of mind dump!

06 Feb

Alternate to IRINN IRR manual entry / ALTDB

IRINN (Indian Registry for Internet Names and Numbers) is a NIR (National Internet Registry) for India operating under the APNIC RIR (Regional Internet Registry). IRINN is run and managed by NIXI. It’s a decent NIR and was set up in 2012. Indian organisations have the option to either maintain relation with APNIC or with IRINN.

A large number of small networks prefer IRINN because it’s annual charges are 25000 INR / $351 USD against APNIC’s membership fee which is over 2x of that.

There are a couple of key disadvantages of using IRINN:

The membership portal is rather limited and the entire process of creating, updating route objects as well as AS SETs via IRINN is rather a manual process. One raises a ticket and during working days IRINN processes those requests & updates the IRR entries. In backend entries just go in the APNIC’s IRR.
Process of creating/updating/maintaining RPKI ROAs is also rather manual.

As of now there’s not much one can do about #2 other than just following manual process by opening a ticket with IRINN but for #1 if you have a challenge that it’s rather slow to update/change because it is a manual process.

Introduction to IRR’s

IRR or Internet Routing Registry is just the public register for BGP related activities. Logic is that one first publishes what one wants to do and then does it. So I can say I want to originate 2402:b580::/32 from AS58901, publish it one of the publicly visible registers (known as IRRs) and then I actually announce the pool.

So who can run an IRR?

Anyone can! IRRd software is open source and one can use it to set up an IRR server. The old and most widely used IRRd is available here and the new version of IRRd (IRRd version 4) is here which is quite advanced, offers many excellent features. What makes IRRs run by APNIC, ARIN, RIPE, RADB etc popular is the fact that RADB mirrors them all and a large number of tools default to RADB for generating the filtering config. As per data available here, RADB mirrors 23 other IRRs besides serving from its own database. So one can get entry to either of these available IRRs based on relation one has with them for creating the route objects or AS SETs and it just works.

ALTDB

Many people know of RADB which is run by merit. It’s a paid option and many networks use that for maintaining route objects. There’s also a free IRR called ATLDB. It’s free to create an account on it and approval of account is a manual process but once approved, creation/updation of route objects as well as AS SETs is all automated and works via email with a specific syntax. Unfortunately, there’s not much on the ALTDB website except a whois lookup tool, however, if you want to read in detail about IRR as well as using ALTDB, the Fremont Cabal Internet Exchange has an excellent guide on that (here).

The basic logic here is to do following: Create a maintainer object > Define routing policy of the ASN (peers, transit, downstream etc) > Create AS SET (if you have BGP downstream customers) > Create route objects for your IPv4 and IPv6 prefixes > Publish your ASSET on PeeringDB. Again one can follow the detailed steps given on the FCIX website.

Here’s an example of it in action for a test prefix of the pool I use:
2402:b580::/32

As of now, there’s no corresponding route object with it. Here’s a quick check on RADB:

anurag@devops01 ~> whois -h whois.radb.net 2402:b580::/32
%  No entries found for the selected source(s).
anurag@devops01 ~>

As of now, there’s no route object for it. Let’s say I want to start originating from AS58901, so I will put following in a plain text mail and send it to ALTDB email ID (auto-dbm@altdb.net) which processes these requests automatically.

route6:     2402:b580::/32
descr:      Anurag Bhatia R&D Network IPv6 Pool
origin:     AS58901
mnt-by:     MAINT-AS58901
changed:    me@anuragbhatia.com 20200205
source:     ALTDB

It’s important to send email in cleartext. One can check how to do that in a specific email client or web interface one uses. For Gmail the option is here:

I get following in reply from ALTDB:

And now query to RADB gives us:

anurag@devops01 ~> whois -h whois.radb.net 2402:b580::/32
route6:     2402:b580::/32
descr:      Anurag Bhatia R&D Pool
origin:     AS58901
mnt-by:     MAINT-AS58901
changed:    me@anuragbhatia.com 20200205
source:     ALTDB
anurag@devops01 ~>

And here goes the example of creating AS SET: AS-ANURAG. I sent following to the ATLDB:

as-set:     AS-ANURAG
descr:      Anurag Bhatia's AS SET
members:    AS58901
mnt-by:     MAINT-AS58901
changed:    me@anuragbhatia.com 20200205
source:     ALTDB

Upon confirmation, I get when I query RADB:

anurag@devops01 ~> whois -h whois.radb.net AS-ANURAG
as-set:     AS-ANURAG
descr:      Anurag Bhatia's AS SET
members:    AS58901
mnt-by:     MAINT-AS58901
changed:    me@anuragbhatia.com 20200205
source:     ALTDB
anurag@devops01 ~>

Remember if you happen to use ALTDB, make sure to ask IRINN to delete your route objects after you have successfully created them on ALTDB. Duplicate entries just add to the junk in IRR.