A friend of mine buzzed me yesterday about his missing route objects. Later multiple other ISPs told the same story which triggered me to put this as a question on INNOG Mailing list. Many folks replied of missing route objects there and it seems to be limited to IRINN members only. I also asked the same question on APNIC mailing list and it was again confirmed about the issue.
Before I proceed further, here’s what it is all about.

What are route objects?

Route objects are basically entries created at an Internet Routing Registry (IRR). The key idea here is that an ISPs writes in a public register on what they want to do before they actually do. This gives a way for ISPs to transparently share their intentions in the global routing.

Example of a route object

anurag@devops01 ~> whois -h whois.radb.net 2402:b580::/32
route6:     2402:b580::/32
descr:      Anurag Bhatia R&D Pool
origin:     AS58901
mnt-by:     MAINT-AS58901
changed:    me@anuragbhatia.com 20200205
source:     ALTDB
anurag@devops01 ~>

E.g here I want to originate 2402:b580::/32 with my origin ASN 58901 and IRR which is giving this data is ALTDB. One can read more about route objects in my talk given at INNOG last year here. Another good detailed reading can be seen on FCIX here. Various operators build automated BGP routing filters based on the IRR data. So e.g anyone who is peering with me @ AS5891 can do:

anurag@devops01 ~> whois -h whois.radb.net !6as58901
A83
2402:b580:1::/48 2402:b580:3::/48 2402:b580:2::/48 2402:b580:4::/48 2402:b580::/32
C
anurag@devops01 ~>

This gives currently registered route objects and hence can be allowed on BGP filter.

What is IRINN?

IRINN or Indian Registry for Indian Names and Numbers is National Internet Registry (NIR) which functions closely with Regional Internet Registry (RIR) – APNIC in this region. While RIR is responsible for allocating IP addresses and AS numbers in the region, NIR is responsible for doing that within given National area. Broadly NIRs exists to facilitate and make it easy for local operators to interact directly with a local organisation in the local language, currency, timezone etc. Some other countries which have NIRs include China (CNNIC), Japan (JPNIC), Taiwan (TWNIC) etc. Indian NIR i.e IRINN is operated by NIXI (National Internet Exchange of India) and works closely with Ministry of Electronics & IT, Govt. of India. Indian ISPs, datacentres, web hosting companies, universities, banks and just anyone who needs IP address or ASN can get that from IRINN or APNIC.

Route object management at IRINN

For IRINN members, unfortunately, there’s no local IRR and basically all IRR entries go to the RIR (APNIC) run IRR. So instead of IRINN members managing those route objects directly with APNIC, they request those changes with IRINN and IRINN team does that manually in APNIC’s database. I covered this in detail in a previous blog post here.

Mass outage on 3rd and 4th July 2020

On 3rd July 2020 suddenly a large number of registered IRR route objects went missing from APNIC’s database where the maintainer was IRINN. There are no reported cases of any non-IRINN members with this issue. As far as I personally know – Tata Communications AS4755 is known to be filtering their Indian customers based on IRR data and regenerate filters every 6 hours. In the case of Airtel AS9498, I don’t think they automatically generate filters. Let’s see what was visible in the routing table.

A starting point can be RIPE RIS RRC03 because that is directly fed with full routing table from Tata Communications AS6453 which is technically direct upstream of Tata Communications AS4755 in India. Raw MRT dumps for RRC03 are here. It has around 12 routing table dumps (at time of me writing this blog post) for July 2020. Extracting only AS6453 data from it, that gives following total route announcements of AS6453.

An AS_PATH filter of ^6453 4755_ and counting total routes across 12 dumps in July gives me this.

Dump at 20200703.0800 has 17683 while dump after 8hrs of that at 20200703.1600 shows 17194. That’s a drop of 489 prefixes. That is likely not giving a complete picture since it’s 8hrs of time. Though Airtel, Tata or Jio do not feed Oregon route views collector. I am still curious about looking at Oregon Route-views dumps for _6453 4755_ as well as _9498_ and _9583_ dumps to check for any noticeable drop in the routes in the table. There are 34 visible dumps for July there (at a gap of 2hrs) between 2nd July – 4th July and that can give a more refined picture. Assuming if Tata Communications actually pulled off routes, those would be visible via various other networks who dump full table at Oregon route views.

Also, there is data available for withdrawal and announcements at every 15mins via Oregon route views. So I put code to read 300+ dumps for announcement and withdrawal data and plotted that.

So in the following graph, we find a drop of 600+ prefixes between 12:00 and 14:00 UTC on 3rd July 2020.

Looking at the diff of dump between 7/3/2020 12:00 and 7/3/2020 14:00 we can see 669 prefixes belonging to different ASNs which went offline and were not visible behind Tata Communications AS4755 at least for 4 hours.

Out of these 669 prefixes, I see 566 belong to MAINT-IN-IRINN maintainer and spread across 161 unique network ASNs. I have posted a list of prefixes, AS number and AS names below.

The issue in total took a much longer time as APNIC/IRINN restored the route object database and ISPs across the world re-generated their filters. Likely it took Tata Communications time between 14:00 to 18:00 UTC / 7:30 pm to 11:30 pm IST on 3rd July 2020 to disable IRR based filters.

Besides many ISPs it also impacted a few banks:
AS136252 – BANK OF BARODA
AS137108 – Bank Of India
AS59197 – Bharatiya Reserve Bank Note Mudran Pvt. Ltd
AS132994 – DENA BANK

I end this post with the hope that your ASN is not listed above.