Challenges of building a world class NOG

09-05-2022

This is my first and probably going to be the only blog post talking about a NOG i.e a Network Operator Group. I usually do not talk about that here because NOG is supposed to be facilitating the work of the network operator community and in itself should not be a topic to focus on or talk about. As a matter of fact, I find it a bit irritating when NOGs are presented as the end goal when parties are thrown in just to celebrate certain age of NOG when hours are wasted just telling stories about NOG planning etc.

So why am I making an exception here and talking about it? It’s because we just recently did INNOG 5 and that was my last INNOG as a Program Committee member and more realistically as being the “default route” for the Indian Network Operators Group. I am going to document what worked, what did not work, possible things we did right and things where we messed up.

What is a NOG?

Before proceeding further it’s important to define NOG, and what it is supposed to do and then match our work at INNOG against those parameters to see whether or not it worked. As the internet has grown massively over the last few years, in the delivery of the internet there is a massive ecosystem involved. While the list would be endless but broadly from a network operations perspective following play a very important role in the ecosystem:

Network operators from small to large
Content players like Google, Facebook, Netflix etc
Content Delivery Networks (CDNs) like Akamai, Cloudflare, Fastly, Edgecast etc
Cloud players from Hyperscalars like AWS, GCP and Azure to mid-sized players like Digital Ocean, Vultr, Linode etc
Internet Exchange Operators
Root & ccTLD DNS operators
Hardware vendors from optical to IP
Software vendors
System integrators
Regional Internet Registries

So the idea of NOG is to get them on a common platform to discuss operational issues. This includes things like operating a mailing list (to report & discuss any operational issue, ask a question), doing a periodic conference, doing training workshops etc. This concept was originally pioneered by NANOG - North America Network Operator Group which is the oldest NOG. Besides NOG covering these topics, there’s a fair amount of overlap here in RIR run meetings including that of APNIC, RIPE, ARIN etc (besides covering RIR specific discussions).

INNOG

History

While most NOGs are now over 20-30 years old, we started a bit late in India and likely INNOG is the youngest of any NOG for a country size of ours. Here’s the age of some of the NOGs for comparison:

NOG	Region	*Age (in years)*
NANOG	North America	35*
INNOG	India	4
UKNOF	United Kingdom	17
SANOG	South Asia	19
APRICOT	Asia + Oceania	26
BDNOG	Bangladesh	8
SGNOG	Singapore	11
NPNOG	Nepal	6
LKNOG	Sri Lanka	5
AusNOG	Australia	15
NZNOG	New Zealand	4
MENOG	Middle East	15
NLNOG	Netherlands	8

*NANOG as a community started in 1987, though it did change its entity, charter etc a few times over this long period (reference).

INNOG history starts with changes in SANOG. SANOG is South Asia NOG and was primarily created to cover the South Asian region - India, Nepal, Sri Lanka, Bhutan, Pakistan, Afghanistan etc. They usually do this with help of local partners for logistics. Sometimes around 2013 it was decided to have at least one SANOG event in India every year and other would rotate across the rest of South Asia. This was followed until 2017 with SANOG’s Indian partner which is the ISP Association of India (ISPAI). This setup of one SANOG every year somewhat satisfied the need for having a dedicated Indian NOG. As each NOG happened, there was friction between folks running SANOG & ISPAI. Ultimately it was decided that SANOG will stop sticking to India for their event once a year and would happen in India as it follows its respective sequence. A number of things lead to this sort of “breakup” but one key element one can find is that while SANOG always felt Indian SANOGs as less exciting, and less successful (putting in a polite way), Indian hosts on the other hand wore them as a badge of success. So clearly there was a difference in opinion.

Formation of Indian NOG

Somewhere around this time frame of 2014-15 or so, ISPAI announced the formation of the Indian NOG as one of the events. Not much happened beyond that announcement. Fast forward to APRICOT 2018 in Kathmandu, Nepal I was interacting with Raunak Maheshwari (ExtremeIX), Ajai Kumar (MumbaiCH / DECIX-India) and Sunny (APNIC) about having an Indian NOG. We all were excited, ready to take this over and build a NOG. As we came back, we looked at the most important question “how to do it”. Building a NOG website, and hosting a mailing list is one thing but doing an actual conference every year seemed like a tricky problem. There wasn’t a clear answer to who will burn time in making a legal entity, did we actually need a legal entity, or who will manage the ground logistics of organizing an event where potentially hundreds of folks would come. Ultimately nothing happened. Then at one point, my interaction started with ISPAI and it turned out that we were looking for each other. They were comfortable with ground logistics, knew Indian regulations well, and did SANOG in past. I had a very basic experience of handling programs as around those days I was volunteering on the program committee of a couple of events. Thus we decided to work together and make it happen. We got Mr Srinath Beldona as Program Committee chair to lead the first event INNOG 1. Mr Srinath is an ex-Cisco, he spent 15 years of his carrier at Cisco, and had the experience of working at WIPRO, worked as an APNIC trainer on the contract for a while. He knew how NOGs worked, attended many of them in recent times and was a fit for the PC chair.

INNOG 1 happened with some minor glitches. It was an onsite event in 2018 at India Habitat Center in New Delhi. With some glitches, it just went. It was nowhere close to perfect but was ok enough as a starting event. Fast forward now, we just had INNOG 5. While it says number 5 it was technically 4th INNOG because we skipped number “4” as INNOG 4 was cancelled/postponed twice causing a bit of confusion.

What worked so far?

Few things which worked overall fine so far:

Excluding INNOG 4 multiple cancellations, we were fairly regular with yearly NOG event
We were one of the first NOGs in the region to do an online event as Covid19 hit during 2020
Our PC was able to convince good folks for the keynote as well as to be part of the panel members and also present. Over these years we had keynotes from Tata Communications, Jio, NTT Netmagic, Burt Hubert (a brand in himself!). We had presentations & panel discussions with members from Google, Facebook, Akamai, Cloudflare, Bharti Airtel, Jio, etc. So a nice mix of local as well as international organisations
We started with random short Zoom sessions on various technologies. Initially, I did those for DNS 101, Routing 101, Docker and as we did a few of those, we formalised those and named them INNOG Tech sessions and invited external speakers from Netflix, Google, Fastnetmon etc. This of unique of its kind and had good participation
I was able to utilise INNOG as a platform for sessions as well as the mailing list to push for RPKI ROA coverage. The end result here was a push from 12% all the way to 59% valid ROA coverage for India. As of now, we have the largest signed & valid address space in this region. (live data here)

What did not work?

Despite some of the above good things, a number of things did not work and failed quite badly and literally blew up on our faces.

Five years down the line of the formation of INNOG, we still do not have a functional Organising Committee. Originally it was ISPAI and the idea was that both the organising committee and Program Committee will grow. Fast forward now, the organising committee is still just ISPAI with the majority of members not spending any time in planning & organising INNOG.
The number of people participating in INNOG stays at one of the all-time lows. Take for instance this week’s INNOG 5 we had 350 people registering for it, while only 110 were online during peak. Average numbers would be even lower. That’s in the context of fact that we have over 2400 active ASNs in India visible in the routing table.
The involvement of large Indian players all the way from mobile operators (Airtel/Jio/VI) to backbone/wholesale players (Tata Comm/Sify etc) to datacenter operators (Netmagic, CtrlS, Equinix/GPX India is overall very low to negligible. One would expect a large part of their IP team to be part of INNOG but practically it’s either zero or maybe one at max.
We had almost zero presence of various outsourcing companies whose engineers ultimately work on these projects for Indian as well as outside telcos.
Activity on the INNOG mailing list was also very low. I think there were in total half a dozen of threads discussing ongoing issues or requesting suggestions. Rest all were just INNOG related announcements. This issue wasn’t INNOG specific but more of an Asian NOG specific problem. Mailing lists of APNIC-talk or SANOG or SGNOG or BDNOG are usually silent as well. Culturally people don’t want to ask for help in public or comment about things in public. I can count so many times when I posted a question there and a majority of people replied back to me offlist even when a list reply wouldn’t have hurt them (but would have helped others).
Whether we had a “teamwork” or not is a grey area. If we count 3-5 folks as a team then yes we had a good team in the backend but that is when we have 5 folks in the organising committee, 20 folks in the program committee and so on. Thus (my personal opinion here) we failed to create an environment where the team works together to solve a problem.
There used to be a kind of push from organisers to allow for sponsored talks. We were able to avoid it for the most part excluding one talk during INNOG 2. But since that push is there, eventually that would happen.
We discussed many times but ultimately failed to put localised content for the most part. We had panel discussions on local topics from IXP security in Indian exchanges, IXPs in smaller cities etc but the delivery was always in English. During the “IPv6 on Mikrotik” session last week I could make out how much extra stress attendees were putting in asking questions in English. This was when we had trainers who could speak Hindi, Punjabi, Marathi, Kannada etc. This kept our reach very limited.
We failed to have a good website. Though there was a “team” which managed the website though 90% time I was doing it alone. Website issue is not a technical issue but rather a time bottleneck in handling it.

Problem with the “free model”

INNOG 1 and 2 were onsite and were paid events to attend both workshop and the conference. INNOG 3 & 5 were online and free. Now in high-insight, I would say the free model somewhat badly breaks in the Indian context especially online. We had a lot of people “showing interest” and filling out forms to attend workshops as well as conferences but very few actually showed up. On the other side similarly, we had many people agreeing to volunteer for things like managing programs to websites to instructing workshops and that did not go well. My fellow countrymen somehow do not take “volunteering” seriously. In the end, all we have is a commitment with no follow-through, people not showing up at meetings or doing things on time. I think we should have paid at least the workshop instructors. I would have still done it for free but that way we can “expect and demand” from others. This is very different from say my friends in Bangladesh doing BDNOG. In BDNOG to JANOG one would find real volunteers willing to put their time. This would include a diverse set of volunteers from small to large operators. SGNOG (Singapore NOG) also has a similar system where volunteers work.

During INNOG 5 we had a presentation from Martin Pels from NLNOG (Netherlands NOG) about how around 6 volunteers go together to volunteer and run a “ring” of 500+ servers. And here I personally struggled to get that many folks to just ensure that they review submissions. Similarly, INNOG itself isn’t a strong enough brand to attract program committee members and thus we should have planned some sort of incentive for them instead of expecting their free time.

The idea of IXP run by volunteers

Once a popular datacenter in Mumbai wanted to start an IXP. A common friend between us contacted me and thew the idea of me leaving my job at Hurricane Electric to start an IXP for them. I rejected the idea stating that there isn’t much to do at an IXP. Once you set it up, technically it does not need that many skills to run an IP exchange. I instead offered them to volunteer to set it up. The datacenter instantly rejected the idea stating that they cannot promote IXP as a product with SLA backing when IX itself is relying on “volunteers”. I didn’t agree with them at that time but now I totally agree with their thinking. They probably have seen it from their experience of how “volunteer-run” stuff barely survives in India.

Thus point I am trying to make here with free model issues both in NOG & IXP context is that it ultimately doesn’t work in our region. It’s hard to know why but maybe because people are already stretched at their jobs way more than in the West.

Regulation challenges

One question which multiple PC members, friends and attendees have asked me over time is why we even went with ISPAI? It’s a valid question since the ISP Association of India is more of a lobby group and isn’t really into the operational technology side of ISPs.

The reason is hard regulation. India is a heavily regulated market in many aspects and regulations are often very outdated, complex to follow and often do not make any sense. This discussion of Kushal Mehra with Gautam Chikermane (ORF) and Rishi Agrawal covers this in great detail. If INNOG was done at a low cost where some of us just bear the cost, it would be easy & possible. But as soon as one involves the cost of the venue, food & various other things, the cost quickly goes up and then you need sponsors. As soon as getting money from sponsors is concerned, one needs a legal entity to take that cost, do financing, book management, audits and more. The majority of tech geeks in the Program Committee would just prefer to stay away from it. This is why we thought ISPAI doing this for INNOG was a good idea. In high insight, it turned out to be a bad idea.

Out from INNOG Planning

As INNOG 5 ended, I have come to the decision not to proceed anymore with the planning of the conference. It felt like a tiring exhaustive exercise of building something which isn’t exactly in as much demand as we thought it would be. There is a lack of commitment from both sides i.e people planning the event and people attending the event. This lack of understanding of reality tends to happen when you are in your own eco-chambers. When you speak to people from ISPAI, SANOG, APNIC etc and everyone will tell you how much is the demand while that just won’t co-relate with ground realities.

I can make use of my personal time & resources in doing something more productive. I am mentoring fellows as part of the INNOG 5 fellowship program and will continue doing so. Might build some public-facing tools like RPKI tracker, do some small workshops etc.

Lesson learnt time to move on!