24 Mar

Internet Exchanges – Place where the networks interconnect!

Earlier this month I got an opportunity to be part of IXP workshop in Kolkata. It was a 3-day event organised by ISOC Kolkata and supported by APNIC. There was also a workshop on DNSSEC and Champika Wijayatunga (from ICANN) was the instructor along with Anand Raje. It was a nice event and I come to know of other interesting projects ISOC Kolkata is doing like Indian IETF capacity building program apart from the IXP they are running in Kolkata. Mr Anupam Aggarwal and Anand showed the IX and it looks very good. I think it’s the first and only IX I know in India which is a real IX with proper policy. It’s an IX by a non-for-profit group, allows anyone to connect, a real layer 2 IX and welcomes anyone including ISPs, content players and root DNS servers. Presently IIFON-IX in Kolkata has few member ISPs besides the L root from ICANN and one of Verisign gTLD nodes (which host zones for .com, .net etc). I also saw a rack with some of Akamai CDN servers. This brings decent content right there. IX’es play an extremely important part of current internet infrastructure ecosystem. It’s very likely that content of this blog is travelling from my server to your browser from an Internet Exchange. 🙂

IIFON-IX Kolkata


I hope IIFON-IX does well in terms of becoming a key point for internet traffic exchange in Kolkata. There are a number of benefits of a good IX. Some of them are:

  1. Helps in reducing overall cost of bandwidth by keeping it local
  2. Helps in giving ultra-low latency interconnections among members
  3. Builds resilience across the internet by removing the dependency on transit players for local traffic and a lot more!

Besides IIFON-IX, I have been to BDIX in Dhaka, HKIX in Hong Kong, INEX IX in Dublin, LoNAP in London, Netnod in Stockholm etc. One thing common across all these exchanges is local involvement, usually a non-for-profit model and a collective community effort to make internet infrastructure better.

INEX – Dublin


Nick Hilliard (CTO at INEX) was kind enough to give me ride to one DCs hosting the IX.
Here’s a pic with Nick:

It was very interesting to see the use of passive DWDM MUX in the IX to connect between INEX PoPs across Dublin.

The one in this picture is an MRV MUX. Each port in passive MUX has it’s own wavelength and requires either pre-tuned or tunable SFP for that specific port. Such MUX can help to increase capacity on a pair of the strand by offering 40 channels and each can carry up to 10Gbps thus a capacity of 400Gbps for shorter distances (20kms or so).
The channels numbers mentioned on the port correspond to ITU Channel table and here’s the frequency wise table of actual channel number:


LoNAP – London

During my trip to UK, my good friend Andy Davidson helped in arranging a tour of LoNAP and Robert Lister was kind enough to give me a nice detailed tour of the IX fabric.


It was nice to see 10Gig flavours of Bidi SFPs i.e SFPs which do 10Gbps on a single strand of fibre. These are slightly different in terms of wavelengths as they run on 1270/1330 pair (opposite on the other side) while 1G Bidi optics runs on 1310/1490 pair.

I also saw a QSFP-LR4 which can do 100Gbps up to 10kms on a pair of fibre. 100Gbps or 100,000,000,000 bits per second or 100,000,000,000 zeros and ones flowing from fibre every second!

This runs on 1310mn and I don’t think there’s any Bi-Di option yet for 100G. There was ongoing 100G deployment at LoNAP during last year summers when I visited. One can spot an Arista switch with 100G ports.

And here’s a picture with amazing Rob after the IX-tour ended. 🙂


BDIX – Dhaka

Some pictures from Bangladesh Internet Exchange in Dhaka.

Mr Kabir from BDIX gave me an excellent tour of BDIX during my visit to Bangladesh back in 2015. You can see a lot of 1G fibre media converters in the rack. There’s quite a lot of fibre in Dhaka and cost of media converter is low and hence using them instead of the optical switch makes commercial sense though that’s changing fast. IX runs it its own dedicated building floor. IX has over 78 members.
Here’s a picture with Mr Kabir 🙂

Where should IX fabric be located?  
This is one of key question which comes in the mind of many engineers when talking of IX. Should IX be in a separate dedicated building or it should be rather in a commercial datacenter? The answer is simply that it depends. In larger cities in US or Europe, there are quite a lot of datacenters which are well connected. There’s reasonable competitive fibre available for potential members to reach switch fabric. While in the case of developing world here in Asia there might not be good commercial datacenter and hence one may end up in starting IX in a dedicated building. Also in developing world, there’s a considerable aerial fibre (legal as well illegal) and it’s usually easy for members to reach by pulling fibre or using someone else’s existing fibre. So, for example, IX’es like LoNAP, INEX, LINX, AMS-IX etc all run in well connected commercial datacenters. One can find the list of those datacenters by looking at peeringdb. While IX’es in South Asia like BDIX in Dhaka or NPIX in Nepal run out of their own buildings. Both approaches depend on lot more on existing fibre availability as well as rights of way charges. In the case of India, it’s a rather mix. IIFON-IX is new and runs out of a connected building in Kolkata in Sector 5, Kolkata while NIXI runs out of some good commercial datacenters in some cases and very poor datacenters in some other cases.
What’s wrong in India IX scenario? 
Well Indian IX scenario is a big mess. NIXI or National Internet Exchange of India is presently helping in exchange of 60Gbps traffic across NIXI Mumbai, Chennai, Noida, Kolkata, Bangalore, Hyderabad and Ahmedabad. There has been a jump in NIXI’s traffic in tier 1 Indian cities due to high traffic by Reliance Jio which happens to be exchanging traffic with Airtel via NIXI. But still 60Gbps is just nothing based on Indian scale in terms of a number of end users, amount of time Indians spend on internet etc. Except for NIXI Noida, Mumbai & Chennai, none is even crossing 200Mbps mark. NIXI’s IX in IT hub Bangalore is exchanging less than 1Mbps of traffic which is shameful (MRTG graph here). Same is true for Hyderabad and Ahmedabad.
Following things are wrong w.r.t NIXI and need to be fixed asap:

  1. The policy which allows only Indian ISP license holders to connect to IX (content players are not allowed!)
  2. The policy which does a ridiculous x-y on net inbound traffic and charges at a price of 1 INR /GB (which used to be 50 INR/GB in start)
  3. A misunderstanding that there’s  a mentality to keep above rules in order to keep large telcos. One can just let both: these stupid rules as well as large telcos to drop if they like to while having a large amount of smaller members over circuits of telcos
  4. Better choice of datacenters instead of sticking with not to well-connected datacenters

And of course, we need more efforts like IIFON and other players.
Remember: Somewhere up the transit path, there’s always peering! 🙂 
Disclaimer: This is my personal blog. The opinions expressed here represent my own and not those of my employer. 

23 Dec

Experiences from Bangladesh trip

So last month I had a wonderful trip to Bangladesh for bdNOG. This is bit delayed.

Some thoughts on infrastructure

  1. In terms of infrastructure – roads & traffic, power, quality of builds – it seemed like India in 2000’s.
  2. Specifically roads and traffic was bit terrible and even as an Indian (who manages to drive in Indian traffic!) I still got scared out of traffic in Dhaka. Speeds, roughness and overtaking is pretty high.
  3. There was no Uber and app based services are still pretty low. It was mostly usual “yellow taxi” which one had to call. (And it was expensive by local standards).
  4. There was excessive, just excessive amounts of overhead cabling in Dhaka and most of key city areas. It’s worth noting that there is way more overhead fiber than India. I guess most of it was running “active ethernet” based solutions (not a PON).  Most was just via media converters on both ends.
  5. I got 30Mbps speeds in cheap budget hotel in Dhaka which was more higher then what I have ever seen in India! (Speedtest here)
  6. Bangladesh currently is connected to outside world via SEA-ME-WE4 (landing at Cox’s Bazaar) and a terrestrial cable route via Kolkata.
  7. Overall network connectivity with India is decent since many large Bangladeshi networks buy transit from Tata Communications (AS6453) and Airtel (AS9498). So mostly there’s direct path to India and if not direct then via Singapore which added bit of latency but was not as bad as India-China routes.
  8. Bangladesh has a real & functional internet exchange 🙂


Overhead fiber in Dhaka


Speedtest in Dhaka Hotel

Screen Shot 2015-12-23 at 1.46.41 AM

Some thoughts around bdNOG event

  1. It was amazing to see how wonderful NOG community is in the Bangladesh. It was just like other regular NOGs from Western countries. We (Indians) have pretty much zero NOG involvement. SANOG does happens once every year but still hasn’t got as heavy as response as it would have been in large country like India. I still very much like it but was amazed to see response and user participation at bdNOG.
  2. I found almost all Bangladeshi people very friendly, helpful and relatively open in sharing their knowledge.
  3. May be it’s case due to excessive outsourcing industry full of “consultants“, “dumb managers” and so many other people in chain of managing operations here in India that NOG culture just never picked up. Most of “engineers” are either working at outsourcing firms managing large ISPs network and have limited visibility of overall operations, plus under NDAs and contracts. Thus actual knowledge and knowhow sharing is very low.
  4. In presentations presentation by Md. Zobair Khan on flow was interesting (link here), PKI industry growth by Al Faruq Ibna Nazim (link here), RTT matters by Maz from IIJ (link here).
  5. Dr. Khondkar Siddique-e-Rabbani gave a long but very interesting presentation on his work in the end (link here). It was extremely interesting to see challenges he faced and how he worked around them in local way. It is what I rarely see around in India i.e “real engineering”. I blogged in past how badly our Indian education system has lost sense of engineering and it’s pretty much just about marks, competitive exams, and just blank “management”.
  6. Dr. Khondkar also gave a fair amount of idea on how patents are bad for society and why he still has/had to re-event wheel since old version of wheel is there but patented! His acknowledgement of what Computer industry has achieved in terms of Open Source softwares was remarkable and very enlightening. 🙂
  7. Event ended with pool side dinner and as soon as I came out, I heard songs of Jagjit Singh being played on the music system. Somehow I felt I was never away from home. We were culturally so similar! 🙂

Overall I loved the experience and it was great to see how close we are culturally to Bangladesh (and likely Pakistan too!). I never appreciated Bollywood as much I did after this trip. It’s success somehow keeps our culture and Hindi (movies & songs!) very popular in the region. Bangladesh also has licensing around in telecom market just like India but it was different in certain ways. In terms of retail broadband market, capping is less common and low rate speed plans (with unlimited data transfer / no caps) are more common. It’s hard to know how feasible is business of domestic startups is here in India but I greatly appreciate the online/app culture they are bringing along with domestic hosting (bringing up demand for domestic datacenters). Bangladesh is likely to see those kind of things picking up within next few years.

11 Aug

Notes from SANOG 26 – Mumbai

Just finished with SANOG 26 conference and tutorials. It went very nice.
Interestingly this time conference did not start early morning like it did in SANOG 24 at Noida. It was rather late in afternoon. Also, on very good note – there were less Govt. bureaucrats to bore attendees with usual stuff they always talk about but have very little idea. One specific interesting presentation was  Opportunities and Challenges for Broadband Wireless in India by Prof Abhay Karandikar (from IIT Mumbai). In start I felt it to be usual crappy 5G talk but later realized it was much more interesting. I loved the idea “Have 2Mbps everywhere static broadband” and not some absurd number on mobile wireless broadband as we hear in case of 3G/4G. Although 2Mbps now is much slower and I would rather suggest that we target for 10Mbps everywhere (something which can be supported by copper/coax/fiber hybrid) but anyways it was nice refreshing talk.
His thoughts were interesting but mostly impractical since had high dependence on useless project like NOFN. For the next part, we had a nice theme of keeping network simple which everyone kind of liked. Simplicity in Network Design & Deployments by Dany Pinto (from Colt) and Unified Forwarding with Segment Routing by Mohan Nanduri (from Microsoft Azure Cloud WAN team) were part of that. Santanu Dasgupta gave a presentation about Challenges of L2NID based Metro-E Architecture for vCPE/NFV Deployments and kind of confused everyone. 😛
He did a nice job to state that the way we are proceeding on vCPE concept, we are moving with further complexity on the edge CPEs. Number of technical challenges are breaking off the fundamental requirement which was to make things simple.
On conference day 2 presentation from Vivek Nigam on ROA using RPKI was a nice intro to RPKI and the reasons why it’s not just another form of IRR deployment. On important note we (at Hurricane Electric) give a green signed key in front of prefix on our bgp.he.net tool. I will do a detailed blog post about RPKI later in sometime.  Attack Trends and Mitigation (from Matt Jansen) was nice. What always excites me is the status of IXPs across the region (excluding India of course which is messed up). BDIX in Bangladesh, NPIX in Nepal etc are doing really great. Wherever I go, the fact that I am from country with dysfunctional Internet Exchange bites me.

Traffic at BDIX Vs traffic at NIXI

BDIX traffic (as taken from here)
BDIX Traffic
Comparing this with NIXI traffic (taken from here)
Screen Shot 2015-08-11 at 12.26.29 am
Except for NIXI Mumbai, all other NIXI’s have pretty low traffic than that of BDIX alone!
Note: NIXI Chennai is somewhat standing in comparison while NIXI Delhi/Noida has very low traffic after they recently shifted to Netmagic Sector 63, Noida facility.
If we exclude random drop of traffic, then also NIXI traffic is somewhere around 35-36Gbps PAN India while we can see that in a community run IXP like BDIX that too in a relatively much smaller country like Bangladesh has a traffic of about 4.5Gbps. NIXI has been so called “supported and promoted” by Govt of India as well as strong Indian telcos but still it’s almost non-existent. Combined with this fact, even none of large Bangladeshi telcos have joined BDIX and it remains with other local competitive ISPs and content players (member list here). And even in absence of Google, Microsoft or Limelight it’s doing amazingly well. India has got around 300,340,854 as compared to 10,637,566 users in Bangladesh (as per wikipedia data). Thus even with 28x more users in India, we have barely 8x times more traffic at well funded NIXI. What a shame! 🙁 
One of very important part of so called “Digital India” push by Govt. of India should be a functional Internet Exchange with good policies. India’s Internet core need some serious fixes.

Cell phone towers everywhere…

Another interesting thing I found was the presence of cell phone towers just everywhere. Quite a few of them were using microwave backhaul while lot of them seemed to have local LCO fiber based backhaul.
Tower I saw a railway station…
Cell Tower at Kurla Railway Station, Mumbai
This one clearly is on fiber backhaul (notice the cables around and absence of microwave). While most of road side towers appeared to be microwave backhauled.
IMG_20150806_123329 IMG_20150806_123331
Clearly top metros are getting desperate for voice and data connectivity while limited spectrum, lack of competition in broadband and lack of IP telephony is all together killing a possible good market.
Some of immediate and important changes we need asap:

  1. Fix of NIXI/peering ecosystem in India. (Can’t stress anymore on that)
  2. Super easy terms for operating network in India including domestic bandwidth selling, use of transport services, and ofcourse International Long Distance circuits (so refered to as “ILD” in India)
  3. Open last mile and middle mile access of BSNL’s infra. Just like they are moving their towers to a subsidiary (news here), they also need to make their fixed line copper infra and long haul domestic fiber for use by telcos so telcos can compete over it. It’s seriously huge and seriously unused infra. Take for instance they have got around 37k exchanges (basic Tier 1/2 facilities with fixed line copper termination) and over 600,000 Kms of fiber optic cables connecting them (as per data from BSNL site here).
  4. Permission for IP telephony so that private VoIP companies can offer DID and call termination in India and this will result in migration of voice traffic over data networks freeing up hell lot of strain on limited 2G/3G spectrum across metros.
  5. More of wifi ofload of calls for native telecom calls. (Remember latest iOS offers wifi calling, none of Indian telcos are using that yet). Could be regulatory reason since haven’t seen any similar VoIP >> PSTN based app/offering from any telco yet). All all major enterprises across big cities have decent office wifi, this can remove strain of that kind of voice traffic from cell phone towers.


Competition takes care of it…

This time I travelled on flight booked via ibibo (quite cheap flight), stayed in hotel booked via Oyo rooms, travelled across Mumbai in local train, Uber and OLA. I occasionally ordered food via Foodpanda (and it was good!).
Furthermore ISP at hotel was relatively local Mumbai operator – Syscon Infoway and they had a wired wifi router installed in each room (backhauled over wired cat6 to their core). I was getting 4Mbps symmetric speeds with quite low latency. I can guess it was either fiber to the hotel or a fiber/coax hybrid.

In short this was much better than travelling via crappy airline booking site, staying in traditional Mumbai hotels or roaming across city in usual taxis or using hotel wifi with creepy proxy based authentication and sub 1Mbps speeds.
So without going into fine details of companies I named above, as consumer I very much enjoyed it and it was easier life. So competition just takes care of issues because someone can make money while solving problems. Mobile apps and internet play an important medium here and I would very much like to work in direction of fixing our internet connectivity problems. This reminds me of very nice post by Matt at wirelesscowboys blog – WISPs to AT&T Customers – We Got This.
Hoping for a better Internet future for India. Stay connected and keep peering! 🙂