26 May

India, DOCSIS, last mile broadband and more…

In my previous post, I shared how I am running redundant uplinks at home (in non-BGP based setup) with the primary link on RF and secondary on DOCSIS. One of my good friends asked me the reason for the sudden jump in DOCSIS-based players across India, especially in smaller cities.

Well, there are multiple reasons for it: 

  1. Fibre is less hard but still remains a cost point. ISPs in large metro cities ran fibre to the user’s building and go on cat5e/cat6 beyond that. That’s pretty much a lot of FTTH players do in India. That’s not possible in smaller cities which do not have multi-dwelling units.
  2. While the cost of fibre is low, the cost of CPE (i.e ONU in the case of GPON or media converter for active networks) is still slightly high. It has come dome drastically in last few years and is about to reach a point where it’s cost won’t be a consideration in near future. Presently at around 2000-2500 ~ $40 USD that is still a hard cost for mass deployment. On the other hand cost of DOCSIS 2.0 as well as DOCSIS 3.0 modems is extremely low.
  3. A considerable part of long coax plant of LCOs got converted into Hybrid Fibre Coax (HFC) as the digital television broadcast is pushed further across the country. So a lot of fibre is already present now (hanging around!) near to a large number of homes even in smaller cities.
  4. The demand for bandwidth is increasing but a lot in terms of GB usage a month while not in terms of raw speeds in megabit per second at the same rate. So while 2Mbps to 5-10Mbps is a major upgrade. Beyond 10-20Mbps there’s just not much demand as yet. It would be great once 4k streaming and more VR stuff get popular but at current levels, demand for speeds isn’t increasing beyond 10-20Mbps due to lack of “killer” application which takes more bandwidth.
  5. DOCSIS networks can be eventually upgraded to GPON all the way to end user with the fair amount of changes. Operators even do hybrid DOCSIS-GPON where they move high capacity end users on dedicated cable headends & even to GPON while leaving low demand users on DOCSIS.
  6. The cost of termination coax is low & relatively easy to handle compared to splicing fibre to pigtail.

 

Thus DOCSIS is a cheap way to build the last mile in tier 3 cities. While the cost of RG-6/RG-11 cable is on the high side (against the cost of fibre) the equipment is priced pretty low.

#2 here is important. While $40 may sound not much in West, that cost is really high. As per available data, 12.4% of Indian population is below the poverty line. Furthermore, if we speak of per capita income was Rs 93,293  ~ $1400 for year 15-16. And thus pricing really matters. It’s extremely large number of not connected population and extremely price sensitive market. Here people typically pay little over 1000Rs / $16 a month for broadband in metros & large cities and little less than that in smaller cities. Providers in Western countries charge typically 4-5x of that and much more once bundled with digital cable TV.

Some of the players who did large scale DOCSIS setup in recent times mostly are MSOs and come from the cable TV world.

  1. Hathway broadband – AS17488
  2. DEN cable  – AS45184
  3. Siti broadband – AS17747
  4. Netplus broadband – AS133661
  5. Ortel – AS23772
  6. Asianet broadband – AS17465

and more!

One thing is common across these players is good plans. Reliability does vary but is decent enough for the most part except few exceptions. Also, a large number of these players do CGNAT, and I am not aware of anyone of them giving IPv6 to end users as yet. 🙁

A quick view on plans

Hathway plans in Delhi

 

 

DEN Broadband in Delhi

 

If you are thinking this kind of plans where 50Mbps is being offered for 1200Rs /  $18 exist only Delhi, let me share some of the plans from smaller cities.

 

Netplus broadband in Punjab

 

Speedtest by one of their users and my friend Damanjit:

 

So how come these networks are cable to offer relatively good plans as compared to larger telcos who are mostly on DSL? One of lawyer friend from Bangalore asked this question earlier this month. This is due to following:

  1. The cost of IP transit is getting lower and lower across the world. While it’s still quite high in India compared to global standards but the decline is phenomenal. Thus these players are able to buy high capacity bandwidth drop at wholesale prices.
  2. The cost of point to point circuits is getting lower along with more content coming to India along with high-value content getting concentrated over CDNs. Over around half (actually 60% or so) of typical Indian eyeball ISP traffic can be offloaded via GGC, Google Peering, Akamai cache, Microsoft, Amazon, Limelight and so on besides generic HTTP and P2P caching. There’s a decent competition these days on the point to point circuit provider across India in all major cities. There are Govt. options (Railtel, Powergrid) as well as private players (Tata Comm, Airtel, Reliance, Vodafone, Aircel) etc offering circuits (which is known as NLD or National Long Distance in India).
  3. In terms of the last mile, these networks are typically DOCSIS 3.0 (for MSOs) backhauled over GPON or GEPON (yes, DOCSIS backhauled on PON instead of direct Ethernet circuits!).  From a central location where CMTS is located, traffic is pushed towards societies over HFC plant, in societies usually, remote PHY devices are deployed to do optical to coax conversion & then on each headend lot’s of end users are connected. So the layer 1 aggregation happens on these phy devices hanging around.

 

The capacity on last mile as well as the middle mile is quite questionable on these networks due to multiple layers of contention (DOCSIS as well as xPON) but since traffic growth is relatively predictable and happens over a period of months, this part is somewhat managed. FTTH, of course, is the way to go in the long term, but if these players can cut costs and offer reasonable bandwidth at cheap prices, DOCSIS is a fine option. In Rohtak, for instance, Siti broadband’s 5/10/20/50 Mbps packages are competing with Govt. owned BSNL’s 2/4/8Mbps plans with low FUPs.

 

 

GPON, as well as DOCSIS, offer a flexible approach of splitting circuit coming from OLT (in GPON) or Phy (in DOCSIS) as the need comes. So for instance in the case of FTTH on GPON, one can aggregate 16 or more users over such passive splitter:

 

in the case of DOCSIS, it’s usual old school (though improved) coax taps:

While DSL is very different as compared to it. The cost of aggregating DSL userbase over DSLAM is much higher and capacities on ADSL are quite low.

This is how a typical DOCSIS remote PHY looks like

 

Ending this post with some pictures of outdoor fibre plant from my recent Bangalore trip. That’s part of our Digital Slum here in India.

24 Mar

Internet Exchanges – Place where the networks interconnect!

Earlier this month I got an opportunity to be part of IXP workshop in Kolkata. It was a 3-day event organised by ISOC Kolkata and supported by APNIC. There was also a workshop on DNSSEC and Champika Wijayatunga (from ICANN) was the instructor along with Anand Raje. It was a nice event and I come to know of other interesting projects ISOC Kolkata is doing like Indian IETF capacity building program apart from the IXP they are running in Kolkata. Mr Anupam Aggarwal and Anand showed the IX and it looks very good. I think it’s the first and only IX I know in India which is a real IX with proper policy. It’s an IX by a non-for-profit group, allows anyone to connect, a real layer 2 IX and welcomes anyone including ISPs, content players and root DNS servers. Presently IIFON-IX in Kolkata has few member ISPs besides the L root from ICANN and one of Verisign gTLD nodes (which host zones for .com, .net etc). I also saw a rack with some of Akamai CDN servers. This brings decent content right there. IX’es play an extremely important part of current internet infrastructure ecosystem. It’s very likely that content of this blog is travelling from my server to your browser from an Internet Exchange. 🙂

 

IIFON-IX Kolkata

 

 

I hope IIFON-IX does well in terms of becoming a key point for internet traffic exchange in Kolkata. There are a number of benefits of a good IX. Some of them are:

  1. Helps in reducing overall cost of bandwidth by keeping it local
  2. Helps in giving ultra-low latency interconnections among members
  3. Builds resilience across the internet by removing the dependency on transit players for local traffic and a lot more!

 

Besides IIFON-IX, I have been to BDIX in Dhaka, HKIX in Hong Kong, INEX IX in Dublin, LoNAP in London, Netnod in Stockholm etc. One thing common across all these exchanges is local involvement, usually a non-for-profit model and a collective community effort to make internet infrastructure better.

 

INEX – Dublin

 

 

Nick Hilliard (CTO at INEX) was kind enough to give me ride to one DCs hosting the IX.

Here’s a pic with Nick:

 

It was very interesting to see the use of passive DWDM MUX in the IX to connect between INEX PoPs across Dublin.

 

The one in this picture is an MRV MUX. Each port in passive MUX has it’s own wavelength and requires either pre-tuned or tunable SFP for that specific port. Such MUX can help to increase capacity on a pair of the strand by offering 40 channels and each can carry up to 10Gbps thus a capacity of 400Gbps for shorter distances (20kms or so).

The channels numbers mentioned on the port correspond to ITU Channel table and here’s the frequency wise table of actual channel number:

 

LoNAP – London

 

During my trip to UK, my good friend Andy Davidson helped in arranging a tour of LoNAP and Robert Lister was kind enough to give me a nice detailed tour of the IX fabric.

 

 

It was nice to see 10Gig flavours of Bidi SFPs i.e SFPs which do 10Gbps on a single strand of fibre. These are slightly different in terms of wavelengths as they run on 1270/1330 pair (opposite on the other side) while 1G Bidi optics runs on 1310/1490 pair.

 

I also saw a QSFP-LR4 which can do 100Gbps up to 10kms on a pair of fibre. 100Gbps or 100,000,000,000 bits per second or 100,000,000,000 zeros and ones flowing from fibre every second!

 

This runs on 1310mn and I don’t think there’s any Bi-Di option yet for 100G. There was ongoing 100G deployment at LoNAP during last year summers when I visited. One can spot an Arista switch with 100G ports.

 

And here’s a picture with amazing Rob after the IX-tour ended. 🙂

 

BDIX – Dhaka

Some pictures from Bangladesh Internet Exchange in Dhaka.

Mr Kabir from BDIX gave me an excellent tour of BDIX during my visit to Bangladesh back in 2015. You can see a lot of 1G fibre media converters in the rack. There’s quite a lot of fibre in Dhaka and cost of media converter is low and hence using them instead of the optical switch makes commercial sense though that’s changing fast. IX runs it its own dedicated building floor. IX has over 78 members.

Here’s a picture with Mr Kabir 🙂

 

Where should IX fabric be located?  

This is one of key question which comes in the mind of many engineers when talking of IX. Should IX be in a separate dedicated building or it should be rather in a commercial datacenter? The answer is simply that it depends. In larger cities in US or Europe, there are quite a lot of datacenters which are well connected. There’s reasonable competitive fibre available for potential members to reach switch fabric. While in the case of developing world here in Asia there might not be good commercial datacenter and hence one may end up in starting IX in a dedicated building. Also in developing world, there’s a considerable aerial fibre (legal as well illegal) and it’s usually easy for members to reach by pulling fibre or using someone else’s existing fibre. So, for example, IX’es like LoNAP, INEX, LINX, AMS-IX etc all run in well connected commercial datacenters. One can find the list of those datacenters by looking at peeringdb. While IX’es in South Asia like BDIX in Dhaka or NPIX in Nepal run out of their own buildings. Both approaches depend on lot more on existing fibre availability as well as rights of way charges. In the case of India, it’s a rather mix. IIFON-IX is new and runs out of a connected building in Kolkata in Sector 5, Kolkata while NIXI runs out of some good commercial datacenters in some cases and very poor datacenters in some other cases.

 

What’s wrong in India IX scenario? 

Well Indian IX scenario is a big mess. NIXI or National Internet Exchange of India is presently helping in exchange of 60Gbps traffic across NIXI Mumbai, Chennai, Noida, Kolkata, Bangalore, Hyderabad and Ahmedabad. There has been a jump in NIXI’s traffic in tier 1 Indian cities due to high traffic by Reliance Jio which happens to be exchanging traffic with Airtel via NIXI. But still 60Gbps is just nothing based on Indian scale in terms of a number of end users, amount of time Indians spend on internet etc. Except for NIXI Noida, Mumbai & Chennai, none is even crossing 200Mbps mark. NIXI’s IX in IT hub Bangalore is exchanging less than 1Mbps of traffic which is shameful (MRTG graph here). Same is true for Hyderabad and Ahmedabad.

Following things are wrong w.r.t NIXI and need to be fixed asap:

  1. The policy which allows only Indian ISP license holders to connect to IX (content players are not allowed!)
  2. The policy which does a ridiculous x-y on net inbound traffic and charges at a price of 1 INR /GB (which used to be 50 INR/GB in start)
  3. A misunderstanding that there’s  a mentality to keep above rules in order to keep large telcos. One can just let both: these stupid rules as well as large telcos to drop if they like to while having a large amount of smaller members over circuits of telcos
  4. Better choice of datacenters instead of sticking with not to well-connected datacenters

 

And of course, we need more efforts like IIFON and other players.

Remember: Somewhere up the transit path, there’s always peering! 🙂 

Disclaimer: This is my personal blog. The opinions expressed here represent my own and not those of my employer.