11 Aug

Notes from SANOG 26 – Mumbai




Just finished with SANOG 26 conference and tutorials. It went very nice.

Interestingly this time conference did not start early morning like it did in SANOG 24 at Noida. It was rather late in afternoon. Also, on very good note – there were less Govt. bureaucrats to bore attendees with usual stuff they always talk about but have very little idea. One specific interesting presentation was  Opportunities and Challenges for Broadband Wireless in India by Prof Abhay Karandikar (from IIT Mumbai). In start I felt it to be usual crappy 5G talk but later realized it was much more interesting. I loved the idea “Have 2Mbps everywhere static broadband” and not some absurd number on mobile wireless broadband as we hear in case of 3G/4G. Although 2Mbps now is much slower and I would rather suggest that we target for 10Mbps everywhere (something which can be supported by copper/coax/fiber hybrid) but anyways it was nice refreshing talk.

His thoughts were interesting but mostly impractical since had high dependence on useless project like NOFN. For the next part, we had a nice theme of keeping network simple which everyone kind of liked. Simplicity in Network Design & Deployments by Dany Pinto (from Colt) and Unified Forwarding with Segment Routing by Mohan Nanduri (from Microsoft Azure Cloud WAN team) were part of that. Santanu Dasgupta gave a presentation about Challenges of L2NID based Metro-E Architecture for vCPE/NFV Deployments and kind of confused everyone. 😛

He did a nice job to state that the way we are proceeding on vCPE concept, we are moving with further complexity on the edge CPEs. Number of technical challenges are breaking off the fundamental requirement which was to make things simple.

On conference day 2 presentation from Vivek Nigam on ROA using RPKI was a nice intro to RPKI and the reasons why it’s not just another form of IRR deployment. On important note we (at Hurricane Electric) give a green signed key in front of prefix on our bgp.he.net tool. I will do a detailed blog post about RPKI later in sometime.  Attack Trends and Mitigation (from Matt Jansen) was nice. What always excites me is the status of IXPs across the region (excluding India of course which is messed up). BDIX in Bangladesh, NPIX in Nepal etc are doing really great. Wherever I go, the fact that I am from country with dysfunctional Internet Exchange bites me.




Traffic at BDIX Vs traffic at NIXI


BDIX traffic (as taken from here)

BDIX Traffic


Comparing this with NIXI traffic (taken from here)

Screen Shot 2015-08-11 at 12.26.29 am


Except for NIXI Mumbai, all other NIXI’s have pretty low traffic than that of BDIX alone!

Note: NIXI Chennai is somewhat standing in comparison while NIXI Delhi/Noida has very low traffic after they recently shifted to Netmagic Sector 63, Noida facility.

If we exclude random drop of traffic, then also NIXI traffic is somewhere around 35-36Gbps PAN India while we can see that in a community run IXP like BDIX that too in a relatively much smaller country like Bangladesh has a traffic of about 4.5Gbps. NIXI has been so called “supported and promoted” by Govt of India as well as strong Indian telcos but still it’s almost non-existent. Combined with this fact, even none of large Bangladeshi telcos have joined BDIX and it remains with other local competitive ISPs and content players (member list here). And even in absence of Google, Microsoft or Limelight it’s doing amazingly well. India has got around 300,340,854 as compared to 10,637,566 users in Bangladesh (as per wikipedia data). Thus even with 28x more users in India, we have barely 8x times more traffic at well funded NIXI. What a shame! 🙁 

One of very important part of so called “Digital India” push by Govt. of India should be a functional Internet Exchange with good policies. India’s Internet core need some serious fixes.


Cell phone towers everywhere…

Another interesting thing I found was the presence of cell phone towers just everywhere. Quite a few of them were using microwave backhaul while lot of them seemed to have local LCO fiber based backhaul.

Tower I saw a railway station…

Cell Tower at Kurla Railway Station, Mumbai


This one clearly is on fiber backhaul (notice the cables around and absence of microwave). While most of road side towers appeared to be microwave backhauled.

IMG_20150806_123329 IMG_20150806_123331



Clearly top metros are getting desperate for voice and data connectivity while limited spectrum, lack of competition in broadband and lack of IP telephony is all together killing a possible good market.


Some of immediate and important changes we need asap:

  1. Fix of NIXI/peering ecosystem in India. (Can’t stress anymore on that)
  2. Super easy terms for operating network in India including domestic bandwidth selling, use of transport services, and ofcourse International Long Distance circuits (so refered to as “ILD” in India)
  3. Open last mile and middle mile access of BSNL’s infra. Just like they are moving their towers to a subsidiary (news here), they also need to make their fixed line copper infra and long haul domestic fiber for use by telcos so telcos can compete over it. It’s seriously huge and seriously unused infra. Take for instance they have got around 37k exchanges (basic Tier 1/2 facilities with fixed line copper termination) and over 600,000 Kms of fiber optic cables connecting them (as per data from BSNL site here).
  4. Permission for IP telephony so that private VoIP companies can offer DID and call termination in India and this will result in migration of voice traffic over data networks freeing up hell lot of strain on limited 2G/3G spectrum across metros.
  5. More of wifi ofload of calls for native telecom calls. (Remember latest iOS offers wifi calling, none of Indian telcos are using that yet). Could be regulatory reason since haven’t seen any similar VoIP >> PSTN based app/offering from any telco yet). All all major enterprises across big cities have decent office wifi, this can remove strain of that kind of voice traffic from cell phone towers.


Competition takes care of it…

This time I travelled on flight booked via ibibo (quite cheap flight), stayed in hotel booked via Oyo rooms, travelled across Mumbai in local train, Uber and OLA. I occasionally ordered food via Foodpanda (and it was good!).

Furthermore ISP at hotel was relatively local Mumbai operator – Syscon Infoway and they had a wired wifi router installed in each room (backhauled over wired cat6 to their core). I was getting 4Mbps symmetric speeds with quite low latency. I can guess it was either fiber to the hotel or a fiber/coax hybrid.

In short this was much better than travelling via crappy airline booking site, staying in traditional Mumbai hotels or roaming across city in usual taxis or using hotel wifi with creepy proxy based authentication and sub 1Mbps speeds.

So without going into fine details of companies I named above, as consumer I very much enjoyed it and it was easier life. So competition just takes care of issues because someone can make money while solving problems. Mobile apps and internet play an important medium here and I would very much like to work in direction of fixing our internet connectivity problems. This reminds me of very nice post by Matt at wirelesscowboys blog – WISPs to AT&T Customers – We Got This.


Hoping for a better Internet future for India. Stay connected and keep peering! 🙂

05 Mar

Different CDN technologies: DNS Vs Anycast Routing

And I am back from Malaysia after attending APRICOT 2014. It was a slightly slow event this time as less people came up due to change of location from Thailand to Malaysia. But I kind of enjoy the APRICOT in start of year. 🙂

It has been quite sometime when I blogged. After getting into Spectranet I got relatively more busy along with bit of travelling to Delhi NCR which has been taking lot of time. I wish to blog more over time. 

In recent time I got chance to understand in detail the working of CDN from the point of view of delivery and this brings me to this post where I will be working on putting in detail how the popular CDN networks work and where they are dependent on DNS recursors and where on anycast routing. 


Understanding CDN

CDN’s as we know are Content Delivery Networks and these are specialized networks which are designed for the content delivery to the edge networks by serving content from as close location as possible. The location of servers and type of connectivity heavily depends on each CDN provider and their business model. E.g Google maintains it’s own delivery network consisting of large number of GGC (Google Global Cache) nodes placed on ISPs network and help in serving Google’s static content while other large networks like Akamai (whose core business is into Cache delivery) put their servers on large number of edge networks but they stay as disconnected small islands. While the new comers in the industry like Limelight,  Cloudflare’s model of deployment is around putting node in major datacenter and direct connection to major networks via peering from IXPs. 


The key features of almost all these CDNs are:
  1. Low latency delivery of content giving very fast throughputs.
  2. Making networks more efficient by caching near to the point of serving and not consuming long haul International bandwidth.
  3. Ensuring that content is delivered with optimum performance with as low as possible dependency on middle networks/backbone. 
  4. Ensures that there is no single point distribution and hence during high load, traffic serving can be optimized. 


Technical side of “edge cache serving”

In order to make the “edge delivery” concept work, CDN providers have multiple options and it is slightly tricker here. Challenge here is to ensure that all users go to their nearest CDN node and get served from there rather then a node far away from them. 


Here we have ISP A with a Cache A deployed very near to it, ISP B with Cache B deployed just next to it and so does ISP C with Cache C right next to it. Assuming that end users visit a website which has services from the CDN provider. Here end user will get a url like “http://cdn.website.com/images/image1.jpg” and here cdn.website.com is supposed to be going to “nearest node”. Thus we expect that when users try to reach cdn.website.com on ISP A, it should hit Cache A, from ISP to Cache B and so on (under normal circumstances). 


Two fundamental ways to achieve that:

  1. Have DNS to do the magic i.e when users from network ISP A lookup for cdn.website.com, they should get a unicast IP address of Cache A in return, similarly for users coming from ISP B network, Cache B’s unicast IP should return. 
  2. Have routing to route to nearest cache node based on “anycast routing” concept. Here Cache A, Cache B and Cache C will use same identical IP address and routing will take care of reaching the closest one. 


Both of these approaches have their own advantages as well as challenges. Some of very large CDN providers like that of Akamai, Amazon Cloudfront rely on DNS. While some of new entrants like Cloudflare rely very much on anycast routing. I have discussed DNS and it’s importance in CDN and node selection in some previous posts, but will be going through this quickly in this one. 


Making use of DNS for CDN

DNS is pretty basic protocol. It’s role is simply into “hostname to IP resolution” (and vice versa). What makes is powerful is that based on certain logic, we can influence this “hostname to IP resolution” and do many cool things like load balancing, high availability, and more. However the key challenge in doing all that is first result of DNS changes usually is not instance since there is lot of caching by the “recursive DNS servers” and second that since recursive DNS servers contact authoritative DNS servers, thus authoritative DNS servers (as by default protocol design) don’t really know of end users. They only know that to which DNS recursor they are talking with (based on source IP of DNS recursor) which many times has relation with end users since primarily ISPs run the recursive DNS servers. But in modern world of large Open DNS recursors like OpenDNS, Google Public DNS – it faints out that impact. 


Here’s how DNS based CDN services work




Here we have users on ISP A requesting for “cdn.website.com” IP address. Requests will go to DNS recursor of ISP which will further hit authoritative DNS servers of CDN provider via DNS hierarchy. Green lines here show the flow of DNS information. Eventually based on IP of requesting DNS recursor, authoritative DNS will reply back with the IP address of cache node close to network A. 


Some of key features of this approach:
  1. Optimization logic is pretty much with authoritative DNS server which can change around IP in order to give a location which can serve off request in optimum manner. If one of edge servers is down, algorithm can take care of it by serving other location.
  2. In most of such deployments cdn.domain.com points to cdnxx.cdn-provider.com via cname record and thus actual resolution logic stays within domain of cdn-provider.com. The records like cdnxx.cdn-provider.com have very low TTL (less then a minute) to make changes reflect instantly. 
  3. These approaches fails significantly if end users do not use DNS recursors of their ISP since reply is very much dependent on location/GeoIP parameters of source IP of DNS recursor. 


Some of new CDN networks have came up with full anycast based setup with very little dependency on DNS. E.g Cloudflare.


Here’s how anycast routing based CDN providers work




Here  we have User1 & User 2 on ISP A connected to ISP A router, User 3 & User 4 on ISP B connected to ISP B router & finally User 5 & User 6 on ISP C connected on ISP C router. All off these routers are have CDN provider caches nearby and get multiple routes. So e.g for ISP A router, CDN server A is 1 hop away, while CDN server B is 2 hops away and CDN Server C is 3 hops away. If all servers use the same IP then ISP A will prefer going to CDN ServerA, B will go to CDN server B and so on with C. 


Some of key features of this approach:

  1. Optimization is based on BGP routing and announcement with little role of DNS. 
  2. This setup is very hard to build up and scale since for anycast to work perfectly at global level, one needs lot’s and lot’s of peering and consistent transit providers at each location. If any of peers leaks a route to upstream or other peers, there can be lot of unexpected traffic on a given cluster due to break of anycast. 
  3. This setup has no dependency on DNS recursor and hence Google DNS or OpenDNS works just fine. 
  4. This saves a significant amount of IP addresses since same pools are used at multiple locations. 



With that beings said, I hope you are getting served from nearest cache for static content of my blog. (since I use Amazon Cloudfront for static content). 🙂


Disclaimer: This is my personal blog and does not necessarily reflect thoughts of my employer.