27 Apr

Cloudflare hosting F root server

A few days some folks in internet community noticed Cloudflare AS13335 announcing F root server’s routes covering prefix

Above tweet shows that case is clearly not a mistake but rather some sort of arrangement between Cloudflare and ISC (which is responsible for F-root). There was another discussion on DNS-OARC mailing list here.
From our bgp.he.net tool, one can analyse route propagation for F root’s AS3557.

Here we can see that routes are visible from AS13335 to Telefonica AS13335. One can safely assume that AS12956 (considered to be a Tier1 / transit free network as per list here) is not a customer of Cloudflare. So the fact that still route is being announced to Telefonica gives an impression that AS3557 is downstream of Cloudflare.
It’s hard to say on the kind of arrangement as I still see many of instances of F-root are being hosted directly by ISC from hostname.bind query triggered via local RIPE Atlas probes in those regions. It could be that Cloudflare is hosting a part of this setup say in developing region where they have their caching PoPs. Yet to see a detailed blog post from Cloudflare about it. (I like their detailed blog!)
Update: 2nd April 2017 
Henry from Cloudflare pointed me to their official announcement blog post written after around 5 months of this post. You can find their official announcement here.

06 Apr

Route filter generation for Mikrotik RouterOS via IRR

A while back I posted about routing filter generation via bgpq3 for Cisco (ios and XR) and Juniper JunOS based routers. I have received a number of emails in last few months about automated filter generation for Mikrotik routeros. Since Mikrotik’s CCRs are getting quite popular across small to mid-sized ISPs.
So this blog post is about ways for generating filter config for a given ASN via IRR. One can use such logic with some kind of remote login mechanism like rancid (look for mtlogin here).
I tried building around bgpq3 but it seems more easy with another popular tool in the domain called IRR Power Tools. Once IRR Power Tools (IRRPT) is setup, it allows us to fetch prefixes based via Internet Routing Registries and also aggregates them.
So, for instance, let’s pick AS54456:

anurag@tools:~/irrpt$ bin/irrpt_fetch 54456
Processing AS54456 (Record 1)
   - Importing /home/anurag/irrpt/db/54456                   version 1.1
   - Importing /home/anurag/irrpt/db/54456.4                 version 1.1
   - Importing /home/anurag/irrpt/db/54456.6                 version 1.1
   - Importing /home/anurag/irrpt/db/54456.agg               version 1.1
   - Importing /home/anurag/irrpt/db/54456.4.agg             version 1.1
   - Importing /home/anurag/irrpt/db/54456.6.agg             version 1.1
Completed processing of 1 IRR object(s).

So now we have got prefixes and this includes both basic route objects as well as aggregates.

anurag@tools:~/irrpt$ cat /home/anurag/irrpt/db/54456.4
anurag@tools:~/irrpt$ cat /home/anurag/irrpt/db/54456.4.agg

It offers a nice interface for generation of config for Cisco, Juniper, Extreme, Foundry and Force10. Example:

anurag@tools:~/irrpt/bin$ ./irrpt_pfxgen -f cisco 54456
conf t
no ip prefix-list CUSTOMER:54456
no ipv6 prefix-list CUSTOMERv6:54456
ip prefix-list CUSTOMER:54456 permit le 24
write mem
anurag@tools:~/irrpt/bin$ ./irrpt_pfxgen -f juniper 54456
policy-options {
    replace: policy-statement CUSTOMER:54456 {
        term prefixes {
            from {
                route-filter upto /24;
            then next policy;
        then reject;
policy-options {
    replace: policy-statement CUSTOMERv6:54456 {
        term prefixes {
            from {
            then next policy;
        then reject;

So I put a routeros instance in a VM to test and create config from their CLI. Config looks something like this:

[admin@MikroTik] > routing filter add chain=Cloudaccess prefix= prefix-length=22-24 action=accept
[admin@MikroTik] > routing filter add chain=Cloudaccess  action=reject
[admin@MikroTik] >

This seems logical and can be scripted. So one can have a script to read the aggregate file and if aggregate says /24 one can put it directly in the filter else allow filter up to /24 from whatever range the pool starts and similar logic in IPv6.
So here’s the script:

# Script for generating BGP filter for Mikrotik RouterOS
# Input name of chain via $1 and ASN via $2
# Grab latest filters via RADB / other IRRs using IRRPT
echo "Grabbing prefixes for AS$2"
php $irrpt/bin/irrpt_fetch $2
echo "***Start of Mikrotik routeros config below***"
# IPv4 config part
cat $irrpt/db/$2.4.agg | while read prefix
masklength=`echo $prefix|awk -F '/' '{print $2}'`
if [ "$masklength" -eq 24 ]
	# Prefix is a /24 - generating config without defining prefix length
	echo "routing filter add chain=$1-IPv4 prefix=$prefix action=accept"
elif [ "$masklength" -lt 24 ]
	# Prefix is greater than /24 - generating config with prefix length upto /24
	echo "routing filter add chain=$1-IPv4 prefix=$prefix prefix-length=$masklength-24 action=accept"
# Last entry for denial of pools
echo "routing filter add chain=$1-IPv4 action=reject"
cat $irrpt/db/$2.6.agg | while read prefix6
masklength6=`echo $prefix6|awk -F '/' '{print $2}'`
if [ "$masklength6" -eq 48 ]
	# Prefix is a /48 - generating config without defining prefix length
	echo "routing filter add chain=$1-IPv6 prefix=$prefix6 action=accept"
elif [ "$masklength6" -lt 48 ]
	# Prefix is greater than /48 - generating config with prefix length upto /48
	echo "routing filter add chain=$1-IPv6 prefix=$prefix6 prefix-length=$masklength6-48 action=accept"
# Last entry for denial of pools
echo "routing filter add chain=$1-IPv6 action=reject"
echo "***End of Mikrotik routeros config***"

So the script works except with a small bug in IPv6 aggregation which is the issue with IRRPT and I have reported same on their GitHub project page here.
An example of the script in progress for Cloudaccess AS54456:

anurag@tools:~/irrpt$ ./routeros.sh Cloudaccess 54456
Grabbing prefixes for AS54456
Processing AS54456 (Record 1)
Completed processing of 1 IRR object(s).
***Start of Mikrotik routeros config below***
routing filter add chain=Cloudaccess-IPv4 prefix= prefix-length=22-24 action=accept
routing filter add chain=Cloudaccess-IPv4 action=reject
routing filter add chain=Cloudaccess-IPv6 action=reject
***End of Mikrotik routeros config***

Here’s another example of it in action with NPCI’s AS132351

anurag@tools:~/irrpt$ ./routeros.sh NPCI 132351
Grabbing prefixes for AS132351
Processing AS132351 (Record 1)
   - Importing /home/anurag/irrpt/db/132351                  version 1.1
   - Importing /home/anurag/irrpt/db/132351.4                version 1.1
   - Importing /home/anurag/irrpt/db/132351.6                version 1.1
   - Importing /home/anurag/irrpt/db/132351.agg              version 1.1
   - Importing /home/anurag/irrpt/db/132351.4.agg            version 1.1
   - Importing /home/anurag/irrpt/db/132351.6.agg            version 1.1
Completed processing of 1 IRR object(s).
***Start of Mikrotik routeros config below***
routing filter add chain=NPCI-IPv4 prefix= prefix-length=22-24 action=accept
routing filter add chain=NPCI-IPv4 action=reject
routing filter add chain=NPCI-IPv6 prefix=2001:df0:2f0::/46 prefix-length=46-48 action=accept
routing filter add chain=NPCI-IPv6 action=reject
***End of Mikrotik routeros config***


Thinking to automate? 
The config between ***start*** and ***end*** can be pasted directly in CLI with Mikrotik. I would not recommend using it for manual filtering of any larger network. Automated filtering where filters are generated regularly makes sense but manual filtering without automation can be damaging. One can use a script like this for connecting to smaller networks. Also, IRRPR offers diff management via CVS (I hope they come up with git on that part) and it comes with an option to trigger email update so Network admins can know when to manually update. I would prefer that for non-commit based platforms since with Cisco ios or Mikrotik routeros it can be tricky to auto update prefix list. If one does no ip prefix list before triggering update it will cause a major noticeable impact. So ideal way to manage that on non-commit based devices would be to maintain a list of prefixes separately in the plain text file or a database and diff it against old one & only push for changes. Do-able and should be preferred that way instead of deleting and re-adding the whole list while automating.
Time to get back to work! 🙂

01 Apr

India's digital slum problem

India has a slum problem as many of us know. Slums are a serious problem and there’s just no easy way to fix them. One cannot just push thousands and thousands of people out while at the same time quality of life in slums is terrible. One thing which happens a lot in India is the fact that Govt. does nothing when slums are getting established and once they are established situation gets out of control.

Coming to digital slums

This exact problem is applicable even to digital India. Optical fibre is one of the key and extremely important components of the “Digital India”. While Digital India may sound more like a fancy marketing word but one cannot simply expect much unless connectivity is there and with connectivity I mean affordable, reliable, super-fast connectivity with low latency. Most of the wireless technologies cannot deliver it or simply cannot scale up. So we need more and more fibre. Just like everywhere else we need to push fibre closer to the end user. Our approach in that design needs to be far different from Western world especially US, UK, etc. They lag behind considerably in last mile fibre deployment and the conditions are far different in deployments over there Vs of India.
Say for instance:

  1. Fixed line infra using twisted copper & coax is well built and is present in most parts of the Western world. In India, it’s almost missing.
  2. The cost of replacing existing infra is high in the Western world due to the expectation of certain quality, compliance etc and that makes it extremely expensive in terms of per user cost of delivering FTTH. I have read a number of estimates where it can cost $2000 – $3000 per household in an urban FTTH deployment.Even if it’s half of this it’s still on the quite higher side. In India, it’s a fraction of the cost.
  3. “Fibre to the node and something else beyond” can give high bandwidth. It may not be able to give 1Gbps but can easily do 50-100Mbps. Technologies like VDSL, DOCSIS 3.0 (and upcoming 3.1), use of cat5e / cat6 wiring inside buildings can give high bandwidth for a fraction of the cost. In India, that’s not an option (at large scale) due to a number of issues ranging from location/real estate cost of mini-PoP, cooling requirements, reliable & backup power requirements & ensuring no one steals the stuff! All that makes FTT(n) harder than FTTH in India. 

So is the picture so rosy and everything is fantastic? Can we expect lot’s of FTTH deployment now and for a really low price? Before I come to that, let me post some of the pictures I took recently of Salte Lake Electronic Complex, Kolkata.

It’s ugly, unscalable, will soon have reliability issues and of course it’s extremely likely illegal. It’s not just about Kolkata. Pick any of commercial areas in any of large Indian city and it would be similar. These are what we can call as “Digital slums”. Govt. isn’t focusing on them, they are coming up, getting established, serving gigs of capacity to various high revenue generating companies around. Same is true not just for India but most of developing countries around us.

Some pictures from streets in Dhaka


Some pictures from Kathmandu

and some pictures from last month’s travel to Ho Chi Minh, Vietnam:

Here’s a quick video showing it feels like over there…

That’s about Vietnam. Things would be scary if we try to replicate such model in a country like India. Thus one can clearly establish that this is going be a major problem and needs to be addressed.  At this point of time, one may ask why fibre gets deployed in this way? I asked this to a number of Indian networks and here is the summary of why:

  1. Rights of way are deadly expensive. Rights of way or RoW is referred to the charge network operators need to pay to local municipal bodies because of interruption when the fibre is deployed. Cost depends on a lot on the area but in key areas in Delhi, it can be as high as 80lakhs – 1crore per KM ($1.54k). Remember we are not talking about the cost of fibre (which is very low) or even hardware, or anything. This is a one-off cost that goes to municipal bodies. In my own city, I have 3 telco pits within 400m of my house and one of the telcos gave me an unofficial quote of 15lakhs ($23000!) for extending their fibre 400m.
  2. Because of above there’s a huge market of so-called LCOs (Local Cable Operators) and they lay the fibre. LCOs have (mostly) unofficial contracts with other LCOs and a considerable amount of last mile infra is not of any telco or ISP, it’s indeed of LCOs.
  3. The above model does not scale up since multiple LCOs put multiple fibre cables & a large part of it is illegal, undocumented and hence not worth much money on paper resulting in “hard to prove asset” for any private funding for expansion.

Illegal but ethical?

While a lot of that is purely illegal I cannot say it should not be allowed at all and Govt. should remove all of them right away. I know many smaller ISPs operating in tier 2, tier 3 cities and they provide excellent service, great competition and very good network built on fibre and mostly with GPON / GEPON etc. Almost all of them offer an excellent competition to state-run BSNL which is poor in most of the aspects of service deployment to technology, from sales to customer support. In other words, we very much need smaller private players to lay networks and they can do really well. Which makes the whole problem similar to “slums” and hence we can call it “India’s Digital Slum” problem. We can’t just get rid of them while we ignored and still very much ignoring the problem as it’s building up.

Possible solution?

So what can be a solution to this problem? Cheap RoW is not a solution. Already Indian cities struggle very much with basic infrastructure and cheap RoW will result in excessive digging, more broken roads, and more outages in utility services. While there has been a lot of innovation in the application layer, layer 3 as well as even transport layer there’s not much in the optical fibre. What I am trying to point out essentially is that fibre optic cables are more of a commodity now and single mode cables are cheap. The overall technology whether one does active ethernet or passive PON – the cost of technology is very low. A design of 100% underground cables is needed and will be ideal setup with the flexibility of adding more networks. If the optical fibre is laid by one or few players it can lead to the dangerous non-competition condition. While Govt’s efforts to do FTTH via BSNL & MTNL are nothing but a terrible waste of taxpayers money in the inefficiencies of those organisations. While interestingly other Govt. players who are doing fibre on long haul are doing much better. Take the case of RailTel or Powergrid Telecom. Both very much compete with their private counterparts for IP transit as well as high capacity circuits on long haul  (we refer those as NLD in India).
Here’s a possible solution for last mile deployment without ugly cables & which can work:

  1. A cabinet on each and every street in the city rolled out in phased manner across the country. One can have a single cabinet at inter- section. (This already exists at large scale for BSNL’s & MTNL’s copper infra btw!)
  2. Cabinet has to be active (with power!) so that one can put a switch or GPON OLT. Logic has to be to the aggregate traffic of all homes in the neighbourhood.
  3. It should be Govt. which lays cables from each of this street cabinet to atleast one of 4-5 neutral exchanges across a city (whichever is nearby). Number of exchanges very much depend on the size of the city. So, for instance, Delhi or Kolkata would need lot more. As long as there is 288 strand fibre available from core exchange area to the street, one can have as many as 100 ISPs + (each taking two strands up to cabinet). Now ISPs can decide which last mile technology to use. One can do with lower strand depending on the city. Even a 48 core can give over 20 ISPs per area which is fair amount of competition. Ideal would be to connect each cabinet to two exchanges so ISPs can create their own “rings” for redundancy reasons.
  4. From cabinets to each home ideal would be to have underground fibre laid by Govt. but doing that will be terribly expensive. It makes lot more sense to have overhead (well planned) cables from cabinet to each home. This won’t look as ugly as one may think and reduces cost significantly. Overhead at such shorter scale gives the option of “connecting home as demand comes”. While entire underground approach has to cover all and that doesn’t work well in terms of very low internet penetration.
  5. Connectivity beyond the central exchanges can be very well done with 100% underground fibre with existing expensive RoW since fibre beyond this point will be of telcos and by logical design would be used by lot’s of players with DWDM and cost won’t be a challenge. It’s similar to existing significant underground fibre reaching BTS sites across the country.

So under such design one can have a number of larger networks building their own fibre or simply buying waves from existing fibre players to reach exchanges in the city, a number of smaller ISPs can colocate their routers in neutral exchanges and take 2 strands of fibre to any of the streets via cross connect. And put a switch or OLT or any other technology which comes later or even a direct patch all the way from one central exchange in the city to end user. I cannot imagine any other solution which can possibly work without either making it more expensive or existing illegal way. Such model can have very least dependency on Govt. as the Govt can do basic pipes and neutral passive infra while leaving service deployment, plans & packages, marketing and various other aspects of a connection to private players. Such infra can very well be used for small cell sites which can service that neighbourhood and hence makes it possible to reduce strain on larger sites. Models like Stokab project followed in Stockholm is impressive. It tells how it can be done by Govt. / Municipal without burning tax payer’s money and in fact generate revenue out of it. Current Indian approach of overhead cables works and is fine for a shorter time but will be a massive “digital slum problem” in near future as more and more people are connected.