11 May

Building redundancy on home network

I posted about the home network in multiple other posts in past. I recent time I switched from Microtik SXT Lite 5 to Power Beam PBE-M5-400. This gave me a jump from 16dbi to 25dbi which gives much sharper beam. I also got a harness & climbed BTS myself (after getting permission from the manager) this time to switch gear. I think I can do a better job than wasting time in finding guys from local WISPs to do it. ūüôā
Also, Essel Group launched Siti broadband in my home area and they are using DOCSIS. The network is overall fine though initially faced many outages due to fibre cuts here & there. As of now, the connection is reasonably stable. I am paying 860Rs/month ~ $14 for 10Mbps uncapped link which gives me 10Mbps down and 1.5Mbps up. From a price point, it’s an excellent connection to have for redundancy reasons. Now as the connection is stable enough to explore auto-failover. For last few months I took both primary links¬†as well as backup links to the router in the form of tagged VLANs and used to push specific traffic based on source IP (device at home) or destination IP/port combination using policy based routing.

Here both links drop on the TP-Link router which I use as a layer2 switch. I tag both links on different VLANs and carry them to my room over a single cable. TP-link 1043nd flashed with OpenWRT and it allows me to do simple layer 2 aggregation and maintains 1Gig link with other switch placed in my room.
It’s tricky to do an auto-failover in such static setup where I am not using BGP and hence WAN IP changes when the connection is switched. I use Ubiquity Edge router¬†as core router at home and it comes with the option of “load balancing” features where one can load balance or simply put a secondary interface in failover mode.
Here’s how the config looks like now:
(Note: VLAN10 / routing table1  РPrimary link and VLAN20 / routing table 2: Secondary link)

anurag@router01# show protocols static table 1
 description "Primary Link"
 route {
     next-hop $Provider1 - Router {
anurag@router01# show protocols static table 2
 route {
     next-hop $Provider2 - Router {
         description "Secondary Link"

So this is simply putting two different routing tables in the router besides the main table known as “main”. Next, is the load balancing config:

anurag@router01# show load-balance group Home-HA-Zone
 interface eth2.20 {
     route {
         table 2
     route-test {
         initial-delay 60
         interval 5
         type {
             ping {
 interface eth2.10 {
     route-test {
         count {
             failure 6
             success 12
         initial-delay 60
         interval 5
         type {
             ping {
 lb-local enable

So here I have eth2.20 defined for failover only and it uses routing table 2 while the primary link is eth2.10 which uses the main table. It’s basically sending 6 pings (one in every 5 seconds) and hence if 6/6 fail during 30 seconds long outage, a primary link would be considered dead and traffic will move to secondary link. The further router will keep on trying to ping the defined IP and once there are 12 successful pings (one in every 5 seconds) in a 1min period, it would be assumed live again. New sessions will switch over to primary while existing ones will stick with secondary to avoid outage on them.
Next, load balance config is called on a firewall modify instance:

anurag@router01# show firewall modify SOURCE_ROUTE rule 30
 action modify
 description "High Availability on Production LAN"
 modify {
     lb-group Home-HA-Zone

and this “SOURCE_ROUTE” is called on the LAN-facing interface to apply this policy on the interface:

anurag@router01# show interfaces ethernet eth2  vif 2 firewall in modify

And that’s all about it. It ensures that regular internet usage (not SSH sessions), streaming, Chromecast, etc all can stay live with a maximum impact of 30 seconds in case of the issue on the primary link.
Some misc notes:

  1. If primary link goes down, IPv6 would be still broken and I have yet to put a script to disable IPv6 on LAN in the case of an outage on the link.
  2. I noticed Ubnt¬†doesn’t behave well in terms of failover if I do not specify IPv4 test address. It tends to use a test string which was pointed to Amazon CDN (which is fine btw) but as a primary link fails, DNS resolution also fails and devices seem to be re-trying DNS resolution instead of assuming failure instantly.
  3. I focused on testing primary link with an IP far away in Europe. The secondary link does not really matter because it’s just not being used and the case when it is being used it is the only option. Hence extensive testing makes no sense on the secondary link.

Here’s output of this load-balancing setup:

anurag@router01:~$ show load-balance watchdog
Group Home-HA-Zone
  status: Running
  failover-only mode
  pings: 2857
  fails: 0
  run fails: 0/3
  route drops: 0
  ping gateway: - REACHABLE
  status: Running
  pings: 2744
  fails: 6
  run fails: 0/6
  route drops: 0
  ping gateway: - REACHABLE
anurag@router01:~$ show load-balance status
Group Home-HA-Zone
  interface   : eth2.10
  carrier     : up
  status      : active
  gateway     : $Provider1
  route table : 201
  weight      : 100%
      WAN Out : 11767
      WAN In  : 14446
    Local Out : 2
  interface   : eth2.20
  carrier     : up
  status      : failover
  route table : 2
  weight      : 0%
      WAN Out : 0
      WAN In  : 0
    Local Out : 0

Sidenote: I am in Bangalore for Rootconf 2017. I would be presenting about Eyeball routing measurement using RIPE Atlas. If you are around in Bangalore, drop me a message and it would be great to meet!

03 Nov

Last mile broadband technology for PRESENT!

Came across this impressive cover of last mile broadband issues in¬†Orcas Island in Washington state in Arstechnica.com. It’s very true on how so many areas are just not served and likely will never be served because when you have large telecom players bidding for billion dollar worth of Spectrum, all they care next for is very high value returns. And if they do not see those kind of returns, areas stay unserved. India has even poor story where it’s challenging to get wired broadband in most areas of country including key metro cities.
Few months back I posted story of my home fixed wireless connection and it works great. Sharing video story of¬†Orcas Island citizens about their broadband issues and how they fixed them with fixed wireless. This is a technology which is already somewhat used and needs to be used right away for most of less populated areas and villages. It’s not fiber but yes it makes much more economical sense to get more people with 30-40Mbps symmetric pipes right away rather than waiting for years and years for fiber connection (and paying a hefty $5 billion on a project like NOFN/BBNL) or worst – giving people fiber connections with 5-10Mbps of plans!

Oh and btw I would be presenting a small research work at bdNOG 4 next week in Bangladesh. Meet and greet if you are around in Bangladesh attending the event!

28 Sep

Good bye BSNL (AS9829) | New link at home!

A blog post dedicated to BSNL AS9829. It just¬†tried so hard to become as irrelevant as it can¬†from everyone’s life (and that doesn’t excludes me now).
So what really is BSNL btw?

  • A Govt of India telco sitting at a extensive fiber of over 600,000 Kms across the country (staying just¬†unused¬†and unavailable for anyone’s use!)
  • A telco which has an extensive¬†last mile copper (which is very poorly maintained and barely works!)
  • A backbone with over¬†200Gbps of IP transit capacity (which completely sucks due to¬†rotten routing)
  • An integrated telecom provider offering services from landline to DSL broadband, from leased line to datacenter services! (out of which everything fails miserably from product line to technical ground level operations)
  • An extensive manpower (which is terribly arrogant and from top to ground level staff anyone barely works!)
  • Although telecom industry just boomed, it¬†went from 10,000 crore profits in 2004 to 8000 crore losses in 2015. And still politics goes around it!
  • While private sector was busy with focus on 4G LTE deployment, BSNL’s market share dropped below 10% in 2014
  • While private sector firms like Sterlite, Radius Infratel focused on FTTH rollouts, BSNL rolled out FTTH plans for 4000 INR/month for 50GB cap and FUP speed of (amazing) 512Kbps to ensure no one uses it
  • While Reliance Jio is about to come, Airtel is extensively launching 4G LTE, cool companies like ACT¬†are getting¬†more investment, BSNL is putting 6000 crore in public wifi infrastructure to give few mins of free wifi and with hop of users paying it afterwards. (Wow?!)

All above tells nothing but ways in which BSNL is 100% screwed up for now. I don’t expect it to ever pick up again. Politically, technically, and fundamentally it’s a mess.
I became BSNL broadband user in 2008 and it has been over 7 years of (painful and terrible) experience with them. As a company which put so much of infrastructure to connect India worked extremely hard to do as many stupid things as possible. For me trouble remained that in my city they were only wired telecom provider for retail services.
Last month¬†I got a long haul circuit from Airtel (provisioned on fiber) between my city and a friend’s ISP PoP for 10Mbps bandwidth. Circuit is delivered at a Airtel¬†BTS site location (slightly¬†away from my home) and I have installed Microtik SXT Lite 5’s shooting link from there to my home (around 1km link with clear LoS). This is a usual long range fixed wireless RF link over un-licensed 5.8Ghz band. (Thankyou govt. of India for delicensing¬†it in 2007 and making available for public use). Thanks to companies like Microtik and Ubiquiti for opening up world of good fixed wireless radios and antennas which really work great and are available for quite good prices. I got pair of SXT Lite5’s from Amazon.in at 7700 INR (~$116).
Fortunately BTS site has a private WISP tower and the owner of tower agreed to let me use his tower for my radio for reasonable price.

Some statistics about my new link


Airtel BTS site

Airtel BTS Site

LoS of tower (from home)


Radio at my rooftop

Radio on rooftop
(Water tanks pipes were tall enough that I didn’t had to mount any pole and used those pipes)

Closer look

Radio at home

Link quality checks

Radio link stats
I am getting end to end bandwidth of around 35Mbps between radios (while provisioned bandwidth is 10Mbps on backend). I am using 5Mhz of channel bandwidth with 802.11 protocol and usual WPA2-PSK works to have encryption between radios.
End to end latency between Rasberry Pi (connected via wired to my home router) to other end radio:

PING ( 56(84) bytes of data.
64 bytes from icmp_req=1 ttl=63 time=1.57 ms
64 bytes from icmp_req=2 ttl=63 time=1.03 ms
64 bytes from icmp_req=3 ttl=63 time=1.02 ms
64 bytes from icmp_req=4 ttl=63 time=1.02 ms
64 bytes from icmp_req=5 ttl=63 time=1.24 ms
--- ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4005ms
rtt min/avg/max/mdev = 1.023/1.179/1.570/0.214 ms

And lastly speedtest from a server far away from here:
(Note: Hided ISP name to avoid un-needed DDoS attack on them which are hitting my blog from few weeks)
Some thoughts on fixed wireless links

  1. Work great if LOS and free channels are there. India does has serious problem of very low unlicensed open spectrum permitted for outside use.
  2. Hard to predict capacity for large country like India – may work somewhere, may not somewhere.
  3. WISP stupidly use 20Mhz and HT beams of 40Mhz when even 5Mhz can do job for many of their links. (More “bandwidth” usage = reducing channels for others + more potential chances of interference).
  4. Links work well given 1st Fresnel zone is cleared. Special thanks to my friend Brough Turner for pointing this out. He runs an ISP based on this technology in Boston & surrounding areas. (Checkout netblzr)
  5. Fixed wireless is NOT mobile wireless (understand the difference!).
  6. Some other successful ISPs using this technology – MonkeyBrains in San Francisco (on unlicensed spectrum) and Webpass (using microwave links).
  7. Tikona in India used it a bit but with mesh to increase coverage and eventually got a network with latency & packet loss issues. Wireless links work well but for point to point and very little point to multi-point. Not good choice for a large network with wireless nodes acting as transport in between. Indian media as usual stupidly took technology as swiss knife solution to broadband issues. (checkout NDTV review of Tikona).
  8. Tech and NOG community across India have to support for more un-licensed spectrum for use in India. (Excellent article on this here)
  9. I am overall motivated by excellent paper – America’s Broadband Heros which gave very detailed understanding of technology and limitations
  10. I am overall happy with 2.5x increase in download speed but a whopping 20x increase in upload speeds. Fixed wireless has a good edge over upload speeds when compared to DSL

Ending this blog post with Cacti graph of my home broadband connection for last one month.¬†There’s high amount of systematic transfers of routing table data and some other stuff. I do keep a Rasberry Pi running all the time as home server. ūüôā
Home Broadband Graph