11 May

Building redundancy on home network

I posted about the home network in multiple other posts in past. I recent time I switched from Microtik SXT Lite 5 to Power Beam PBE-M5-400. This gave me a jump from 16dbi to 25dbi which gives much sharper beam. I also got a harness & climbed BTS myself (after getting permission from the manager) this time to switch gear. I think I can do a better job than wasting time in finding guys from local WISPs to do it. 🙂


Also, Essel Group launched Siti broadband in my home area and they are using DOCSIS. The network is overall fine though initially faced many outages due to fibre cuts here & there. As of now, the connection is reasonably stable. I am paying 860Rs/month ~ $14 for 10Mbps uncapped link which gives me 10Mbps down and 1.5Mbps up. From a price point, it’s an excellent connection to have for redundancy reasons. Now as the connection is stable enough to explore auto-failover. For last few months I took both primary links as well as backup links to the router in the form of tagged VLANs and used to push specific traffic based on source IP (device at home) or destination IP/port combination using policy based routing.



Here both links drop on the TP-Link router which I use as a layer2 switch. I tag both links on different VLANs and carry them to my room over a single cable. TP-link 1043nd flashed with OpenWRT and it allows me to do simple layer 2 aggregation and maintains 1Gig link with other switch placed in my room.

It’s tricky to do an auto-failover in such static setup where I am not using BGP and hence WAN IP changes when the connection is switched. I use Ubiquity Edge router as core router at home and it comes with the option of “load balancing” features where one can load balance or simply put a secondary interface in failover mode.


Here’s how the config looks like now:

(Note: VLAN10 / routing table1  – Primary link and VLAN20 / routing table 2: Secondary link)


So this is simply putting two different routing tables in the router besides the main table known as “main”. Next, is the load balancing config:


So here I have eth2.20 defined for failover only and it uses routing table 2 while the primary link is eth2.10 which uses the main table. It’s basically sending 6 pings (one in every 5 seconds) and hence if 6/6 fail during 30 seconds long outage, a primary link would be considered dead and traffic will move to secondary link. The further router will keep on trying to ping the defined IP and once there are 12 successful pings (one in every 5 seconds) in a 1min period, it would be assumed live again. New sessions will switch over to primary while existing ones will stick with secondary to avoid outage on them.


Next, load balance config is called on a firewall modify instance:

and this “SOURCE_ROUTE” is called on the LAN-facing interface to apply this policy on the interface:


And that’s all about it. It ensures that regular internet usage (not SSH sessions), streaming, Chromecast, etc all can stay live with a maximum impact of 30 seconds in case of the issue on the primary link.


Some misc notes:

  1. If primary link goes down, IPv6 would be still broken and I have yet to put a script to disable IPv6 on LAN in the case of an outage on the link.
  2. I noticed Ubnt doesn’t behave well in terms of failover if I do not specify IPv4 test address. It tends to use a test string which was pointed to Amazon CDN (which is fine btw) but as a primary link fails, DNS resolution also fails and devices seem to be re-trying DNS resolution instead of assuming failure instantly.
  3. I focused on testing primary link with an IP far away in Europe. The secondary link does not really matter because it’s just not being used and the case when it is being used it is the only option. Hence extensive testing makes no sense on the secondary link.


Here’s output of this load-balancing setup:



Sidenote: I am in Bangalore for Rootconf 2017. I would be presenting about Eyeball routing measurement using RIPE Atlas. If you are around in Bangalore, drop me a message and it would be great to meet!

13 Jan

Vyatta based VyOS – Linux based network OS

VyOS is quite interesting OS. It’s a open source Linux based network operating system based on Vyatta. It’s config style seems bit like JunOS in terms of hierarchy and set/edit/delete options while editing configuration.


Can one use it in a small ISP or a Corporate LAN setup? 

Someone asked me recently if we can have complete open source based router in smaller network doing basic stuff. Not with not-so-streamlined Linux shell but networking OS where network engineers favorite tool “?” works in CLI with options.

Let’s take a possible case with bunch of routers, a server with speedtest-mini running on it and end desktop with Ubuntu-desktop on it along with VyOS based router. Goal here is to have basic features to work (to start with!). I am conducting this test and setup on the VM infrastructure at home but that should have zero impact/configuration of network devices and hence not going to focus on that part. All devices including server, desktop and router are pretty much running on virtual machines or KVM containers.



To configure and test:

  1. Configuration of interfaces and basic static routing with reachability
  2. Source-NAT on RTR01 for LAN side connectivity with DHCP
  3. Traffic shaping on RTR01 on per user basis
  4. IPv6 autoconfig
  5. Basic firewalling to protect server01


Note: I am treating 10.x.x.x range here as public IP wan range in demo while as LAN IPs which would be NATted behind WAN IP.


1.Static routing with reachability

Configuring VyOS is very much like that of JunOS – basic edit to go inside to hierarchal levels and set to add to config, delete to delete from config.


Configuration on transit router’s interface


Configuration on RTR01 interfaces


Transit router is without any default route, while server01 & RTR01 have default towards transit and as usual end user desktops have default towards RTR01.


This gives basic connectivity from RTR01 all way up to the server01.


Connectivity test


2.Source-NAT on RTR01 for LAN side connectivity

So we have eth1 on the transit router facing on LAN side. Just like all other LANs, it would need a combination of DHCP + NAT of private against WAN IP for outgoing packets.


Configuration DHCP


Note: Here subnet and default router IP belongs to IP used on eth1 interface. Now we have as LAN pool, and we need to NAT it against WAN IP which is on eth0 –

NAT configuration

This ensures that all packets with source IP from pool going to outside world via eth0 – WAN would be “re-written” to  source IP address of WAN.

Once this is done, Ubuntu-desktop shows IP which it learnt from RTR01.

Ubuntu Desktop - learnt IP via DHCP



Testing it’s connectivity to end destination server01 which is on WAN IP

Connectivity test from desktop


Quick check on NAT table on RTR01


3. Traffic shaping on RTR01 on per user basis

All works well. Now, I am running speedtest-mini script on the server01. So browsing on end desktop to test speeds.

(Note: As I said this is virtualized environment and hence extremely likely I am going to hit resource limit much before actual theoretical speeds of port. Aim of this test is just sample and to assist in bandwidth shaping.)


Speedtest on Desktop

Screen Shot 2016-01-13 at 2.10.32 AM


Screen Shot 2016-01-13 at 2.12.04 AM



This shows quite decent speeds. Now coming to traffic shaping, let’s shape everyone on LAN to 10Mbps symmetric. Here we can make use of “traffic shaper“. One key thing with traffic shaper is that it works only in OUT direction and hence it can shape packets going in out direction from port. Thus to have symmetric effect, we can apply it on both eth0 – WAN (for capping uploads) and eth1 – LAN (for capping downloads).


Next, we need to call appropriate shaper on interfaces. Calling “User-Upload-Cap” on eth0-WAN in outbound direction and “User-Download-Cap” in  outbound in eth1-LAN direction.


Speedtest after cap

Speedtest after cap


And capping works! One can more fine tune it as per requirement like capping just specific traffic for specific set of users etc.


4. IPv6 autoconfig

So next goes is to have full IPv6 auto configuration. I would also dual stack interfaces of server01 to make IPv6 work end to end between server01 to desktop.

Following RFC 3849, I will use 2001:DB8::/32 for this sample configuration / documentation purpose.

2001:DB8::/32 – Main allocation

2001:DB8:1::/48 – Transit Router Pool
2001:DB8:1:1::/64 – Connectivity with server01
2001:DB8:1:2::/64 – Connectivity between Transit Router and RTR01
2001:DB8:2::/48 – IPv6 pool for RTR01 routed to it
2001:DB8:2:a::/64 – LAN pool for allocation to end customers


Dual stacking eth0 interface of Transit-Router which is facing server01 & eth1 facing RTR01


Next, routing 2001:DB8:2::/48 behind RTR01 (2001:DB8:1:2::2)



Configuration RTR01 interfaces with IPv6


Setting up router advertisements

Checking now on end user desktop

IPv6 connectivity test

(This shows IPv6 getting automatically configured on the end user’s desktop)


IPv6 connectivity test


Speedtest on IPv6


Speedtest over IPv6 is also capped since cap was based on interfaces and traffic flow and hence works on both IPv4 and IPv6 world.


5.Basic firewalling to protect server01

In current configuration, server01 has all open ports. Let’s try to allow only and only tcp port 80 traffic and drop everything else. Out ports from server01 to outside should work.


Step 1 – Create firewall rule to drop all traffic by default and just allow tcp port 80 and established + related connections so that return traffic (initiated from server) could work.


Step 2 – Next, apply this policy on the “out” direction on the interface connecting the server01 so packet going towards the server can be filtered.


And finally testing and looking at firewall statistics



All works pretty well. We can surely use such box in a small network.


Next logical step – using it for more cool features like full redundancy using VRRP, OpenVPN tunnels, BGP with route-maps etc.



Note: I would presenting a paper on “Disconnected Network Islands” at SANOG 27 on 25th. Meet and greet if you are around! 🙂