22 Nov

My home network…

This is a common discussion topic when I tell friends in Indian network operators that I work from home. As soon as I say that, they ask me – “How good is the connectivity at your home?” And of course like all answers in engineering – it depends. 🙂

So I have two links at my home: IAXN and Siti broadband. IAXN is a FTTH connection with 50Mbps down and 25Mbps up, while Siti broadband is a DOCSIS connection with ~60Mbps down and 25Mbps up.

Both have reasonable but not 100% uptime. So to get close to 100% uptime, I use both together. These are consumer grade connections with no BGP. These days many routing platforms support running multiple WAN links for the redundancy reasons. I use Ubnt Edgerouter Lite which my good friend Nat Morris gifted me a while ago. Both links are defined in the “load balancing” where one link acts as primary and other for failover only with multiple routing tables. Next, policy based routing on the LAN VLAN sub-interface takes care of routing packets as needed. This documentation covers the setup in detail. For wifi I use a Asus device which runs purely as a access point in bridged mode with no routing.

Some other things in use at home network:

  • A Raspberry pi 3 stays on a dedicated VLAN & runs multiple site to site Wireguard VPN tunnels (over multiple WAN links) to multiple of my remote locations.
  • It also runs OSPF over FRR to ensure dynamic routing table changes whenever a link is changed. I can switch over traffic by defining the OSPF cost.
  • My server in Munich runs a NGIX proxy & apart from doing various tasks, it also hosts a test URL which does reverse proxy via Raspberry Pi at my home over Siti broadband (only). UptimeRobot monitors that URL for availability and that’s how I monitor my Siti broadband link which is without any public IP and totally behind the CGNAT.
  • Site to site VPNs over multiple links with OSPF taking care of dynamically moving traffic also takes care of things like SNMP monitoring of home devices. I use LibreNMS which is hosted remotely & keeps an eye on home network.
  • Raspberry Pi at home also runs Smokeping where certain predefined targets are moved forcefully out of each WAN link to plot latency. That helps in keeping eye on latency to ISP’s core, as well as upstream telco cores via each link.
  • I also host a node for Galmon project node to keep an eye on (American) GPS satellites, European, Chinese & Russian navigation satellites. The wonderful map here shows the receivers. Lately project is getting good coverage for it’s stats (reference here)
  • I run a DNS resolver at home (again on the raspberry pi)

While there’s auto switching in case of failure or packet loss beyond certain rate on the primary WAN link, I also have a ansible playbook which can be used to tweak the primary/secondary choice & the playbook is available via Semaphone web UI based interface so that my family can switch if they need to.

So the end result is close to 100% uptime (30 seconds outage if primary fails) as well with no irritating wifi switching as well as push notifications on my phone about an outage (via Uptime Robot) for both links. Usually there’s outage once in 30 days not because of WAN links but because I have shut things to clean up the dust.

17 Oct

New rules related to road safety

(A very Indian specific post, International readers feel free to skip this one!)

From last month Motor Vehicles (Amendment) Act, 2019 came to existence. It increases fines for various violations a lot more then what it used to be. I don’t need to cover what has changed since it has been extensively covered in various news articles. One can refer to the Times of India article here which has a nice comparison of old Vs new fine.

This change is a great move and I am little surprised at the stupidity of Indian politicians who even made it an election move to halt it. Due to the federal structure of India, it’s the State Governments who can decide whether or not to apply some parts of it. While there are various sections of society who romanticize with the idea of giving power to states and even more to local city bodies, I found often local folks way more corrupt, idiotic than their National counterparts. The fact that we rank really bad on road safety it’s really needed.

Discussion with a friend

I was once in Muscat for MENOG and I decided to stay for 2 extra after the conference to the roam around but found there was almost no public transport around in a place where I was staying. Plus it was quite expensive to take a taxi. I mentioned this to a good from the UK and he suggested me to rent a car. I promptly replied “no” hinting that it would be hard for me to drive in Oman because there are many “obvious” rules which don’t look obvious to me from the (Indian) system I am used to. He replied back – “If you don’t know the rules, you shouldn’t be driving in India anyway!“. Besides ignoring his amazing level of thinking, I just terminated the discussion. 🙂

My driving has gone from decent to safe over the last few years as a result of looking at how people drive across the world. But I must say from my experience average Indian driving has gone really bad. This is the interesting part – while all other infra like roads, bridges, power infra, telecom has gone much better in say last 10-20 years, average driving has gone from bad to worst. The only place where I have seen driving even worst then ours is in Bangladesh.

Documentation, Digilocker and some thoughts

One of the commonly reported issue with new rules is the fact that it’s very expensive to be driving with a driving license and registration certificate. The new rules have fine of 5,000 INR ($70 USD) for that. That’s actually a real challenge as it beings to dilemma whether to carry driving license regularly at risk of losing it Vs the fine. One thing which comes at the rescue is the Digilocker platform which is a Govt. run platform for pulling the documents from various Central and State Govt. bodies. As per various notifications are given on the Digilocker website (here), it is an accepted valid proof for showing driving license as well as the registration certificate. The traffic policeman checking the documents can anyway validate it from their handheld device using the unique numbers on these. This option is much better than to carry originals since replacement for lost ones is still a bit challenging. The RTO (Regional Transport Office) are often crowded with people applying for new licenses. Though the fact that due to issues in name, missing of surnames, father’s name etc it does breaks sometimes for pulling the document.

Few more things which will help in fixing millions of people to drive better:

  1. Police reforms so that local politicians cannot simply push away a traffic policeman when they fine someone.
  2. Addition of content in the school curriculum for safer driving, showing some scary driving videos & it results to young people will further help.
  3. An overhaul of traffic signal lights across the country. It’s quite poor, doesn’t work many times, doesn’t responds to traffic as it should, designed so poorly that someones a street light pole blocks the visibility of signal etc.
  4. Mandating footpaths across all roads and at least the ones which are built now. This issue results in pedestrians being forced to walk on the road.
  5. Make it mandatory for bicycles riders to use the helmet as well as lights during the night time.
10 Sep

NIXI permits content players!

I am in Chiang Mai, Thailand for APNIC 48 conference. Earlier today attended APIX meeting where many IX members from Asian community gave an update including NIXI i.e National Internet Exchange of India.

As per the update NIXI now allows content players to peer at the exchange. NIXI earlier had a strict requirement of telecom license for anyone to peer but as of now it allows anyone with IP address and AS number to be part of the exchange just like all other exchanges. This is a really good development coming this year after their announcement of the removal of x-y charge. One strange thing remains that their website is still not updated to reflect that which is probably just work in progress. As per representative from NIXI they now openly welcome all content players to peer at NIXI.

What more needs to be done?

Well, a couple more changes are needed.
Here’s what I requested to the representative from NIXI:

  1. Removal of forced multi-lateral peering policy. Forced multilateral is bad idea in modern times. Many large networks (especially the ones dealing with anycast based service) would usually not like to peer at route server.
  2. So far NIXI has discouraged bilateral BGP sessions in the policy. Technically any members can create bilateral sessions but were denied in the policy. That needs to be changed. Bilateral/multilateral peering is a technical decision and should be left to individual networks operators.
  3. NIXI needs to migrate to software-based route servers like bird with auto-config generation to include features like IRR filtering, RPKI filtering, BGP communities support and much more.

Remove forced multilateral, but what about Indian incumbents?

This is an old interesting discussion. Basically due to “forced” multilateral peering policy – everyone is on route servers and that includes incumbents like Tata Communications, Airtel, BSNL and other large networks like Jio and Voda/IDEA as well. The argument there is if multi-lateral peering is not forced, local ISPs won’t be able to peer with these networks. Part of these arguments comes from a mindset which still believes the world’s traffic flows from tier 1 operators to tier 2 and tier 3. A very large part of modern internet traffic flows from content players to eyeball networks (where “eye balls” are). Content players deliver this traffic through a mix of backbone + peering as well as by putting caching nodes inside the ISPs network (like Google’s GGC, Facebook’s FNA, Netflix OCA, Akamai caching node etc). The success of an IX depends on meeting the content players and eyeball players. To connect eyeball players (which are usually spread across the region) one needs circuits i.e dark fibre, DWDM waves, or any sort of transport network. As long as these three things are in place, IX can be a success.

Look at any large IX doing over few hundred Gigabits of traffic or even terabit of traffic and ask does local incumbent telco peers openly at that IX? Does BT openly peers at LINX in London? Do Deutsche Telekom peers openly at DECIX Frankfurt? Does AT&T, Verizon, Comcast peer openly at various Equinix and Coresite exchanges spread across the US? Answer to most of these is no and that is just fine. I would personally hope that these networks do but if they do not it’s not something which blocks the development. In the same way, having Airtel/Tata/Jio/Voda-IDEA at NIXI is great and I would hope they stay but the success of NIXI does not depend on these. As long as transport circuits are available (which they are!) from enough players with competition it would be fine. As of now across India, one can take transport circuit from Airtel, Tata, Voda/IDEA, Powergrid. Railtel, Reliance Communications and more!

Forced multilateral and routing issues…

One recent issue which E2E Networks (a cloud provider based in India) faced was to send traffic to Jio. Suddenly traffic going from E2E to Jio via Tata Comm was going from outside India. As per discussions, routing was tweaked to send traffic out to Jio via Airtel and still, there were issues of routing from outside India. The issue went on for a couple of days and was eventually fixed by speaking to both ends. Issue was there due to ongoing peering issues between Jio – Tata Comm and Jio – Airtel.

Now the funny thing here is that Netmagic (which takes care of routing for E2E) connects to NIXI and Jio also connects to NIXI. During this entire time, both Netmagic (on behalf of E2E) and Jio had enough capacity at NIXI but that could not be utilised because of forced multilateral peering policy. If routes were announced to NIXI route server, that would have probably fixed the Jio issue but would also attract traffic from Airtel/Tata which was not specifically desired due to their known port capacity issues at NIXI.

Ending this post with my quote which I often give in these discussions – “Remember somewhere up the transit path there’s always peering!” 🙂

Disclaimer:

  1. This post is done in my personal capacity and nowhere reflect the views of my employer.
  2. I am on board of the mentioned organisation E2E Networks (more on it here)