Monitoring

For a while, I have been looking for a smokeping alternative for latency monitoring from different servers spread around. While smokeping has survived well over time, in 2023 it feels like an outdated package, with limited options, lacks federation etc. This post from Karan Sharma / Zerodha on “Monitoring my home network” was exciting. His setup included a telegraph agent on a local server, Prometheus to scrap data and Grafana to draw latency data. I explored doing the same but in a distributed manner a bunch of servers spread around. After some tries, I didn’t like Telegraf. Don’t get me wrong - it’s a good “agent” to run on Linux servers but is primarily designed assuming push target against a time series database like InfluxDB which created it. I am still exploring using it for a different use case (which is pulling SNMP data from switches).

Background

I am running my own authoritative DNS servers for the last few years. In earlier stages I just used registrar-provided DNS, later moved to “Cloud provider” provided DNS and ultimately settled for running my own auth DNS.

Two major requirements pushed me to self-host auth DNS:

1. Requirement of REST API for DNS needed by the web servers to resolve Letsencrypt certbot DNS-based challenge. This allows me to have internally hosted tools with Letsencrypt issues TLS certificates instead of self-signed ones. The API access is mostly missing in the registrars hosted DNS.
2. Occasional DDoS on my blog. There have been occasional DDoS on my blog (probably from random people who enjoy doing volumetric attacks). This always worried me about DNS bills during DDoS, especially for low TTL records. The last hosted DNS option I had over a year ago was Google Cloud DNS and they charge $0.40 per million queries per month. This can add a significant amount to the bill if under massive DDoS resulting in many millions of DNS queries. Plus per zone 20 cents charge gets expensive at a scale with a half a dozen domains.

After exploring a few options I settled for running PowerDNS with MySQL backend. This is kind of a comfort zone since I ran similar systems for my employers in past and it worked well. PowerDNS is a great option for authoritative DNS as it has nice documented REST API, CLI utility pdnsutil for easy high-level scripting, supports a bunch of backends to store DNS records from BIND like text files to MySQL. It is also good (automated) support to handle DNSSEC for signing the zone.