SSH key automation at automation workshop!
Next month is SANOG 39 in Dhaka, Bangladesh. SANOG is a South Asian Network Operator Group event and a good place for meeting a number of ISPs, telecom players, Ops team members of content networks, internet exchanges etc. Besides attending the conference, I will be doing a workshop on Network Automation. It will be a four-day workshop covering Containers, Ansible, Gitlab CI/CD pipeline and REST APIs for automation in the workflow.
About the lab
The “lab” for the workshop consists of 30 servers + 30 routers (technically all are just virtual machines, thus 60 VMs in total). I deploy these lab VMs on Hetzner or Vultr. Both do hourly pricing and have a decent REST API to automate VM creation.
I must confess last two runs of the workshop had an ugly system to get SSH keys from the attendees. I used a Google form, made them select their “attendee number” and submit the ssh public key. Next, I copy pasted the SSH public key at a specific location on my laptop and then Ansible read the key, created accounts & pushed the key for everyone.
This kind of flow makes me the bottleneck while 30 attendees stair each other & I do the copy-paste and push of the keys in batches. Plus there is no authentication in that case. I allocate attendee number 1 to someone, ask them to upload their key & mention it’s for attendee number 1. There is no way to validate it and a single rogue attendee can screw things up by uploading random keys for random attendees.
I put some time this Sunday night to put a better system in place.
Automated lab access for attendees
The core idea for this fix is that I get an attendee list a few days before the lab that has the email IDs. I can trust those email IDs and build the rest of the trust system assuming those email IDs are correct and people have access to their email IDs.
The system uses three basic components:
- Budibase for frontend web UI
- Gitlab CI/CD pipeline to accept jobs via Budibase via REST API
- Ansible triggered via Gitlab CI job to push the keys on the devices
Budibase
This a good open-source tool which can create basic CRUD-based apps. A CRUD app is a specific type of software application that consists of four basic operations; Create, Read, Update, and Delete. The majority of things these days fit within this logic and hence I don’t need to write frontend from scratch with an authentication layer. Budibase can be self-hosted, plus it supports REST APIs to import data as well as they can be called to push certain data via REST API.
Gitlab CI/CD pipelines
While it comes from the programming world with the core idea of “testing before commit”, it has added lots of features that it is useful enough to act as a REST API endpoint to spin up docker containers on the request. In my case, it accepts data from Budibase, spins up the Ansible docker container (basically an Alpine container with Ansible installed) and passes the variables to the pre-defined Ansible playbook.
Ansible
Ansible takes care of “speaking to devices”. It can speak to various sets of devices from Linux servers to Cisco routers, from Juniper switches to VyOS devices. The page listing all the modules it supports is endless and a good start when trying to build automation. In my setup, I have a container with Ansible and some basic config. The playbook is passed to the container on the fly via Gitlab CI with the required variables and then it speaks to servers and routers over SSH and deploys the keys.
Here’s how frontend looks like for the attendees:
Once the key and entered and pushed, in the backend there is a budibase automation which passes the attendee ID (picked after the authenticated user logs in with the email ID) and SSH public key to Gitlab.
These bindings are then passed to the REST API:
and when the key is pushed, here is how the Gitlab job seems like:
With this setup, it now takes around 13 seconds for the Budibase to speak to Gitlab, and Gitlab to trigger a docker container with Ansible and Ansible to finish pushing the key to a single device. I think with 3-4 runners taking this load, they should be able to finish this in a couple of mins as people push their keys.
Scope for improvement
I don’t see a secure, easy way to expose Linux servers directly via REST API. VyOS for instance now supports REST API and I might be able to skip Gitlab for that. I can expose the docker engine directly via REST API but it’s not made to be exposed over the internet, it lacks extensive key-based authentication. One can do it in theory over a closed private network / VPN.
Another way to make things very fast can be to use Ansible pull instead of push. I can configure all servers to keep looking at the repo and push the keys in the repo.
Also, this assumes I have to add accounts in bulk for all the attendees. Many of those email IDs would be @gmail.com and one can use Google SSO for the majority of the users.
If you attending SANOG 39 - do buzz me. Would be good to catch up!