I was having this discussion with someone recently on possible software to manage an IXP. Lately, IXP Manager has become the de-facto choice for managing IX. It’s a good tool. Nick and INEX team has built a fantastic open-source tool. But I still feel it’s a bit overloaded for a small 1-2 DC IX operation.
If I have to set up a small to mid-size IX, I would rather do that with arouteserver instead of IXP Manager as I did in case of BharatIX in Mumbai (until it shutdown!). One of the problems with arouteserver is that it can be script intensive and one may need something around it to manage it for things like build config on clients.yml update, regularly update filters etc.
Can that be automated?
Yes! To a large extent, it can be by making use of docker image + Gitlab CI/CD pipeline. Thus end result can be a simple clients.yml sitting on the Gitlab where any addition/removal of clients would trigger a push of the config to the route server running BIRD.
clients: - asn: 15169 # Google ip: - "192.0.2.2" - "2001:db8:1:1::2" - asn: 16509 # Amazon ip: - "192.0.2.3" - "2001:db8:1:1::3" - asn: 55836 # Jio ip: - "192.0.2.4" - "2001:db8:1:1::4" - asn: 9498 # Airtel ip: - "192.0.2.5" - "2001:db8:1:1::5"
So here's a basic Ansible playbook one can run on a fresh VM/server running Ubuntu to provision route server software (BIRD) and a few other basic tweaks.
Live demo with Google, Amazon, Jio & Airtel configured in peering as a demo
Now more fun here can be to generate config an already running IX. Take the example of DE-CIX Mumbai. This is an IX with hundreds of peers. I can point the arouteserver to an IX and it will generate its config. We must remember that this is putting clients.yml without any BGP md5 passwords. If one is using passwords, they have to be manually updated in the clients.yml.
anurag@server7:~$ docker run -it --rm \ -e SECRET_PEERINGDB_API_KEY=$PDB_API_Key \ pierky/arouteserver:latest arouteserver \ clients-from-peeringdb ARouteServer 2022-08-02 15:54:24,073 INFO Checking latest version (current version is 1.17.0)... Loading list of IXs... OK Select the IXP for which the clients list must be built Enter the text to search for (IXP name, country, city): Mumbai ID IXP description 224 IN, Mumbai, National Internet eXchange of India (NIXI Mumbai) 832 IN, Mumbai, (DE-CIX Mumbai) 1623 IN, Mumbai, Amsterdam Internet Exchange Mumbai (AMS-IX Mumbai) 1627 IN, Mumbai, ExtremeIX Mumbai (Extreme IX Mumbai) 2062 IN, Mumbai, Bharat Internet Exchange in Mumbai (Bharat IX - Mumbai) 3463 IN, Mumbai, Smart-Peering Internet eXchange (SP-iXP Mumbai) 3697 IN, Mumbai, (ICX Mumbai) 3708 IN, Mumbai, X3M Infra Pvt Ltd (X3M-IX) 3841 IN, Mumbai, (BGP.Exchange - Mumbai) Enter the ID of the IXP you want to use to build the clients list: 832 Building clients list from PeeringDB Net IX LAN ID 832... asns: AS10075: as_sets: - AS-FGL AS10310: as_sets: - AS-YAHOO AS131442: as_sets: - AS131442:AS-DIGITALNETWORK AS13150: as_sets: - AS-CATONETWORKS-ASNS AS132116: as_sets: - AS132116:AS-ADJANCENCIES AS132203: as_sets: - AS132203:AS-TENCENT (and more...long 2000+ line output)
Here’s how I can update and build RS config out of it:
Few misc points
To make use of it, you can create a project based on my project and update environment variables with information like route server ASN, route server IDs etc.
You should have an SSH keypair configured with the SSH public key on both route servers. And private SSH key provided as Gitlab CI variable - $SSH_PRIVATE_KEY
You should update the IPs of rs01 and rs02 in the inventory file and ensure the runner can connect to it. If those IPs are private, you can set up a Gitlab runner inside your organisation where it can speak to route servers over SSH / TCP port 22. Documentation on setting up Gitlab runner here.
The entire pipeline for DE-CIX Mumbai with 434 peers and two route servers took 4.5mins.
This demo ran against two independent VMs and they were not exactly sitting with other peers in the layer 2 fabric. Hence BGP sessions do not come up. If this is followed in layer 2 IX, sessions will come up!
Use of the peeringdb key is optional but recommended as they tend to rate limit non-authenticated API calls.
For building a docker container with alpine with Ansible in it, one has to use docker inside docker i.e DIND and for that to work, the runner should have the privileged mode enabled.
Besides config update on push, one can schedule Gitlab to automatically update config say in every 4hrs as shown in the last part of the 2nd demo.
Arouteserver is quite extensive and takes care of building filters based on IRR, RPKI, denying transit-free networks, denying “do not route via RS” ASN, adding BGP community support and much more. Do look into arouteserver excellent documentation here.
For running large IX in production, I would probably add a dummy route server where config is pushed onto it, some tests are run and then config is pushed to production. With CI/CD pipeline possibilities are endless!
Project link with sample code here