IX management via Gitlab CI!

Isp-Column

I was having this discussion with someone recently on possible software to manage an IXP. Lately, IXP Manager has become the de-facto choice for managing IX. It’s a good tool. Nick and INEX team has built a fantastic open-source tool. But I still feel it’s a bit overloaded for a small 1-2 DC IX operation.

If I have to set up a small to mid-size IX, I would rather do that with arouteserver instead of IXP Manager as I did in case of BharatIX in Mumbai (until it shutdown!). One of the problems with arouteserver is that it can be script intensive and one may need something around it to manage it for things like build config on clients.yml update, regularly update filters etc.


Can that be automated?

Yes! To a large extent, it can be by making use of docker image + Gitlab CI/CD pipeline. Thus end result can be a simple clients.yml sitting on the Gitlab where any addition/removal of clients would trigger a push of the config to the route server running BIRD.

Sample clients.yml

clients:
  - asn: 15169 # Google
    ip: 
    - "192.0.2.2"
    - "2001:db8:1:1::2"    
 
  - asn: 16509 # Amazon
    ip: 
    - "192.0.2.3"
    - "2001:db8:1:1::3"    
 
  - asn: 55836 # Jio
    ip: 
    - "192.0.2.4"
    - "2001:db8:1:1::4"    
 
  - asn: 9498 # Airtel
    ip: 
    - "192.0.2.5"
    - "2001:db8:1:1::5"

So here's a basic Ansible playbook one can run on a fresh VM/server running Ubuntu to provision route server software (BIRD) and a few other basic tweaks.

Next, one can make use of the arouteserver docker image to build the configs. Here’s how all this ties together a Gitlab CI config file (.gitlab-ci.yml).

Live demo with Google, Amazon, Jio & Airtel configured in peering as a demo

Now more fun here can be to generate config an already running IX. Take the example of DE-CIX Mumbai. This is an IX with hundreds of peers. I can point the arouteserver to an IX and it will generate its config. We must remember that this is putting clients.yml without any BGP md5 passwords. If one is using passwords, they have to be manually updated in the clients.yml.

anurag@server7:~$ docker run -it --rm \
-e SECRET_PEERINGDB_API_KEY=$PDB_API_Key \ 
pierky/arouteserver:latest arouteserver \
clients-from-peeringdb


ARouteServer 2022-08-02 15:54:24,073 INFO Checking latest version (current version is 1.17.0)...
Loading list of IXs... OK

Select the IXP for which the clients list must be built
Enter the text to search for (IXP name, country, city): Mumbai
     ID  IXP description
    224  IN, Mumbai, National Internet eXchange of India (NIXI Mumbai)
    832  IN, Mumbai,  (DE-CIX Mumbai)
   1623  IN, Mumbai, Amsterdam Internet Exchange Mumbai (AMS-IX Mumbai)
   1627  IN, Mumbai, ExtremeIX Mumbai (Extreme IX Mumbai)
   2062  IN, Mumbai, Bharat Internet Exchange in Mumbai (Bharat IX - Mumbai)
   3463  IN, Mumbai, Smart-Peering Internet eXchange (SP-iXP Mumbai)
   3697  IN, Mumbai,  (ICX Mumbai)
   3708  IN, Mumbai, X3M Infra Pvt Ltd (X3M-IX)
   3841  IN, Mumbai,  (BGP.Exchange - Mumbai)

Enter the ID of the IXP you want to use to build the clients list: 832
Building clients list from PeeringDB Net IX LAN ID 832...
asns:
  AS10075:
    as_sets:
    - AS-FGL
  AS10310:
    as_sets:
    - AS-YAHOO
  AS131442:
    as_sets:
    - AS131442:AS-DIGITALNETWORK
  AS13150:
    as_sets:
    - AS-CATONETWORKS-ASNS
  AS132116:
    as_sets:
    - AS132116:AS-ADJANCENCIES
  AS132203:
    as_sets:
    - AS132203:AS-TENCENT


(and more...long 2000+ line output)

Here’s how I can update and build RS config out of it:


Few misc points

  1. To make use of it, you can create a project based on my project and update environment variables with information like route server ASN, route server IDs etc.

  2. You should have an SSH keypair configured with the SSH public key on both route servers. And private SSH key provided as Gitlab CI variable - $SSH_PRIVATE_KEY

  3. You should update the IPs of rs01 and rs02 in the inventory file and ensure the runner can connect to it. If those IPs are private, you can set up a Gitlab runner inside your organisation where it can speak to route servers over SSH / TCP port 22. Documentation on setting up Gitlab runner here.

  4. The entire pipeline for DE-CIX Mumbai with 434 peers and two route servers took 4.5mins.

  5. This demo ran against two independent VMs and they were not exactly sitting with other peers in the layer 2 fabric. Hence BGP sessions do not come up. If this is followed in layer 2 IX, sessions will come up!

  6. Use of the peeringdb key is optional but recommended as they tend to rate limit non-authenticated API calls.

  7. For building a docker container with alpine with Ansible in it, one has to use docker inside docker i.e DIND and for that to work, the runner should have the privileged mode enabled.

  8. Besides config update on push, one can schedule Gitlab to automatically update config say in every 4hrs as shown in the last part of the 2nd demo.

  9. Arouteserver is quite extensive and takes care of building filters based on IRR, RPKI, denying transit-free networks, denying “do not route via RS” ASN, adding BGP community support and much more. Do look into arouteserver excellent documentation here.

  10. For running large IX in production, I would probably add a dummy route server where config is pushed onto it, some tests are run and then config is pushed to production. With CI/CD pipeline possibilities are endless!

Project link with sample code here