30 May

Automated SSL certificate management for private containers

Lately, I have been playing with many tools and as one gets into deploying those tools, SSL comes as a pain point. A large number of web-based tools I use are internal and on a private network. VPN (with OSPF running over FRR) takes care of connectivity but still, it’s good to have SSL on these machines. Non-HTTPs websites are getting more & more ugly with browsers and even things like password managers do not fill the passwords anymore on their own for non-HTTPS websites.

Possible ways to get SSL certificate for these private web applications:

  1. Expose TCP port 80/443 for few seconds to get SSL cert issued from Letsencrypt. Works but too risky & is practical only with IPv6 as each container cannot have it’s own IPv4. Moreover, the IPv6 AAAA record keeps the container exposed all the time. Can be done behind a firewall but not clean.
  2. Use DNS based validation via Certbot’s DNS plugins. Quite powerful once implemented and depending on implementation, does not add any major thread vector. This does need an authoritative DNS setup which supports API access or some workarounds.

DNS Setup to make it work…

Presently my domain “anuragbhatia.com” uses Google’s authoritative DNS. The reason why Google is simply because authoritative DNS hosting comes free with domain and hence I don’t see a reason to pay $3-$6 a year or any other authoritative DNS host. I can run own authoritative DNS but just avoid that since in past website has faced multiple DoS and DDoS attacks on web server & there’s always a risk of attacker attacking a bunch of DNS servers unless I do it at scale (5 servers with each on 10Gbps uplink and some DDoS filtering etc). That clearly is not worth it. šŸ™‚

Now Google Cloud DNS does has API but Google domain’s DNS doesn’t have API access and hence changes cannot be made via any automation tool. So workaround on this, I used Certbot-DNS-Standalone. It’s smart enough to spin up an authoritative DNS server to listen for DNS queries on port 53 specific to a hostname TXT for ACME DNS validation.

So while looking at #2 I also found challenges with Nginx reverse proxy configurations. I tried finding possible Ansible roles but had one or other issue with each one I tried. Official Nginx role for Ansible is pretty extensive but misses Certbot. Plus to be true it’s way more extensive and somewhat not so fun in use with so many variables. This one is easy to use, fewer variables but supports only either no-SSL or Certbot SSL with HTTP challenge only. One cannot define a custom certificate on this one.

So after a few hours of trying various options, I ended up in putting a setup. It uses a bunch of tools (most of them I was using anyway for other things).

  1. Ansible to generate and push SSL certificate from a server with access to authoritative DNS traffic for that zone.
  2. Gitlab (or any Git hosting as such) to keep the Ansible playbooks
  3. Ansible AWX to execute, schedule, alert etc for the certificate generation as well as deployment.

Here’s how it all works

Say I want to have a Nextcloud instance hosted privately (on private IPs) with SSL

Step1: Get it going (can be via Docker or just any form of install)

Image pull from Dockerhub:

anurag@docker:~$ docker pull nextcloud
Using default tag: latest
latest: Pulling from library/nextcloud
Digest: sha256:b1bf8942e85c76aa86362a9ec7eef7ac12ac3c0f3106dd6a0ee89871186dff73
Status: Image is up to date for nextcloud:latest
docker.io/library/nextcloud:latest
anurag@docker:~$

Next, spin up Docker container:

anurag@docker:~$ docker run -d -p 8080:80 nextcloud
f28de86b03290dd656566bf2b2a6441f18f717be42cfc6f96f52afed3945263e
anurag@docker:~$

This makes it live. In this specific case it’s running on 172.17.0.7 port 80.

Step 2: Make changes in DNS

Let’s say I want to use “nextcloud.anuragbhatia.com” as hostname, so for DNS validation, I would need: _acme-challenge.nextcloud.anuragbhatia.com to reply with the required TXT record.

Here it’s pointing to nextcloud.anuragbhatia.com.ssl.anuragbhatia.com. where “ssl.anuragbhatia.com” is special sub-zone delegated to my server. This gives me a way to only host this specific script and not the parent zone DNS with myself. ssl.anuragbhatia.com has an NS record to server7.anuragbhatia.com and server7.anuragbhatia.com has an A record which takes the query to the container. All this is set up once and all once needs per new hostname/application is a CNAME record as shown in the screenshot above.

Step 3: Generate SSL cert via Ansible playbook called via Ansible AWX

I wrote this playbook where I would pass arguments during run time via Ansible AWX (open source version of the Ansible Tower). The variables in the playbook are:

  • ssl_domain – Private domain/hostname for which SSL cert is needed
  • certbot_email – Email needed for certbot for things like renewal. Though because my setup is automated, there’s not much point of mails.
  • deploy_host – This is the host where we want to deploy the SSL certificate. Can be any server or device running SSH and Ansible should have SSH based access to this device. In my case it’s a server running NGINX reverse proxy
  • deploy_host_user – That’s used for rsync for pushing certificate & keys. I have a special user for Ansible for most of my nodes.

I provide these during run on the fly via survey. So screen appears as:

And with that, I launch the playbook. It generates and deploys the SSL certificate on the required host. On transport, it uses rsync + routed IPs over VPN to add an extra layer of security.

An alternate to this system can be prompts in Ansible so that one can keep on using it with command line without need of having AWX though it will have issue with step 5!

Step 4: Generate the required reverse proxy configuration

Just like for last one, I have put this playbook which needs some information via a variable and based on that it generates the reverse proxy configuration.

Variables used:

  • Domain
  • Backend IP – IP address of the machine hosting the application
  • Backend port – Port for which we want NGINX to do the reverse proxy
  • SSL – “yes” or “no” depending on whether SSL is needed to be deployed

Just like step3: I have Ansible AWX configured to ask for these variables during run time.

And there it goes live!

Step 5: Setup automatic SSL certificate renewals

Since Letsencrypt certificates expire in 90 days, it’s important to have automated renewals. For this again I use Ansible AWX instead of cron jobs. Cron jobs usually are painful when they fail and at times are too picky with environment variables.

So I setup a schedule within the same task with pre-defined variables.

Next, I can put pre-defined run time variable values by clicking prompt.

Once saved, this will run this playbook every week. More logical might be 30 days though. And it also gives me option to notify via email, Slack or any similar channel if this playbook fails during running.

Interested in using these playbooks?
Here’s the Github link: https://github.com/anuragbhatia/Ansible-NGINX-Certbot

Misc notes

I might be tweaking following in this setup to avoid having key/cert pair leave the server running script to NGINX proxy server. Ideally private key should stay on the proxy server and never leave it. Need to see if I can make Certbot DNS standalone plugin to work with that. Another option can be to have the entire setup with an option to use a self-signed certificate which Ansible pushes across all machines for trust. Extra work, but saves one from exposing hostnames of all private containers in SSL transparency reports.

Don’t want to run Ansible, self-hosted containers and more but still want to host these fun projects? Check out my brother’s blog post on How to deploy side projects as web services for free.

27 Mar

Letsencrypt – Free signed automated SSL

Last year a really good project Letsencrypt came up. They key objective of this project is to help in securing web by pushing SSL everywhere.
 
Two key cool features

  1. It offer free signed SSL certs!
  2. It helps in setting up SSL via an agent seamlessly without having to deal with CSR, getting it signed & updating web server configuration.

 
At this stage Letsencrypt is itself a Certificate AuthorityĀ and but it’s root certs are yet not in the browser. It’s probably going to take a while till all major browsers get their certificate.
To help on that one of it’s sponsors IdenTrust has signed their intermediate certs. Hence certs signed by Letsencrypt are accepted by all browsers right away. All certs signed by Letsencypt are signed by Letencrypt Authority X1 which have signature from DST Root CA X3 which is accepted by pretty much all popularĀ browsers. You can read more about How it works here.
 
Here’s an example of SSL setup for say “demo.anuragbhatia.com” test domain which is already up and working without SSL. http://demo.anuragbhatia.com shows a plain text page. This is Apache running on Ubuntu server.
The Apache web config is pretty straightforward.

<VirtualHost *:80>
ServerName demo.anuragbhatia.com
DocumentRoot /var/www/demo.anuragbhatia.com
ErrorLog /var/log/apache2/demo.anuragbhatia.com
LogLevel notice
</VirtualHost>

 
 
Step 1 – Grab the Letscrypt agent
git clone https://github.com/letsencrypt/letsencrypt
 
Step 2 – Execute the auto script
./letsencrypt-auto –help
 
This will grab all needed dependencies and will get the agent working.
 
Step 3 – Execute Letsencrypt auto script with it’s Apache plugin
./letsencrypt-auto –apache -d demo.anuragbhatia.com
 
It takes with a quick wizard and in the end I get:

Congratulations! You have successfully enabled
https://demo.anuragbhatia.com
You should test your configuration at:
https://www.ssllabs.com/ssltest/analyze.html?d=demo.anuragbhatia.com

 
And it’s done!
Wizard got me a signed SSL and installed it in the apache config as well.
Screen Shot 2016-03-27 at 7.22.21 PM
 
Screen Shot 2016-03-27 at 7.22.37 PM
 
The agent created an addional Apache config with nameĀ demo.anuragbhatia.com-le-ssl.conf with following content

<IfModule mod_ssl.c>
<VirtualHost *:443>
ServerName demo.anuragbhatia.com
DocumentRoot /var/www/demo.anuragbhatia.com
ErrorLog /var/log/apache2/demo.anuragbhatia.com
LogLevel notice
SSLCertificateFile /etc/letsencrypt/live/demo.anuragbhatia.com/cert.pem
SSLCertificateKeyFile /etc/letsencrypt/live/demo.anuragbhatia.com/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateChainFile /etc/letsencrypt/live/demo.anuragbhatia.com/chain.pem
</VirtualHost>
</IfModule>

 
HereĀ options-ssl-apache.conf plays an important role by using better security options. It’s config:

# Baseline setting to Include for SSL sites
SSLEngine on
# Intermediate configuration, tweak to your needs
SSLProtocol             all -SSLv2 -SSLv3
SSLCipherSuite          ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
SSLHonorCipherOrder     on
SSLCompression          off
SSLOptions +StrictRequire
# Add vhost name to log entries:
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" vhost_combined
LogFormat "%v %h %l %u %t \"%r\" %>s %b" vhost_common
CustomLog /var/log/apache2/access.log vhost_combined
LogLevel warn
ErrorLog /var/log/apache2/error.log
# Always ensure Cookies have "Secure" set (JAH 2012/1)
#Header edit Set-Cookie (?i)^(.*)(;\s*secure)??((\s*;)?(.*)) "$1; Secure$3$4"

 
Some of the limitationsĀ 

  1. Signed SSL certs are valid only for 90 days and have to be renewed.
  2. Wildcard SSL certs are not supported yet.
  3. IPv6 is not supported in the autoconfig setup via client. One can always get certificate manually and use with IPv6 but agent is yet to support IPv6 (which I guess is from next month).

 
You can read more on their excellent documentation here and can also consider checking Presentation by Ashley Jones from PCH at SANOG on All TLS, all the time.
 
Have fun!