Keyserver

January 17, 2020

Overview

May First relies on public key infrastructure (PKI) throughout the network to ensure encrypted and authenticated communications between the public and our servers and between our servers themselves.

In most cases, we use x509 (the standard format for public key certificates used for https and many other applications of transport layer security). For example:

Members connecting via https to any of our sites, including their own web sites.
Members connecting securely to our mail server to send or relay email
Our internal relaying of email, e.g. between our MX servers and the mail servers that store email on behalf of our members,
Communication between our control panel hub and the petal servers that execute the tasks requested.

For x509 certificats, we rely on Let’s Encrypt to verify and sign our certificate signing requests OR in some cases (e.g. in dev or testing mode or for services that are only used internally) we run our own certificate authority and sign our own certificates.

We also use our key server for other PKI tasks, such as generating DKIM keys and certificates.

Requests for keys and certificates

Requests for keys and certificates come from two sources:

Members requests https certificates for websites and DKIM certificates for domains via the members control panel.
Admins indicate a need for X509 certificates via ansible using the keyserver_certs parameer

The key server

To manage all of these keys and certificates, we run a single key server.

The key server is responsible for generating, renewing, distributing via ssh and running any renewal triggers required for every certificate requested.

When ansible runs, it generates a single file, /etc/ssl/ansible.certs.yml, listing all certificates it need.

When the control panel needs a certificate, it creates a file in /etc/ssl/red.certs.d for each request.

Every time a new request is added, mf-certs-process is run. mf-certs-process looks for any certificate in the configuration files that either does not have a key/cert pair or has an expired cert.

Depending on whether the certificate is set to letsencrypt or mfca, the script will either call certbot or it will generate a cert/key pair using our ca. When creating a letsencrypt pair, it will configure the cert with a renewal hook of mf-certs-distribute <identifier>.

For mfca cert requests, the script calls mf-certs-distribute every time a new certificate is created or renewed.

The mf-certs-distribute copies the keys and certs to the configured servers and runs and renewal trigger commands - all via ssh. For example, certificates for https web sites would be copied to all webstore and webproxy servers and the renewal trigger would restart nginx or apache2.

The default-web-proxies.yml file

The /etc/ssl/ansible.certs.yml and /etc/ssl/red.certs.yml file contain instructions about every certificate we manage, including which hosts the keys and certificates should be distributed to.

Many of these configurations specify a distribute-hosts-file which is set to: /var/lib/red/default-web-proxies.yml. Since our list of web proxy servers may change from time to time, we keep this info in a single file to faciliate updates.

And, our mf-cert-distribute tool helps with that tasks. For example, when adding a new web proxy, you can run mf-cert-distribute --all --send-to-server <new-web-proxy> --copy-only to copy all keys to a new web proxy. See the nginx page for more information on adding a new web proxy.

Validation

The mfca certificates do not require validation since we sign them ourselves.

Lets encrypt certificates are validated via one of two challenges:

All control panel request and most others are handled via HTTP-01. The keyserver runs a web server that accepts requests from any domain. Our webproxy servers transparently redirect all requests to /.well-known/acme-challenge to the key server.
DNS-01: For domains that can’t be validated via http (such as mail.mayfirst.org) we use DNS-01 (TBD - we will need to write some simple code - see this comment for how - replacing nsupdate with an api call to our control panel).

Renewals and maintenance

Lets Encrypt certificates are renewed by a systemd timer that calls certbot.

The mfca certificates are renewed once a day by a systemd timer that calls mf-cert-process.

The mfca certificates require additional maintenance if our mfca certificate expires.

We maintain a password-less certificate authority private key on key001 in /etc/ssl/private/mfca.key (also stored in our ansible vault) and a corresponding certificate authority public key that is distributed to /usr/local/share/ca-certificates/mfca.crt on all servers (and stored in our hosts.yml file in our inventory repository).

To renew:

Generate a certificate signing request:

openssl x509 -x509toreq -in /usr/local/share/ca-certificates/mfca.crt -signkey /etc/ssl/private/mfca.key -out new.csr

Sign it with a 3 year expiration date:

openssl x509 -req -days 1024 -in new.csr -signkey /etc/ssl/private/mfca.key -out mfca.pem

Copy the contents of mfca.pem into the mayfirst_mfca_cert variable in our hosts.yml file.
Copy to all servers: sower --live playbook --tags mayfirst-mfca-cert

Authentication and permissions

Each server that needs a certificate must include the keyserver_consumer: true setting in it’s ansible file. That setting triggers the creation of a local user called keypusher configured to allow access by the root user on the key server. The user has sudo privileges to run any script starting with /usr/local/sbin/mf-certs-renew-*. All servers have mf-certs-renew-base which copies new keys and certs to /etc/ssl/mayfirst/<name>/{privkey.pem, fullchain.pem}. Each requested certificate can include one or more additional renew scripts to be executed as root.

Example

Here’s an example of what you might include in an ansible server file:

# Ensure your renew scripts are copied to the server. Renew scripts
# live in the mayfirst role's files/cert-renew-scripts directory.
keystore_renew_scripts:
  - mf-cert-renew-kibana
# Trigger the creation of the `keypusher` user on this host.
keystore_consumer: true
keystore_certs:
  - name: report.mayfirst.org
    monitor_port: 443
    common_names:
      - report.mayfirst.org
      - www.report.mayfirst.org
    ca: letsencrypt
    challenge: http
    distribute_hosts:
      report001.mayfirst.org:
        - /usr/local/sbin/mf-cert-renew-base
        - /usr/local/sbin/mf-cert-renew-kibana
      webproxy001.mayfirst.org:
        - /usr/local/sbin/mf-cert-renew-base
        - /usr/local/sbin/mf-cert-renew-nginx

Trouble shooting

We have a test file that can be used to easily determine if a given domain is properly configured to work with our key server, simply run:

curl http://YOURDOMAIN.ORG/.well-known/acme-challenge/test

You should get the content:

This domain should pass a Let’s Encrypt challenge.

Exceptions

In some cases it’s preferable to run Letsencrypt locally on a server.

For example, a server that only runs postfix will have an A record pointing to itself, not to the nginx proxy, so the http challenge won’t work.

In these cases, you can configure the server with ansible veriables indicating that it should run letsencrypt and handle certificate creation via the certbot standalone server option.

For example:

letsencrypt_certs:
  - name: monitor001.mayfirst.org 
    common_names:
      - monitor001.mayfirst.org
    renew_hooks:
        - /usr/local/sbin/mf-cert-renew-base
        - /usr/local/sbin/mf-cert-renew-postfix

Monitoring

If you include the monitor_port key and set the value to the port that is listening via tls, then our simplemonitor server on montitor002 will pick it up and automatically monitor the certificate, giving us a warning if it will expire in 7 days or less.