Overview

Hacking on a server configuration tool is hard because you need a fleet of servers to test on.

To make it easier, our ansible repository comes with tools that allow you to create a fleet of servers residing just on your local machine using podman.

When you have completed these tasks you will end up with:

• An authoritative DNS server (nsauth001.mayfirst.dev) set with all domain names in use (the dev sites use the domain: mayfirst.dev)

• A caching DNS server (dnscache001.mayfirst.dev)- with all servers configured to use via /etc/resolv.conf

• A MySQL (mysql001.mayfirst.dev) and Postgres (psql001.mayfirst.dev) server

• A key server (key001.mayfirst.dev) configured to create self-signed x509 certificates and push those keys out to the servers that need them.

• A nginx web proxy server (webproxy001.mayfirst.dev) configured to proxy all web traffic.

• A superfoo server (superfoo001.mayfirst.dev) - which enables the superfoo role - a role with boiler plate code to get you started with an imaginary web-based service called superfoo.

When complete, you should be able to:

• type https://foo.mayfirst.dev into your browser and see a welcome page for the amazing Super Foo web service.

• ssh into all servers as root

• From the superfoo001 server, access a MySQL database (via the host mysql001.mayfirst.dev) and a Postgres database (via the host psql001.mayfirst.dev) from the command line using the username and password of: superfoo.

Below is a quick start set of instructions followed by a more thoughtful explanation of how things work.

Quick Start

Check out the repo

Find a good spot in your file system and:

git clone https://code.mayfirst.org:mfmt/seed.git

Preparing your host

Our playbook has been developed on a Debian system (bullseye and bookworm).

Minimally, you will need:

apt install pipx

Once pipx is installed, use it to instal uv which will handle all dependencies. Run this command as your non-privileged user:

pipx install uv

Copying the default inventory file

The git repo ships with a default dev inventory file that sets up a superfoo web service.

To get started on your own development machine, copy the inventories/dev/hosts.yml.default to inventories/dev/hosts.yml.

You can start with the defaults - which set up a minimal number of hosts focused on testing the basics (it includes DNS service, generating x509 certficates and provides web services and MySQL database services). As you continue working with the code, you may choose to add other hosts.

The are other hosts files available as well, which setup different configurations of servers to test different parts of our infrastructure. Each configuration comes with it’s own automated tests (see the section below on automated tests for more information).

Setup sower

The sower command is our helper script - it provides an easy interface for running common ansible related and inventory related commands.

The repo contains a directory called sower with a python executable also called sower. Ensure the executable is in your $PATH.

If you can type sower and get get a usage explanation, you are good to go. Optionally, see sower for more info on using the command.

Create your containers

Next, you need to generate a bunch of containers to which we will apply our ansible playbook.

This requires a few steps and can take a bit of time (mostly waiting for packages to be installed). The full details are explained in our container initalization file. Be sure to fully complete the steps before proceeding.

Make it happen

Once your containers are setup, you run the command to apply all of our roles with:

sower playbook

This command effectively calls:

ansible-playbook --inventory /path/to/seed/inventories/dev/hosts.yml -e @/path/to/seed/inventories/dev/secrets.yml --vault-password-file /path/to/seed/inventories/dev/secret.txt --limit all:!external /path/to/seed/site-dev.yml

Optionally, see our ansible-vault for more information about the vault business.

Configure the certificate authority for your web browser

It’s very difficult to test web services without an https connection, which means we need x509 certificates. We can’t get Lets Encrypt to generate them when the sites are only available locally, so our ansible script comes with a built-in, private certificate authority that generates and signs all requests.

Additionally, this authority’s public key is installed on all the dev servers deployed, so these signatures are trusted.

But, your web browser won’t trust them and if you try to test a web service you deploy via our dev ansible setup, you’ll get an error.

You can get around this problem by configuring your web browser to trust our dev certificate authority, but, that opens you up to a vulnerability (anyone can access our dev certificate authority private key, use it to sign a domain that do not control and then convince your browser it’s legit).

So, your options are:

• Click through all security warnings when testing sites
• Setup a Firefox profile for this purpose and install our certificate authority for that browser profile. The certificate authority file is located in inventories/dev/mfca.crt for ease of access if you choose this route.

Manually Test

Now for the moment of truth, access your Super Foo Web Service and hopefully see a simple welcome page.

Additionally, try to ssh into your server with:

ssh root@superfoo001.mayfirst.dev

See below for the section on automated tests.

Then, you can connect to your databases with:

mysql -u superfoo -h mysql001.mayfirst.dev -p
psql -h psql001.mayfirst.dev superfoo superfoo

When prompted for a password, type: superfoo

Develop

Now you can copy the inventories/dev/superfoo001.mayfirst.dev.yml file to a new file (and edit inventories/dev/hosts.yml to include the new server). And you can copy the roles/superfoo directory to a new role and make whatever changes you want for a new service.

Slow start

Layout

The git repo is layed out like a typical ansible playbook. The main directories and files of interest are:

• inventories - see dev/hosts.yml - that is the list of hosts we will test
• site-dev.yml and site-live.yml - these files determine which roles are applied
• roles - this directory contains all the playbooks. Most coding happens with roles

Philosophy

There a million decisions that have to be made for every small piece of software added to this repository. Here are some guiding principles to help make them:

• Keep it simple: this repo is for May First, it’s not intended to be useful for anyone else (except as a starting point). Only implement options we need right now.
• Prioritize easy upgrades: When setting configuration files, used conf.d directories when available. When modifying configuration files included in a package, use lineinfile and blockinfile rather than copying our version of the file. We want to always incorporate upstream configuration defaults whenever a package is upgraded.
• Prioritize a few key roundtrip tests rather then loads of smaller unit tests. We don’t want to spend our lives maintaing test code and we want to focus on tests that demonstrate multiple systems are working together over tests that demonstrate a relatively simply operation works. For example: a test the demonstrates an email message can be sent and received is better then a test that demonstrates a Maildir is created.

Tips

Ansible is slow!

The first way to speed things up is to specify one or more comma separated hosts to the end of your sower command to limit hosts to just the ones specified.

In addition, if you want to only run a one or a few tasks, use the tags option and either:

• Specify the --tags argument to the sower command followed by a comma separated list of role names, or

• temporarily, add: tags: debug (or any value) to a task and run with --tags debug.

Elastic Search and friends

By default, elasticsearch, kibana, journalbeat and metricbeat are disabled because they are memory hogs.

You are free to enable them to try them out, however, elasticsearch will not run properly unless the max_map_count setting on your host is at least 262144.

sudo sysctl -w vm.max_map_count=262144

That temporarily sets it (but won’t retain on reboot). If you want it to survive a reboot, add a file in /etc/sysctl.d/local.conf with:

vm.max_map_count=262144

Adding roles

To initialize an empty role directory:

cd roles/admin
ansible-galaxy init rolename

Automated tests

The seed repository comes with a tests role. This role is here to help with automated testing of different hosts files.

The test001 server is built last and the last thing it runs is a the command: python3 -m unittest discover in the greenhouse directory. If that command completes successfully, then all the tests ran without error.

You can specify a tests file to run in your hosts.yml file with the variable: test_file which should correspond to a file name in the files/tests directory of the tests role.

The tests role provides a class called greenhouse the includes some basic helper functions you can use while writing your tests.

Some things to keep in mind while writing tests. Remember, tests require writing and maintaining code, so we focus on a minimal number of tests that test the most critical things.

• Write automated tests for things that are tedious to test manually. For example, it’s tedious to manually send a test email and then ensure that it is received, so that’s a good test. In contrast, it’s easy to test whether adding an ssh public key results in you being able to successfully login, making that less helpful.

• Write automated tests for things that require multiple servers to work together. For example, ensuring that a web proxy, web origin, and key server all work together to properly ensure you can connect via https is a good test. However, writing a test to ensure that a file you upload to a web server is accessible is not a good test.

• Test our ansible code, not ansible itself or a debian package. For example, ensuring that a web service properly connects to a remote database is a good test. But, ensuring that when you specify that a database should be installed, it is installed and functional, is less useful.

• Test from a user perspective, not a systems administrator perspective. Testing whether an email is delivered is better then testing whether clamav-daemon is running (and, if clamav-daemon is not running, your email will not be received, so it’s a win-win).

Git Tags

Git tags (not ansible tags!) are only added if the commit can be built and tested without errors.

If you are having trouble building out all the images in the dev inventory, trying going back to the last git tag and seeing if that works.

Git tags are named after the hosts.yml.<name> file. For example, the tag default-001 is only applied if hosts.yml.default runs successfully.