Ansible - Initialize podman containers

Overview

The sower command can be used to build out a set of podman (previously docker) containers that can mimic a real network of servers, allowing us to test and develop our ansible playbook.

The container images are used for dev and learning purposes only. They should not be used to host any production services.

The script builds out container images for convenience. It builds them in non-standard ways. In particular:

  • It builds our own base debian image so we don’t have to blindly pull images from the Internet and run them on our machines.

  • It builds out images that run systemd on the inside. Most docker images either don’t run an init program (they run the service directly) or they use a simpler init system (like dumb-init). These images are built using systemd so they more closely reflect the way our actual guests are run. And we have switched to podman, which seems to be friendlier to containers running systemd than docker is.

  • It install openssh in the image without re-creating the server keys. This means the private keys are embedded in the image. This is a big security problem for production sites, but a nice convenience for testing because it means you don’t have to keep approving ssh keys every time you re-create containers.

Steps

We run podman in “rootful” mode to take advantage of more convenient networking. So, all podman commands have to be run via sudo. To make this more convenient:

sudo apt install sudo
echo "$USER  ALL=NOPASSWD: /bin/podman" > /etc/sudoers.d/podman

If you haven’t already, ensure you have both podman and debootstrap packages installed:

apt install podman debootstrap

You need to copy your public ssh key to the docker directory and ensure you preserve the .pub suffix. This allows your key to be added to /root/.ssh/authoried_keys thus allowing you to ssh in without any fuss.

Next, run the ./build-os-image <current-debian-release> command, as root (or via sudo), from the docker directory. This script will create a debian stable image so we don’t have to pull it off the Internet. It must be run as root.

Lastly, as your regular user, run the sower docker:init command.

It will:

  • Create a seed-public and seed-local network.
  • Create a seed-base image.
  • Build out all containers used in the dev ansible environment.
  • Warn you if you need to add any entries to your /etc/hosts file. Please follow these directions! Putting the entries in your /etc/hosts file allows ansible to find them via their hostnames.

The following arguments control how it works

  • --force containers|images: force a rebuild of all containers or images.
  • --apt-proxy https://blah.proxy.org/debian: use an apt proxy server

Trouble shooting

If you have trouble running the first podman command with the error:

cannot open sd-bus: No such file or directory

Then, install the runc package and remove the crun package.

You may get an error related to inotify along the lines of:

Failed to allocate directory watch: Too many open files

To avoid this, you have to raise the limit on the inotify max_user_instances setting. You can do it temporarily with:

    sysctl -w fs.inotify.max_user_instances=256

And permanently with:

    echo "fs.inotify.max_user_instances=256" > /etc/sysctl.d/local.conf

Also… in order to control the amount of RAM used by each docker instance, the host needs to boot into linux with cgroup_enable=memory. You can add this to /etc/default/grub, e.g.:

GRUB_CMDLINE_LINUX="console=ttyS0,115200n8 cgroup_enable=memory"