Longshore

longshore is the system used by the Progressive Technology Project to manage it’s Powerbase installations.

Using the longshore command, admins can create and manage docker containers running powerbase.

Setup

The big picture is: Powerbase is civicrm, hosted and maintained by PTP, specifically configured for organizing groups.

PTP runs several physical hosts that are dedicated to the task, some are in the May First cabinet some are in other locations.

Each physical host runs at least two KVM guests:

  • one guest runs docker and creates PHP docker containers for every single individual Powerbase site as well as providing an nginx server on the guest itself; and

  • the other runs a huge MariaDB server for all Powerbase sites.

The KVM guest running docker has ProxySQL installed so it can reach any database server in our network.

Additional, the following KVM guests are in use:

  • bulkmail servers receive email directly from each CiviCRM instance and send it on to relay servers. As an interim step, these servers perform DKIM signing (once our relay servers can perform DKIM signing these servers will stop doing it)

  • one mailstore servers handles all bounce and activity email

Container setup

All longshore servers should be allocated the following data disks:

  • About 500GB for the /srv/ directory (to store all the web server files for each site)
  • About 1TB for the /var/lib/longshore/backups directory (stores the backups of all sites on all servers)
  • About 100GB for the /var/lib/docker directory (to be used by the overlay2 file system for all docker containers)
  • About 10GB for the /tmp partition. By default a 1GB partition is allocated, but we need something much bigger.

These should all be formatted and mounted before pushing ansible to the server.

The base system for a longshore server can be setup via ansible by setting longshore_enabled to true and longshore-profiles to container.

Push ansible to the container.

Next:

  • copy /srv/longshore/etc/longshore.conf.sample to /srv/longshore/etc/longshore.conf and edit it
  • as root: /srv/longshore/bin/longshore init
  • as root: /srv/longshore/bin/longshore-create-base-image
  • as longshore: /srv/longshore/bin/longshore -v image-create –all
  • Copy the public ssh key for the longshore user into hosts.yml (longshore_user_authorized_keys) for longshore containers.
  • Copy the public ssh key for root into the container’s host_vars yaml file.
  • manually copy /home/longshore/.red-api-password from an existing server to this server
  • manually create /srv/longshore/etc/my.cnf with a user name/password/host combo that has GRANT privileges on the central mysql server (see below).
  • as longshore: /srv/longshore/bin/longshore phpmyadmin-download <url> <sha256hash>
  • as longshore: /srv/longshore/bin/longshore phpmyadmin-start

Database setup

In addition to a KVM guest to hold the docker containers running civicrm, a KVM guest must also be allocated to hold the databases.

Any MariaDB host that will host Powerbase sites must have a user that has root privileges that will be used by longshore to create the database and users for Powerbase.

Create the user with:

 GRANT ALL PRIVILEGES ON *.* TO 'longshore-creator'@'%' IDENTIFIED BY 'REPLACE-WITH-PASSWORD' WITH GRANT OPTION;

Bulkmail servers

Each physical host runs its own bulkmail server so it can more quickly process email sent by each Powerbase site.

The mail addresses are named after their city of origin:

  • newyork.bulk.progressivetech.org
  • amstercdam.bulk.progressivetech.org
  • etc

These servers are configured to accept and relay email sent by the IP addresses in use by the container guests documented above.

In addition, until the May First relay servers start adding DKIM signatures, these bulkmail servers manage the dkim signatures as well.

DKIM handling

The DKIM keys are created by the longshore command issued on the container itself (longshore site-dkim). Via ssh, the dkim keys are created on the designated central bulkmail server and the public key is returned to the longshore command, which then automatically generates a DNS record.

The central bulkmail server rsync’s the keys to all the other bulkmail servers in use to ensure that all Powerbase sites can relay via all bulkmail servers.