https://admin.docs.mayfirst.org/ Recent content in Welcome on May First Admin Docs Hugo -- gohugo.io en-us Thu, 13 Aug 2026 13:53:34 +0000 https://admin.docs.mayfirst.org/cacti/ Tue, 06 Jan 2026 13:53:34 +0000 https://admin.docs.mayfirst.org/cacti/ We use cacti to query our switch and record network traffic. Login to https://cacti.mayfirst.org/ (password in keyringer) to view network traffic. Use the “mayfirst” username to get a simplified view that only shows graphs. The admin user is only needed to trouble shoot or change graphs. Some things to keep in mind: The upstream graph shows all traffic in and typically hovers around 100Mbit/second All other graphs include internal traffic (which includes database queries) that are not recorded in our upstream Setup Most of the configuration is in the database - so when moving or re-installing be sure to import the database backup. https://admin.docs.mayfirst.org/lvm-caching/ Tue, 26 Aug 2025 13:53:34 +0000 https://admin.docs.mayfirst.org/lvm-caching/ Getting Started We use LVM caching when we have a server with both spinning disks and SSD disks and the spinning disks are experience disk i/o slow downs. Before you can create a cache device, you have to add the following setting to /etc/lvm/lvmlocal.conf so that logical volumes can be available as physical volumes: devices { scan_lvs = 1 } In addition, ensure that thin-provisioning-tools is installed. The commands For example, on a server in which the spinning disks are in vg0 and the SSDs are in vg1, we can add a cache called cache001 to the logical volume weborigin002 with the following commands: https://admin.docs.mayfirst.org/disk-management/ Wed, 22 Mar 2023 13:53:34 +0000 https://admin.docs.mayfirst.org/disk-management/ When a disk on a virtual guest starts to fill up, we have two options: Allocate a new disk: if the disk in question is a data disk serving member data on a web server or a mail server, we generally want to allocate a new disk so our data disks don’t grow too large, making them more difficult to backup or to run disk utilities on. The web and mail servers use symlinks from the /home directory, allowing us to mount as many separate disks in the /media directory as needed. https://admin.docs.mayfirst.org/monitor/ Sun, 06 Dec 2020 13:53:34 +0000 https://admin.docs.mayfirst.org/monitor/ May First uses a variety of tools that work together to provide monitoring and alerting of our systems to warn us of impending problems or immediate outages. There are three main jobs: Detecting problems Communicating problems to a central storage location Notifying us of the problems The heart of our system is our central elasticsearch database. Information is sent from each individual server to elasticsearch via: journalbeat, which sends select data from journald metricbeat, which sends server health data, filebeat, which sends selected log files, pigeon, which detects anomolies and outputs them to journald simplemonitor, which tests network connectivity and runs a command to ensure elasticsearch is properly working. https://admin.docs.mayfirst.org/network-boot/ Thu, 06 Feb 2020 13:53:34 +0000 https://admin.docs.mayfirst.org/network-boot/ Network Boot Sometimes we have to boot strap a server into an operating system via the network. There are two main ways we do that: PXE Boot PXE Boot. Almost all servers have the option to boot from a network. The cabinet with the server you want to boot must have a server configured to provide TFTP and DHCP and have the proper images. In telehouse, clr is currently setup to provide netboot services. https://admin.docs.mayfirst.org/ansible/ Wed, 21 Aug 2019 13:53:34 +0000 https://admin.docs.mayfirst.org/ansible/ Ansible is the server configuration automation tool we use to ensure our servers are configured in a uniform way. Our ansible setup involves two git repositories: seed is the name of the git repository containing our roles and most of our ansible code. It also includes the inventories/dev directory, which allows you to create a minimal set of container images that demonstrates how our ansible repo is designed to work. https://admin.docs.mayfirst.org/ipmi/ Fri, 13 Mar 2020 13:53:34 +0000 https://admin.docs.mayfirst.org/ipmi/ Ansible Configuration When you first boot the server, IPMI has to be enabled and assigned an IP address. We use a non-routable IP address so the only way to connect to IPMI is via a machine in the colo itself. To pick an IP for the new server, grep the host_vars directory in our inventory for ipmi_address to see the currently assigned address and pick the next incrementally from the output. https://admin.docs.mayfirst.org/troubleshoot-kvm-guest/ Sat, 13 Jul 2024 13:53:34 +0000 https://admin.docs.mayfirst.org/troubleshoot-kvm-guest/ Sometimes KVM guests fails to start. To troubleshoot, we can boot into a grml live installation. On the host, download the small grml live install into /usr/local/share/ISOs As the kvm guest user, create a symlink: ln -s /usr/local/share/ISOs/grml64-small_YYYY.MM.iso ~/vms/${USER}/cd.iso Reboot the KVM guest and quickly enter the screen session Hit “Esc” key to get to boot menu Choose the CD/DVD option Under “Additional boot entires for grml64-small”, choose “serial console” (at the bottom) Chroot The really get at the installation, you may need to mount the directories in /mnt and use chroot to fix what is wrong. https://admin.docs.mayfirst.org/get-involved/ Sat, 12 Dec 2020 13:53:34 +0000 https://admin.docs.mayfirst.org/get-involved/ May First Movement Technology is a membership-based organization with a mission to: engage in building movements by advancing the strategic use and collective control of technology for local struggles, global transformation, and emancipation without borders. Want to get involved? May First has a Technology and Infrastructure Services team open to all members of the organization. If you are interested in joining, please login to our discourse instance, view our invitation to participate, and join the group. https://admin.docs.mayfirst.org/install-debian-host/ Thu, 06 Feb 2020 13:53:34 +0000 https://admin.docs.mayfirst.org/install-debian-host/ Server Installation Initial steps Add the server to the hosts.yml file and create a host_vars file. Ensure there is a console entry and then run ansible to the console server (in Telehouse it’s clr). Read the top of the IMPI page as well to get the settings right. Ensure that network boot is setup in the cabinet in which you are installing the server. Plug in monitor and keyboard Enter Bios/Setup. https://admin.docs.mayfirst.org/ansible-hacking/ Wed, 21 Aug 2019 13:53:34 +0000 https://admin.docs.mayfirst.org/ansible-hacking/ Overview Hacking on a server configuration tool is hard because you need a fleet of servers to test on. To make it easier, our ansible repository comes with tools that allow you to create a fleet of servers residing just on your local machine using podman. When you have completed these tasks you will end up with: An authoritative DNS server (nsauth001.mayfirst.dev) set with all domain names in use (the dev sites use the domain: mayfirst. https://admin.docs.mayfirst.org/ansible-sower/ Wed, 10 Nov 2021 13:53:34 +0000 https://admin.docs.mayfirst.org/ansible-sower/ Overview Our ansible scripts and our seed inventory combine a lot of useful information about our infrastructure. The sower command is designed to easily parse the information and manipulate it. Some details Setup To effectively use sower, you should ensure that the path to the executable is in your $PATH environment variable. If you have access to our live inventory, then you should also set an environment variable: SEED_LIVE_DIR=/path/to/seed/inventory/directory This environment variable is not necessary when operating in dev mode. https://admin.docs.mayfirst.org/ansible-docker-initialize/ Wed, 21 Aug 2019 13:53:34 +0000 https://admin.docs.mayfirst.org/ansible-docker-initialize/ Overview The sower command can be used to build out a set of podman (previously docker) containers that can mimic a real network of servers, allowing us to test and develop our ansible playbook. The container images are used for dev and learning purposes only. They should not be used to host any production services. The script builds out container images for convenience. It builds them in non-standard ways. In particular: https://admin.docs.mayfirst.org/ansible-vault/ Wed, 21 Aug 2019 13:53:34 +0000 https://admin.docs.mayfirst.org/ansible-vault/ Ansible provides a mechanism for storing secrets in a dedicated and encrypted file. It’s not ideal to have to keep secrets at all, but it some cases it seems worth the risks, particularly in order to have continuous integration working. The ansible vault file should be placed in the inventory directory, e.g.: inventories/dev/secrets.yml To ease continuous intergration, for the dev and testing inventory, we have a secret.txt file that holds the password to unlock the secrets. https://admin.docs.mayfirst.org/stress-test-new-hardware/ Thu, 08 Oct 2020 13:53:34 +0000 https://admin.docs.mayfirst.org/stress-test-new-hardware/ All new hardware should be put through a stress test to detect any failing components. Launch the tests Boot into debirf image. If stretch, install pip: apt install python-pip iperf3 lshw stress-ng and then install stressant: pip install stressant. If later, install stressant instead of pip. Start long smartctl tests on each disk: for disk in $(cat /proc/partitions | egrep -o 'sd[a-z]$'); do smartctl -t long "/dev/$disk"; done. These will take many hours to complete. https://admin.docs.mayfirst.org/kvm-manager/ Fri, 07 Feb 2020 13:53:34 +0000 https://admin.docs.mayfirst.org/kvm-manager/ Overview May First uses KVM Manager to manage virtual guests on each of our physical host machines. KVM Manager uses the qemu-kvm virtualization software. KVM Manager is a set of simple bash scripts that help us by setting up new virtual server instances and allows us to manage each instance via systemd. The main benefits of KVM Manager are: Use isolation: nothing runs as root on the host. All virtual servers run via systemd as a non-provileged user. https://admin.docs.mayfirst.org/modlock/ Thu, 13 Aug 2026 13:53:34 +0000 https://admin.docs.mayfirst.org/modlock/ Several recent linux vulnerabilities are triggered when kernel modules (that are not needed by us) are loaded by non privileged users (see dirty frag and copy fail. To avoid that in the future, we have installed modlock. This program, when initially setup (with --setup), creates a list of modules currently loaded and then creates a systemd service that ensures they are loaded on startup before disabling loading of any other modules. https://admin.docs.mayfirst.org/vaultwarden/ Tue, 12 May 2026 13:53:34 +0000 https://admin.docs.mayfirst.org/vaultwarden/ Vaultwarden is an alternative implentation of the Bitwarden password manager server. It works as a drop in replacement with all Bitwarden clients. It can handle a variety of types of data, it is an end to end encrypted service, with admins unable to view the passwords of users, and with the ability to create teams that can share items including TOTP MFA prompts. Podman We chose to use podman for the deployment of Vaultwarden. https://admin.docs.mayfirst.org/mailq-management/ Tue, 23 Dec 2025 13:53:34 +0000 https://admin.docs.mayfirst.org/mailq-management/ If our email is being blocked or deferred or if we just trying to warm up new IPs, we wil need to get reporting information about our mail queues and/or move messages between queues to ensure they are delivered. A few concepts Be sure to read up on relay servers on our postfix page and our relay servers page. To review: We have multiple mail relay servers (name mailrelay001, mailrelay002, etc). https://admin.docs.mayfirst.org/z-mx/ Sat, 06 Dec 2025 03:53:34 +0000 https://admin.docs.mayfirst.org/z-mx/ All domain name MX records should point to one of our MX server {a,b,c}.mx.mayfirst.org. However, some membes use App River to filter viruses and spam and then configure App River to relay on to May First. This causes one major problem: Our MX servers will not be able to properly run DMARC and SPF checks because we can’t properly determine what the last hop was. To resolve this problem we have a special MX server (z. https://admin.docs.mayfirst.org/shared-web-server-mounted-home-directories/ Fri, 07 Nov 2025 13:53:34 +0000 https://admin.docs.mayfirst.org/shared-web-server-mounted-home-directories/ Overview All site directories on web servers are located in the /home/sites directory and named after the control panel item_id for the particular web configuration (e.g. /home/sites/123). To avoid creating a massively huge /home/sites partition that would be infeasible to fsck, the actual block devices supporting these sites are mounted under /media/dataNNNN (where NNNN is a four digit number) and kept to a maximum of 250GB each. Each site directory under /home/sites is bind-mounted from its corresponding /media/dataNNN directory, e. https://admin.docs.mayfirst.org/mailman/ Mon, 27 Oct 2025 13:53:34 +0000 https://admin.docs.mayfirst.org/mailman/ May First runs the mailman mailing list manager. Components Mailman3 is broken into several discrete components: mailman - the core component that controls the sending of email, configured via /etc/mailman3/mailman.cfg. List settings are stored in a postgres database. mailman-web - the web-based companion service (written in django), configured via /etc/mailman3/mailman-web.py that consists of two django apps: postorius - the django app that allows list owners to manager their own list settings and subscribers hyperkitty - the django app that provides access to list archives if a list is configured to archive messages. https://admin.docs.mayfirst.org/virus-detection/ Wed, 23 Jul 2025 13:53:34 +0000 https://admin.docs.mayfirst.org/virus-detection/ We use ClamAV to detect viruses in incoming email. ClamAV is invoked by clamsmtp so that Postfix simply sends all email through clamsmtp which then re-injects all email without viruses back into the postfix queue. Sometimes ClamAV has a false positive - you can find them by searching the elastic search logs for: message: "250 Virus Detected; Discarded Email" You can then search, on the specified mailfilter server, for the reason via journalctl -u clamsmtp, e. https://admin.docs.mayfirst.org/bot-challenge/ Sat, 12 Jul 2025 03:53:34 +0000 https://admin.docs.mayfirst.org/bot-challenge/ Overview When a site is being hit with bot traffic that uses a user agent string typically reserved for humans, we can route its traffic through anubis, which poses a challenge to the user in an attempt to determine if it’s a human or not. How to turn it on Determine which web proxy is being used by the host via DNS lookup. Edit the /etc/nginx/sites-enabled/siteNNNNN.conf file (see below for details) When tested, edit the web conf in the control panel, set Cache Type to “Custom cache settings” and copy the nginx config contents to the Cache Settings field. https://admin.docs.mayfirst.org/chat/ Sat, 18 Jan 2025 03:53:34 +0000 https://admin.docs.mayfirst.org/chat/ We provide XMPP services via prosody. And we provide an XMPP client via movim. Prosody Prosody is installed via debian. It authenticates via Cyrus SASL, which is configured to check /etc/passwd via PAM. The control panel simply uses useradd to add users to the system in order to enable their ability to login to our prosody server. Movim Movim is installed via git. Initial configuration After installing movim, run: php daemon. https://admin.docs.mayfirst.org/scheduled-jobs/ Tue, 10 Dec 2024 03:53:34 +0000 https://admin.docs.mayfirst.org/scheduled-jobs/ User are able to create scheduled jobs in the control panel. These can be set to run on a given interval or “forever.” All jobs are translated into systemd service and timer files, which are saved in the site’s /.config/systemd/user directory on the weborigin. Users, when ssh’ed into the server, can manipulate them via systemctl provided they prefix the command with --user, e.g. systemctl --user status red-item-1234.service. When logged in as root, you can’t do that. https://admin.docs.mayfirst.org/postgres/ Sat, 13 Jul 2024 13:53:34 +0000 https://admin.docs.mayfirst.org/postgres/ Postgres database have to be uprade as manually after a Debian upgrade. First become the postgres user: su - postgres Next, list all clusters: pg_lsclusters See if any clusters have databases other than “postgres”, “template0”, and “template1”: psql --port NNNN --list (replace NNNN with the port number returned by the pg_lsclusters command, start clusters if necessary with: pg_ctlcluster VERSION main start) When the most recent version has no other databases other than the defaults, you can drop it and then upgrade from the version that has a database. https://admin.docs.mayfirst.org/civicrm/ Wed, 19 Jun 2024 13:53:34 +0000 https://admin.docs.mayfirst.org/civicrm/ May First maintains our own CiviCRM installation via a standard hosting order using the address outreach.mayfirst.org. It’s running CiviCRM Standalone. Code base The entire site (except the public and `private’ file directories) is kept in git. The main code we maintain is in the “mayfirst” CiviCRM extension. This code provides our custom membership workflow, displays, scheduled jobs, reports and other functionality. Tests Note: running tests in this manner is no longer working. https://admin.docs.mayfirst.org/nginx-proxy-snippets/ Mon, 06 May 2024 13:53:34 +0000 https://admin.docs.mayfirst.org/nginx-proxy-snippets/ By default, all web sites are configured to pass through our Nginx Proxy servers un-altered. However, if a site comes under attack or is overloaded, we can add a custom nginx proxy configuration to try to cache or block some of the requests. Big picture The first step is to determine which web proxy is being used by the site. You can dig the domain name or look it up in the DNS section of the control panel. https://admin.docs.mayfirst.org/onlyoffice/ Sun, 24 Mar 2024 03:53:34 +0000 https://admin.docs.mayfirst.org/onlyoffice/ Onlyoffice is a web based application providing office (Document, Spreadsheet, etc) like functionality. It is integrated into our Nextcloud installation. We specifically install our own version of OnlyOffice provided via infrared. This repo includes a docker and docker compose file. A few important considerations: The docker compose file includes an nginx container, but we ignore it and provide nginx via the host, which is configured via /etc/nginx/conf.d/proxy-https-to-http.conf You can control both the 0xacab and upstream git versions via ansible variables in the onlyoffice001. https://admin.docs.mayfirst.org/mysql-trouble-shooting/ Fri, 01 Mar 2024 03:53:34 +0000 https://admin.docs.mayfirst.org/mysql-trouble-shooting/ Errors On modern (bookworm and above) versions of MariaDB, errors are logged to journald and should be exported to elastic search. On earlier versions, look in /var/log/mysql/. Slow Queries You can temporarily enable slow query logs (will be unset when MariaDB restarts) with: SET GLOBAL slow_query_log = 'ON'; SET GLOBAL long_query_time = 10; SET GLOBAL slow_query_log_file = '/var/log/mysql/mariadb-slow.log'; This may adversely affect performance. You can turn it off with simply: https://admin.docs.mayfirst.org/alert-response/ Fri, 23 Feb 2024 13:53:34 +0000 https://admin.docs.mayfirst.org/alert-response/ All alerts should come with a link to this page, which explains how to fix the problem. elasticsearch health The elasticsearch_health alert happens when accessing https://log.mayfirst.org/_cat/health does not return a 200 http response code. That might mean our elasticsearch database is under heavy load, is restarting or has failed. Login to log001.mayfirst.org and check /var/log/elasticsearch/elasticsearch.log and journalctl -u elasticsearch. missing pigeon heartbeat Every host running pigeon should send a heartbeat message once a minute. https://admin.docs.mayfirst.org/red-validation/ Wed, 14 Feb 2024 13:53:34 +0000 https://admin.docs.mayfirst.org/red-validation/ Since we have tightened our validation rules without ensuring existing data meets the new validation rules, we have a lot of data in the control panel that will trigger a validation error when you try to edit, disable or delete it. The following are common red validation errrors and how to fix them: A DNS record with the same value already exists for example.org / txt. This error may have a different domain than “example. https://admin.docs.mayfirst.org/dns/ Sat, 03 Feb 2024 13:53:34 +0000 https://admin.docs.mayfirst.org/dns/ There are many kinds of domain name servers in our network: function: authoritative vs caching scope: public vs internal profile: origin vs copy vs static Caching servers Our caching DNS servers are also known as resolving DNS servers. Every physical host runs unbound directly, providing DNS caching services for all hosts in our network. This service is configured to only respond to requests from IP addresses in our ansible playbook, including hosts in other locatins. https://admin.docs.mayfirst.org/email-abuse/ Thu, 28 Dec 2023 13:53:34 +0000 https://admin.docs.mayfirst.org/email-abuse/ We can access data about email complaints from several sources. Note: if we are being blocked or deferred by a provider, reach this page and also read our mailq management page. Our official abuse email address. Our official email for receiving abuse complaints is abuse [@] m.o. Almost all email sent to this address is in the abuse reporting format, which means it’s not useful to read email sent to this mailbox directly. https://admin.docs.mayfirst.org/proxysql-mosh-transition/ Sun, 17 Dec 2023 13:53:34 +0000 https://admin.docs.mayfirst.org/proxysql-mosh-transition/ To transition the databases from the MOSH’es to the network database servers, we first re-configure MySQL to listen on localhost port 3307 (instead of 3306). Then we install proxysql and instruct it to proxy for the local MySQL server. Then, we can transfer each database one at a time with minimal downtime. The steps are: In the puppet .pp file of the mosh add (replace $HOST with the actual name of the host): class { "mayfirst::m_proxysql::mosh": transition => true } In the Ansible Seed Inventory repository, add to the MOSH’es yaml file: proxysql_profile: proxy Ansible push to mysql00[45678] (and all proxysql databases) sower --live playbook --tags proxysql mysql00[45678]. https://admin.docs.mayfirst.org/shared-mailserver/ Mon, 11 Dec 2023 13:53:34 +0000 https://admin.docs.mayfirst.org/shared-mailserver/ The following steps should be taken to setup a new shared mail server: In ansible seed repo, edit hosts.yml to copy each line with mailstore002 to your new name. Execute sower --live generate:mailstore --parent hostxxx.mayfirst.org to generate a new mailstore yaml file. Follow regular kvm guest setup steps Be sure you created the ssh key in the step above! Insert the server into the red_server table: INSERT INTO red_server SET server = 'mailstore00x. https://admin.docs.mayfirst.org/shared-mysql-server/ Mon, 11 Dec 2023 13:53:34 +0000 https://admin.docs.mayfirst.org/shared-mysql-server/ The following steps should be taken to setup a new shared mysql server: In ansible seed repo, edit hosts.yml to copy each line with mailstore002 to your new name. Execute sower --live generate:mysql --parent hostxxx.mayfirst.org to generate a new mysql yaml file. Follow regular kvm guest setup steps Be sure you created the ssh key and enabled quotas in the step above! Update all weborigins to let them know about the mysql server so they can connect via proxysql: sower --live playbook --tags proxysql-build-origins-file weborigin*. https://admin.docs.mayfirst.org/mfca/ Sun, 19 Nov 2023 03:53:34 +0000 https://admin.docs.mayfirst.org/mfca/ While May First mostly depends on Lets Encrypt for our PKI infrastructure, we do operate our own certificatte authority for signing certificates in our dev and/or testing environment and for internal only connections. The certificate authority key is saved in our ansible vault using the variable name vault_mfca_key and the signed certificate key is stored in the variable mayfirst_mfca_cert. The certificate is saved to the directory /usr/local/share/ca-certificates/mfca.crt, which is picked up by the ca-certificates debian package and incorporated into the /etc/ssl/certs directory so all local applications will trust certificates signed by our key. https://admin.docs.mayfirst.org/shared-web-proxy/ Sat, 11 Nov 2023 13:53:34 +0000 https://admin.docs.mayfirst.org/shared-web-proxy/ Adding In ansible seed inventory repo, edit hosts.yml and add the new host in all places an existing host exists. Execute sower --live generate:weborigin --parent hostxxx.mayfirst.org to generate a new weborigin yaml file. Follow regular kvm guest setup steps Note: The next steps have to happen close together. After you run the first step, changes to any web configuration in the control panel will fail until you have completed the remaining steps. https://admin.docs.mayfirst.org/shared-web-server/ Sat, 11 Nov 2023 13:53:34 +0000 https://admin.docs.mayfirst.org/shared-web-server/ The following steps should be taken to setup a new web server: In ansible seed repo, edit hosts.yml to copy each line with weboriring001 to your new name. Execute sower --live generate:weborigin --parent hostxxx.mayfirst.org to generate a new weborigin yaml file. Follow regular kvm guest setup steps Be sure you created the ssh key in the step above! Grant access to the mysql server so your weborigin server can run proxysql properly. https://admin.docs.mayfirst.org/bad-bots/ Mon, 06 Nov 2023 13:53:34 +0000 https://admin.docs.mayfirst.org/bad-bots/ Overview This page describes our strategy for throttling or blocking bots from accessing web sites on our servers. Bots are programs making web site requests that are not directly driven by a human. There is some gray area. For example, if a user posts a link on Bluesky, a Bluesky bot will lookup the web page and retrieve the name of the page and possibly an image from the page. That’s technically a bot, but it’s one responding directly to a human’s request. https://admin.docs.mayfirst.org/ansible-adhoc/ Thu, 05 Oct 2023 08:24:34 +0000 https://admin.docs.mayfirst.org/ansible-adhoc/ Sometimes we want to run a single command on all of our servers. The ansible command allows us to easily do that. Here’s a sample command: sower --live ls '!noaccess' | xargs -P 10 -I {} bash -c 'echo "--- {} ---" && ssh root@{} /bin/true' --inventory should point to the hosts.yml file in your seed inventory repository. --one-line condenses the output to a single line making it easier to see which commands failed. https://admin.docs.mayfirst.org/transfer/ Sat, 22 Oct 2022 13:53:34 +0000 https://admin.docs.mayfirst.org/transfer/ Overview The mf-transfer script automates the transfer of resources between servers and between partitions. It handles both copying the data and also updating the control panel resource to reflect the new location. The command copies data from the source location to the destination location. If you are moving a resource from one partition to another on the same server, then the source and destination servers will be the same. The command line script should be run as root from the destination server. https://admin.docs.mayfirst.org/red/ Wed, 19 Oct 2022 13:53:34 +0000 https://admin.docs.mayfirst.org/red/ Red is the name of the May First control panel. Debugging failed items When a control panel item is saved or deleted, one or more tasks are queued to be executed (e.g. “run red-node-update on the target server” or “rebuild the email databases” or “generate an https certificate”). The execution of the queued items happens via a timer that runs every 15 seconds, not via the web. Sometimes a control panel item has a “soft-error” or “hard-error” or seems to be stuck in a pending state. https://admin.docs.mayfirst.org/quota/ Wed, 19 Oct 2022 13:53:34 +0000 https://admin.docs.mayfirst.org/quota/ Our control panel restricts disk usage by resource (e.g. a user login or database). Databases For databases, we simply use du to check database usage via a cron job and, if a database has gone over, we disable each database user with write access to that database. Users For email and web site disk usage, we use the ext4 quota sytem to set a quota and depend on it to inform users when they have gone over. https://admin.docs.mayfirst.org/email-test-suite/ Sun, 02 Oct 2022 13:53:34 +0000 https://admin.docs.mayfirst.org/email-test-suite/ Since we use round robin DNS for our client facing servers (mailcf) and we have multiple MX servers that can accept email, it is hard test each specific server to ensure it is working properly. The May First Email Test Suite is a simple script that can iterate through our list of servers and ensure they are all working. The test suite expects to be run from your local workstation and be given a username, password and associated email address capable of sending and receiving email via our network. https://admin.docs.mayfirst.org/verify-mx-domain/ Mon, 11 Jul 2022 13:53:34 +0000 https://admin.docs.mayfirst.org/verify-mx-domain/ The problem Any member is free to add a new domain to the DNS section of their control panel provided no other member already has a domain name record matching it (we use the public suffix domain list to figure out the difference between a root domain and a subdomain). This policy means members could engage in sketchy behavior, like adding gmail.com as their DNS record and then adding maria@gmail.com as their own email address. https://admin.docs.mayfirst.org/dkim/ Wed, 18 May 2022 13:53:34 +0000 https://admin.docs.mayfirst.org/dkim/ Domain Key Identified Mail (DKIM) is a method for digitally (and transparently) signing outgoing email in a way that indicates it was authentically sent by the person who claims to have sent the message. The purpose of DKIM is to stop fraudulent email messages by preventing people from sending email claiming to be from email addresses to which they don’t in fact have access. More specifically DKIM requires the generation of a public/private key pair for each domain name that sends email via May First. https://admin.docs.mayfirst.org/filter-check/ Mon, 16 May 2022 13:53:34 +0000 https://admin.docs.mayfirst.org/filter-check/ filter-check notifies us if email sent via our servers is landing in the spam box of the three major providers - Google, Microsoft or Yahoo. We have created email accounts on all three providers. filter-check iterates over all of our relay IP addresses and over all three providers to send an email from each IP address to each provider. Then, it waits a bit and tries to check for that message via IMAP. https://admin.docs.mayfirst.org/authentication-milter/ Sat, 14 May 2022 13:53:34 +0000 https://admin.docs.mayfirst.org/authentication-milter/ Overview Our MX servers run a mail filter (milter) on incoming email that runs the alphabet soup of checks - DKIM, SPF, DMARC, etc. For some checks (SPF, DKIM) it’s responsible for adding headers indicating whether the message passed those checks. For others (e.g. DMARC) it is responsible for rejecting the message if it fails to pass. We originally used the debian packaged opendmarc and opendkim, but the upstream packages are not well maintained and we noticed some incoming email messages caused opendmarc to crash, so we have switched to authentication_milter. https://admin.docs.mayfirst.org/mistert/ Tue, 12 Apr 2022 13:53:34 +0000 https://admin.docs.mayfirst.org/mistert/ Mr. T provides a number of email reporting features based on mail log data collected by elastic search. See the upstream README for documentation on usage - in particular how to get a list of senders and recipients to track down potential email problems. General review and searching mistert list and mistert stats print lists and statistics about email sent through our network. For example: mistert --since 2h --until 1h list will print a line for every message sent between 2 hours ago and one hour ago, including who it was sent from and the message id. https://admin.docs.mayfirst.org/relay/ Tue, 12 Apr 2022 13:53:34 +0000 https://admin.docs.mayfirst.org/relay/ As documented on our postfix page, relay servers send email out to the world. This page provides additional information about how they are configured. Domain Names and IP addresses Since these servers rely on dozens of IP addresses to send email in parallel, adding and removing IP addresses to the configuration files is an on-going task. Unlike most IP addresses, special DNS records are required for our outgoing relay IPs. https://admin.docs.mayfirst.org/backup-restore/ Fri, 29 Oct 2021 13:53:34 +0000 https://admin.docs.mayfirst.org/backup-restore/ To restore from the backup, always start from the server hosting the data being restored. Do not restore from the backup server. From the source host (source_host), examined /etc/backup.d/71_backup.borg. Look for the host, port, user, and directorty values. With this info, execute the command: borgbackup list ssh://${source_host}-sync@${host}:${port}${directory} You should see a list of dated backup archives (archive). Next, get a file list of the available files with: borgbackup list ssh://${source_host}-sync@${host}:${port}${directory}::archive Next, cd into the directory you want to restore the files. https://admin.docs.mayfirst.org/jitsimeet-simultaneous-interpretation/ Thu, 09 Sep 2021 13:53:34 +0000 https://admin.docs.mayfirst.org/jitsimeet-simultaneous-interpretation/ Overview Jitsi Meet has an extensive API, including one designed to work via an iframe. In other words, you include your main Jitsi Meet web page via an iframe, and then write javascript code to control how it works. JSI We have written a simple wrapper called JSI using the iframe API to provide a intrepretation slider. It’s installed on the same server running Jitsi Meet and enabled via a separate nginx configuration file. https://admin.docs.mayfirst.org/planning/ Tue, 08 Jun 2021 13:53:34 +0000 https://admin.docs.mayfirst.org/planning/ Overview The planning page maps out the major projects on the horizon and the rough schedule for implementation. Purple items are items Jamie is taking the lead on. New dues: Switch from managing dues in our Red control panel to managing them via CiviCRM and implement the new dues schedule. Physical Servers to Ansible: Transition all physical servers from puppet to ansible and upgrade to buster in the process. Non-red servers to Ansible: Transition all virtual servers that are not tied to the control panel to ansible and upgrade to buster (or bullseye) in the process. https://admin.docs.mayfirst.org/jitsimeet/ Mon, 03 May 2021 13:53:34 +0000 https://admin.docs.mayfirst.org/jitsimeet/ Jitsi Meet Jitsi Meet is a web-based video conferencing tool based on XMPP and HTML5 web browser video standards. (It is easily confused with with Jitsi, which is a XMPP client.) The Jitsi Meet world has many components: The server: The server component includes the nginx web server, prosody XMPP server, jitsi-videobridge2 (the main unique component that makes everything work), and some html and javascript glue to tie it all together. https://admin.docs.mayfirst.org/jitsimeet-jibri/ Mon, 03 May 2021 13:53:34 +0000 https://admin.docs.mayfirst.org/jitsimeet-jibri/ Overview The jitsimeet jibri server provides a recording and broadcasting service. We are currently only using the recording service. Jibri should be installed on a separate machine from the jitsimeet server because jibri can be a CPU hog. However, if you want a Jitsi Meet instance to integrate with jibri, be sure to set jitsimeet_jibri: true in the jitsimeet instance’s yaml file. Using the jibri debian package takes care of most of the aspects of installing jibri with the exception of the /etc/jitsi/jibri/jibri. https://admin.docs.mayfirst.org/jitsimeet-jigasi/ Mon, 03 May 2021 13:53:34 +0000 https://admin.docs.mayfirst.org/jitsimeet-jigasi/ Overview The jitsimeet jigasi server provides integration with a SIP gateway enabling call-in and dial-out features for Jitsi Meet and also transcription services via a variety of back ends. We are using the transcription service via the Jitsi meet skynet API which uses the Faster Whisper LLM models, which are based on the OpenAI Whisper models, which is a general purpose speech recognition model. This LLM is run on May First infrastructure and does not leak any conversations. https://admin.docs.mayfirst.org/jitsimeet-server/ Mon, 03 May 2021 13:53:34 +0000 https://admin.docs.mayfirst.org/jitsimeet-server/ Overview The jitsimeet server is the normally the easiest component to install. We essentially follow the quick install instructions. Most of the configuration changes we make are in the /etc/jitsi/meet/domain-config.js file and we make this via lineinfile changes. Our main goal with the changes from the defaults are to reduce bandwidth and CPU pressure on the clients that connect. We also copy in the domain-interface-config.js file, but so far we are not making any modifications. https://admin.docs.mayfirst.org/alerter/ Fri, 01 Jan 2021 13:53:34 +0000 https://admin.docs.mayfirst.org/alerter/ When it’s time for a monitoring tool to push an alert to an administrator, it calls the alert command with the appropriate arguments: --team - either ptp or mayfirst, depending on which team should receive the alert. --priority - either critical or warning depending on the severity (see the monitor page for an explanation of the difference between the two) --body - the body of the message to send. --subject - optionally provide a subject (only used for email alerts) https://admin.docs.mayfirst.org/signal-cli/ Fri, 01 Jan 2021 13:53:34 +0000 https://admin.docs.mayfirst.org/signal-cli/ signal-cli is a command line signal client used by our alerter script for sending messages to administrators when something has gone wrong. We install the pre-packaged debian binaries. After installation, manual setup is required for a new signal phone number. Primary Host The phone number used by signal-cli is specified in the inventory using the signal_cli_number variable. It should be in the format: +15555555555. In the examples below we use this number as the example number. https://admin.docs.mayfirst.org/gitlab/ Thu, 17 Dec 2020 13:53:34 +0000 https://admin.docs.mayfirst.org/gitlab/ About our GitLab configuration Our GitLab installation is designed to provide a secure, lightweight platform for hosting Git repositories and reviewing code. GitLab includes many additional capabilities—including continuous integration, package registries, container registries, wikis, and deployment tools—but most are disabled by default to keep the interface focused on source code management. Configuration is managed entirely through Ansible using the GitLab Omnibus package for Debian. Configuration management The Debian apt package does not directly write to the main configuration file /etc/gitlab/gitlab. https://admin.docs.mayfirst.org/orientation/ Sat, 12 Dec 2020 13:53:34 +0000 https://admin.docs.mayfirst.org/orientation/ The May First network is composed of a lot of moving parts that work together. As the list of categories on the right demonstrate, it’s a lot to take in! This page is designed to ease you into things by providing a big picture. The code All of our code lives at https://code.mayfirst.org/mfmt/. That’s great if you know what you are looking for, but for those who want a high level summary, keep reading… https://admin.docs.mayfirst.org/glance/ Fri, 11 Dec 2020 13:53:34 +0000 https://admin.docs.mayfirst.org/glance/ glance is a dead simple php script that queries our elasticsearch database to see how many critical, warning and info alerts have been sent in the last 10 minutes and the last hour. It’s publicly available via https://glance.mayfirst.org/. The goal of the site is to provide a quick, easy to access confirmation about whether any problems we have received alerts about are resolved or still active. Additionally, pigeon sends a heartbeat message every 10 minutes, and glance lists any servers that have not sent a heartbeat, so we can know if the reason there are not alerts is because pigeon has stopped running. https://admin.docs.mayfirst.org/pigeon/ Sun, 06 Dec 2020 13:53:34 +0000 https://admin.docs.mayfirst.org/pigeon/ pigeon is our anomoly detection program. pigeon runs on all servers and periodically executes checks for known problem conditions. When a problem condition is found, it simply logs to journald. pigeon reads a yml based configuration file located at /etc/pigon.yml to know which tests should be run for a given server. Additional tests can be added directly to pigeon as the need arises. There pigeon alerts are picked up in three ways: https://admin.docs.mayfirst.org/elasticsearch-upgrade/ Thu, 29 Oct 2020 13:53:34 +0000 https://admin.docs.mayfirst.org/elasticsearch-upgrade/ Elasticsearch has many different components that we use: elasticsearch: The server database itself journalbeat, metricbeat, and filebeat: which feed information from our servers to the databsae kibana: the web interface for reviewing data These three pieces of software are all released with the same version number at the same time. Because they are so tightly integrated, it’s a good idea to upgrade all of them together. To facilitate this process, the elasticsearch version number is included as a variable in the hosts. https://admin.docs.mayfirst.org/elasticsearch-index-lifetime-management/ Thu, 29 Oct 2020 13:53:34 +0000 https://admin.docs.mayfirst.org/elasticsearch-index-lifetime-management/ Index Lifetime Management (ILM) is how elasticsearch manages log rotation. In elasticsearch, the “index” is the log, so managing its lifetime allows us to control how long we retain data. The primary indices we are concerned with are the ones generated by journalbeat and filebeat, since they contain the most sensitive information (mainly IP addresses). However, metricbeat is important because it generates so much data - if we don’t manage the indices the partition holding the elasticsearch data will fill up. https://admin.docs.mayfirst.org/nextcloud/ Wed, 14 Oct 2020 13:53:34 +0000 https://admin.docs.mayfirst.org/nextcloud/ Nextcloud is a file, calendar, contact sharing web application (and much much more). May First administers a Nextcloud instance shared by all members - any user created in the control panel with access granted to Nextcloud can login to this shared instance. In addition, we can also can spin up instances dedicated to specific members. System design There are six main components to a Nextcloud installation: The database service: We use a Postgres server running on a dedicated virtual machine (psql003) The web service: We run nginx on the nextcloud KVM guest (nextcloud002). https://admin.docs.mayfirst.org/backup/ Wed, 07 Oct 2020 13:53:34 +0000 https://admin.docs.mayfirst.org/backup/ We use both backupninja, which provides a tool-agnostic framework for configuring backup jobs, and borgbackup, which performs a network backup of the data. Via ansible, we can designate a server’s backup_profile as either a target (the backup server) or a source (any server that should specify directories to backup). When defining sources, you can specify a list of remotes (in case you want to backup to more then one remote backup servers), and also a list of directories to include. https://admin.docs.mayfirst.org/nginx/ Mon, 06 Jul 2020 13:53:34 +0000 https://admin.docs.mayfirst.org/nginx/ Nginx is a web server. It provides both both a front-end proxy for other nginx or apache web servers in our network and also serves as a standard web service for other services. Web proxy You can configure a server to use an nginx proxy by adding a stanza to the server’s yml file. For example, in an ansible definition for a server called report001.mayfirst.org in which you will be launching a web service listening on port 5601 via the url: https://report. https://admin.docs.mayfirst.org/longshore/ Mon, 04 May 2020 13:53:34 +0000 https://admin.docs.mayfirst.org/longshore/ longshore is the system used by the Progressive Technology Project to manage it’s Powerbase installations. Using the longshore command, admins can create and manage docker containers running powerbase. Setup The big picture is: Powerbase is civicrm, hosted and maintained by PTP, specifically configured for organizing groups. PTP runs several physical hosts that are dedicated to the task, some are in the May First cabinet some are in other locations. Each physical host runs at least two KVM guests: https://admin.docs.mayfirst.org/mariadb/ Mon, 04 May 2020 13:53:34 +0000 https://admin.docs.mayfirst.org/mariadb/ What’s in a name? MySQL is the Kleenex of databases. While it is the name of a specific database, it has also become a generic term used to refer to all databases that are compatible with the original MySQL database. Per Debian defaults, we run MariaDB as our “MySQL” database. And, throughout our code and documentation, we refer to MySQL the same way we refer to Web server, Email server and other generic services: as a generic service that we fulfil with MariaDB. https://admin.docs.mayfirst.org/proxysql/ Mon, 04 May 2020 13:53:34 +0000 https://admin.docs.mayfirst.org/proxysql/ May First uses proxysql so databases appear to be on the localhost, while really living on dedicated database servers (origin servers). The proxysql package is installed on web servers and any server from which a client needs to connect to a database (these servers are referred to as proxy servers). Database clients (and client configurations) have no knowledge of where their database really lives - they simply attempt to connect via localhost. https://admin.docs.mayfirst.org/openssh/ Fri, 20 Mar 2020 13:53:34 +0000 https://admin.docs.mayfirst.org/openssh/ Openssh is used to provide shell access to all servers. We have several different needs for authentication: Our server authenticates regular members We will continue providing both password-based and ssh public key based authentication. Regular members authenticate our server Our user facing shell page will explain how all users will only ssh/sftp into one server with one domain name: shell.mayfirst.org (the server will mount all appropriate directories over sshfs). Therefore, our members will only need to verify one fingerprint. https://admin.docs.mayfirst.org/shell/ Fri, 20 Mar 2020 13:53:34 +0000 https://admin.docs.mayfirst.org/shell/ Goals The goals of our ssh/sftp access configuration are: Provide one address for all members to ssh into (e.g. shell.mayfirst.org) Users do not need to know which weborigin server their web sites are on, they are automatically given access to the right server when the ssh into the shell server. Ability to use ssh or sftp High level overview of how it works When a user ssh’s into shell.mayfirst.org, a ForceCommand configuration in sshd_config automatically ssh’s them into their configured origin server or logs them into the shell server while mounting their weborigin servers directories via sshfs. https://admin.docs.mayfirst.org/console/ Fri, 13 Mar 2020 13:53:34 +0000 https://admin.docs.mayfirst.org/console/ Sometimes network access to a server is not available, or you need to perform an operation before or during boot (like enter an encrypted passphrase). During these times, it’s useful to gain access to the server’s console. All servers, both bare metal and virtual servers, offer console access via ssh through a different server. Virtual servers Consoles on virtual servers can be accessed via a non-privileged user on the bare metal server that provides the virtual server. https://admin.docs.mayfirst.org/console-admin/ Fri, 13 Mar 2020 13:53:34 +0000 https://admin.docs.mayfirst.org/console-admin/ We use the cereal package to manage console access to our physical servers with hardware serial ports and the ipmiutils package to provide serial access for those that depend on IPMI. In both cases, the setup is automatically handled by ansible via the console variable. Physical serial port Here’s an example of setting up a hardware serial server: console: host: clr.mayfirst.org name: wiwa tty: /dev/ttyUSB29 id: FTDI_FT232R_USB_UART_ST161546 The host key indicates which serial console server it should be setup on. https://admin.docs.mayfirst.org/bruce/ Fri, 24 Jan 2020 13:53:34 +0000 https://admin.docs.mayfirst.org/bruce/ Bruce Banner bruce banner is an IP banning tool. It queries our elasticsearch database for intrusion attempts, manages a list of IPs to ban and communicates bans to a network of remote server responsible for enforcing the ban via nftables. See our intrusion detection page for the full picture of how it fits in with our other tools. Topics bruce can be configured to manage different ban “topics” for different services. https://admin.docs.mayfirst.org/dovecot/ Fri, 24 Jan 2020 13:53:34 +0000 https://admin.docs.mayfirst.org/dovecot/ TBD https://admin.docs.mayfirst.org/intrusion-detection/ Thu, 23 Jan 2020 13:53:34 +0000 https://admin.docs.mayfirst.org/intrusion-detection/ Our intrusion detection system is comprised of several components that work together: journalbeat and filebeat are programs that send logging files from every server in our network to our elasticsearch server. elasticsearch is a central database that consumes and indexes all the logs from all our servers. bruce specifically queries the journalbeat information in elasticsearch database for IP addresses engaging in suspicious behavior and sends them to the bruce-server to be distributed to all servers and banned via nft. https://admin.docs.mayfirst.org/keyserver/ Fri, 17 Jan 2020 13:53:34 +0000 https://admin.docs.mayfirst.org/keyserver/ Overview May First relies on public key infrastructure (PKI) throughout the network to ensure encrypted and authenticated communications between the public and our servers and between our servers themselves. In most cases, we use x509 (the standard format for public key certificates used for https and many other applications of transport layer security). For example: Members connecting via https to any of our sites, including their own web sites. Members connecting securely to our mail server to send or relay email Our internal relaying of email, e. https://admin.docs.mayfirst.org/filebeat/ Sun, 05 Jan 2020 08:53:34 +0000 https://admin.docs.mayfirst.org/filebeat/ General Info Filebeat sends selected files to our elasticsearch server. Filebeat can only send log files created by a service that already has a filebeat module. All the available modules are listed in /etc/filebeat/modules.d, with all modules disabled by default. Filebeat can be enabled for a server via it’s yaml file, e.g.: filebeat_enabled: true filebeat_profile: service filebeat_modules: [ "nginx" ] Setup All indexes and setup is carried out when you apply the ansible role to the elasticsearch server (see the index. https://admin.docs.mayfirst.org/postfix/ Sat, 04 Jan 2020 13:53:34 +0000 https://admin.docs.mayfirst.org/postfix/ Overview Postfix sends and delivers our email. We have several classes of postfix servers: MX: The mx servers accept incoming email destined for an address for which we will take responsibility, runs light validation checks, and relays the email to one of the filter servers. The MX servers are also responsible for verifying MX domains. A domain will send email to us if it has an MX record pointing to our one of these MX servers. https://admin.docs.mayfirst.org/simplemonitor/ Fri, 08 Nov 2019 13:53:34 +0000 https://admin.docs.mayfirst.org/simplemonitor/ Simlemonitor is a python daemon that monitors a set of remote services and custom commands and sends alerts whenver it detects a problem. Simplemonitor will also send a follow up alert when the problem is resolved. The set of services to monitor is automatically generated based on the simplemonitor variable in the hosts.yaml ansible file. For example: simplemonitor: # The key will be used as the label to indentify the resource that is offline in the alert. https://admin.docs.mayfirst.org/metricbeat/ Mon, 09 Sep 2019 13:53:34 +0000 https://admin.docs.mayfirst.org/metricbeat/ General Info Metricbeat sends selected server health data to our elasticsearch server. All servers (by default, not a requirement) run metricbeat with the default system module enabled. All servers share a metricbeat username and password authorizing them to connect to the es server. Index Lifetime Management Index Lifetime Management (ILM) is how elasticsearch manages log rotation and retention. It’s covered in it’s own ILM page. https://admin.docs.mayfirst.org/elasticsearch/ Tue, 13 Aug 2019 13:53:34 +0000 https://admin.docs.mayfirst.org/elasticsearch/ Elasticsearch (es) is is a database designed to injest large amounts of data while providing an easy to use query language to search it. It’s installed using es provided apt repos (not in Debian). The es server(s) are dedicated to just running Elasticsearch. Other servers and services work together with Elasticsearch, including: Journalbeat - all servers run Journalbeat, which sends journald data to the es server. Metricbeat - all servers run metricbeat, which sends server metrics data to the es server. https://admin.docs.mayfirst.org/journalbeat/ Tue, 13 Aug 2019 13:53:34 +0000 https://admin.docs.mayfirst.org/journalbeat/ General Info Journalbeat sends all journald logging to our elasticsearch server. All servers (by default, not a requirement) run journalbeat so their logs can be analzed in aggregate. All servers share a journalbeat username and password authorizing them to connect to the es server. Journalbeat doesn’t seem to work properly on /var/run/systemd/journalbeat (more testing necessary) so creating /var/log/journal seems to be required (which means more disk i/o) To minimize information overload, we only send data on select services (ssh by default, but others can be added via the journalbeat_send ansible variable). https://admin.docs.mayfirst.org/kibana/ Tue, 13 Aug 2019 13:53:34 +0000 https://admin.docs.mayfirst.org/kibana/ Kibana provides a user interface for querying the es server. kibana should be accessible via: https://report.mayfirst.org/ If you are in dev mode and you get a tls warning, be sure to import the certificate authority saved in inventory/testing/ca. The password for kibana should eventually be placed in keyringer but for testing purposes a user with the username mayfirst and password mayfirst is created and should grant you access. When you login for the first time: