Mailman email list software

May First runs the mailman mailing list manager.

Components

Mailman3 is broken into several discrete components:

  • mailman - the core component that controls the sending of email, configured via /etc/mailman3/mailman.cfg. List settings are stored in a postgres database.
  • mailman-web - the web-based companion service (written in django), configured via /etc/mailman3/mailman-web.py that consists of two django apps:
    • postorius - the django app that allows list owners to manager their own list settings and subscribers
    • hyperkitty - the django app that provides access to list archives if a list is configured to archive messages.

We run all three parts (and the postgres database) on the same server.

Managing mailman in general

Unfortunately, there is no unified command line tool for managing mailman. Instead, the mailman documentaiton provides a hodge podge of different commands, python code snippets and other methods for making changes.

The main commands available to us are:

  • mailman-wrapper The mailman command is used for managing lists (create, delete, etc). This command should always be run as the list user, not as root, so Debian conveniently provides mailman-wrapper which simply takes all arguments and calls them with mailman but as the list user. If you see Mailman documentation asking you to run the mailman command, always run the mailman-wrapper command instead.

  • API For some reason, there are a lot of mailman configuration options that cannot be set with the mailman command, but instead, if you are running the django web app, can be configured via an API call to the django API (available via http://localhost:8001). This requires authenticating with the api user and password configured in /etc/mailman3/mailman-web.py. There are no convenience commands to deal with this in an abstract way, but we have several specific commands that invoke the API for May First needs (see below).

  • mailman-web Typically, django apps come with a cli that can be used to accomplish tasks via the command line. In a typical django installation, it is invoked with python3 manager.py <command>, which requires you to be in the right directory where “manage.py” can be found and also requires you to run as the web user. Debian provides a convenience command, mailman-web that does this for you. So, anytime you see Mailman documentation directing you to run python3 manage.py <command> you can simply run mailman-web <command> instead.

Managing Mailman - for May First

May First has written a few helpers to facilitate the changes we need and help simplify the integration with the control panel.

Default settings

To ensure all new lists have consisent initial settings, we have written the following python snippets, which get copied into /usr/local/lib/python3.13/dist-packages/ by ansible, and are invoked via mailman-wrapper --run <snippet> --listspec <listname@listdomain.org>:

  • mayfirst_default: Applies base line default settings (e.g. don’t advertise the list, don’t send welcome message, max message size, etc).
  • mayfirst_announce: Settings appropriate for announcement lists
  • mayfirst_discuss: Settings appropriate for discussion lists.

These are applied when a list is created or if a user chooses to change a list from a discussion to announce, or vice versa.

Make owner

When a list is created, we can add the owner as part of the list creation process. But, if a user changes the owner of an existing list, we run:

mailman-wrapper withlist --run mayfirst_make_owner --listspec <listname@listdomain.org> <owner@mail.org> to make the given email address an owner.

Utilities

Finally, have a catchall mayfirst_utilities script for one off functions. It includes the following commands.

  • set_domain_base_url: ensure that any given mail domain is configured with a base_url to ensure that the $mailinglist_url variable used in the footer is populated, so each list can include a link to unsubscribe. It is invoked with: mailman-wrapper shell --run mayfirst_utilities.set_domain_base_url <domain>
  • get_subscriptions: for a given email, report all the email lists they are subscribed to: mailman-wrapper shell --run mayfirst_utilities.get_subscriptions <email>

Templates

Mailman has a complex Templating system. Templates include the welcome message for a new list, the footer sent to each list, and many others.

There are (at least) two ways this is complicated:

  • Requires API Setting or modifying a template requires the use of the API (or it can be set on a list-by-list basis via the web interface by a list owner).
  • Caching Mailman is not configured with the contents of a template, but instead the location of the template. That means the actual template is cached, so if it changes you have to wait or manually clear the cache.

For server wide default templates, we use the following approach:

  • Ansible places the template in a file in /etc/mailman3/mayfirst-templates
  • We call a custom script that invokes the API: mf-mailman-template <template:name> /etc/mailman3/mayfirst-templates/<template.name.txt>

When a list owner sets a template, it is stored in /var/lib/mailman3/templates/lists.

Changing a template

If we want to change one of the default May First templates:

  • Make the change to the template in ansible (in the “mailman” role, templates directory).
  • Use ansible to push the template change to list001.

Note: This will work eventually, but not right away. Because mailman is configured to point to the location of the template, rather than with the contents of the template, mailman caches the content. I have no idea for how long. I have no idea how to properly clear the cache. But this works:

  • su - postgres
  • psql mailman
  • DELETE FROM file_cache;

Django backend

Although many settings are found in /etc/mailman3, some are tucked away in the Django admin web interface. To get there, you have to go to the /mailman3/admin URL of the web interface. The password is in keyringer.

Debugging

Check /var/log/mailman3/ for general logs - or if it’s web login/archiving related, look in /var/log/mailman3/web/

Email constraint errors

You may see this error in the web logs:

psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint “unique_verified_email” DETAIL: Key (email)=(xxxx@verizon.net) already exists.

This error can happen under the following conditions:

  1. Mary creates a user account with their email address (mary@example.org). Then, to help their friend Sam, they add Sam’s email address (sam@example.org) to their own account and ask Sam to click through the link to verify it. Now Mary can manage Sam’s lists for them.

  2. Eventually Sam wants to manage their own lists, so they register for an account. For some reason Mailman let’s this happen. Sam get’s an email to verify their account and they click through it, and now we have a mess.

tldr;

  1. Fix the web database:
  • Login to https://a.lists.mayfirst.org/mailman3/admin as the admin user (see keyringer).
    • Navigate to users
    • Lookup the new user that was created (sam@example.org).
    • Delete it.
  • su - postgres
    • psql mailmanweb
      • DELETE FROM account_emailaddress where email = 'sam@example.org';
      • \quit
    • psql mailman
      • UPDATE address SET display_name = 'Sam', user_id = NULL WHERE email = 'sam@example.org';
      • \quit
  • Tell the Sam to re-register their account

WTF?

Mailman3 is two systems that are completely separate: the mailman core system for sending/distributing email and the mailman web system for archives and user management.

In mailman core, there is a address table for every email address it tracks. Some of these addresses have a user_id table that is a foreign key to the user table.

In mailman web, there is an account_emailaddress table for every email address associated with a user account. All of these records have a user_id field that corresponds to the id field in the auth_user table.

What if anything links these together is a mystery. But it seems that:

  1. When Mary registers her account via the mailman web system, it:
    • It creates the address in the mailman web database’s account_emailaddress table linking it to Mary’s user.
    • creates a record in the mailman core user table.
    • creates or updates the address table for the row containing her address with the id in the user table.
  2. When Mary adds sam@example.org, it does the same thing, but links it to the existing user account for Mary in both databases.

Now, mailman core and mailman web know that both mary@example.com and sam@example.com belong to the same user.

When Sam tries to create a new account, the system gets confused about who owns the sam@example.org and tries to insert it into the mailman web address table

  • but that fails because that address already exists there. And, it seems to be because the corresponding address in the mailman core address table has a value set for the user_id field. By setting that field to NULL, it tells the system: nope, this address is not associated with any users.

One important note: NEVER delete a record from the mailman3 core address table because they will unsubscribe them from every list they are on.

Issue filed upstream.