Mail Queue Management

If our email is being blocked or deferred or if we just trying to warm up new IPs, we wil need to get reporting information about our mail queues and/or move messages between queues to ensure they are delivered.

A few concepts

Be sure to read up on relay servers on our postfix page and our relay servers page.

To review: We have multiple mail relay servers (name mailrelay001, mailrelay002, etc). Each server runs multiple postfix instances (e.g. /etc/postfix-bulk001, /etc/postfix-priority002, etc).

We can move messages between instances, but only instances on the same server.

Also, not all instances have an IP address assigned to {relay,bulk.priority}.relay.mayfirst.org. Some are simply available to be used as “fallback” instances if an active queue is being deferred.

The domain fallback.relay.mayfirst.org is an A record pointing to the IP addresses of fallback instances.

In the ansible file, for a given mail relay instance, you can set these variables to control how frequently an instance delivers email to the same domain:

default_destination_recipient_limit: 2
default_destination_rate_delay: 30m

Be sure to set these variables before you start sending with a given instance that is not warmed up!

In contrast, we maintain the domain warm.relay.mayfirst.org as an A record pointing to the instances that are already warmed up.

General reporting

On monitor002, we have several scripts to help you get a sense of what is happening as a whole.

elastic-mail-status-message-by-instance

This script allows you to search for any message rejected or deferred with a specific error code.

It will break down the count for each postfix instance (with an asterisk to indicate if the instance is currently in rotation).

usage: elastic-mail-status-message-by-instance [-h] [--conf CONF] [--since SINCE] [--until UNTIL] --search
                                               SEARCH [--alert]

Summarize email relayed with a given string.

optional arguments:
  -h, --help       show this help message and exit
  --conf CONF      Path to configuration file, default /etc/simplemonitor/elastic.yml
  --since SINCE    Limit to records since the given time, 10m by default
  --until UNTIL    Limit to records until the given time Now by default
  --search SEARCH  Search for the given string
  --alert          Run in alert mode. Only output number of hits

elastic-mail-delivery-by-instance

This report gives you a summary of each instance of a given type, so you can see if one instance is deferring an unusually large number of messages:

usage: elastic-mail-delivery-by-instance [-h] [--conf CONF] [--since SINCE] [--until UNTIL] --relay-type RELAY_TYPE

Report mail delivery status by instance and relay type.

optional arguments:
  -h, --help                show this help message and exit
  --conf CONF               Path to configuration file, default /etc/simplemonitor/elastic.yml
  --since SINCE             Limit to records since the given time, 10m by default
  --until UNTIL             Limit to records until the given time Now by default
  --relay-type RELAY_TYPE   bulk, filtered or priority

elastic-mail-delivery-by-corporate-host

If you want to assess our performance with a given corporate host, run this command to see the number of messages sent, bounced or deferred.

usage: elastic-mail-delivery-by-corporate-host [-h] [--conf CONF] [--since SINCE] [--until UNTIL] [--corp CORP]

Print delivery stats for the main corporate hosts.

optional arguments:
  -h, --help     show this help message and exit
  --conf CONF    Path to configuration file, default /etc/simplemonitor/elastic.yml
  --since SINCE  Limit to records since the given time, 10m by default
  --until UNTIL  Limit to records until the given time Now by default
  --corp CORP    Either google, microsoft, or yahoo

mistert relaysummary

If you want a summary of who sent email to a given relay host (e.g. outook, yahoo, etc), you can get a sense with mistert.

usage: mistert relaysummary [-h] --relay RELAY

optional arguments:
  -h, --help     show this help message and exit
  --relay RELAY  Choose the relay domain that will be searched, e.g. outlook, or yahoodns. Wild card character will be added to front and back.

mf-mailq-summary

On each mail relay server, run mf-mailq-summary to see which instances on a given host have a back log of deferred or active messgaes.

root@mailrelay001:~# mf-mailq-report
Queue                Ip              Defer  Active
postfix-bulk008      xxx.xxx.xxx       300    0
postfix-bulk020      xxx.xxx.xxx.4     0      0
postfix-bulk021      xxx.xxx.xxx.5     31     1
postfix-bulk040      xxx.xxx.xxx.3     267    2
postfix-bulk041      xxx.xxx.xxx.4     0      167
postfix-bulk043      xxx.xxx.xxx.6     255    1
postfix-bulk044      xxx.xxx.xxx.7     0      0
postfix-bulk045      xxx.xxx.xxx.8     433    0
postfix-bulk046      xxx.xxx.xxx.9     0      191
postfix-bulk047      xxx.xxx.xxx.10    0      127
postfix-bulk048      xxx.xxx.xxx.11    0      150
postfix-bulk049      xxx.xxx.xxx.12    0      146
postfix-filtered005  xxx.xxx.xxx.94    0      0
postfix-filtered006  xxx.xxx.xxx.5     765    0
postfix-priority004  xxx.xxx.xxx.2     90     0
root@mailrelay001:~#

Common tasks

Spammer on the loose

Sometimes a spammer has flooded our servers with spam and we just want to delete every message sent by a given email address.

mf-mailq-delete FROMADDRESS will do this in a every instance on the given mail relay server.

Getting deferred or blocked by a particular provider

If a particular provider is deferring or blocking us, we need to either stop deliveries to that provider or, if possible, reduce the rate at which we send email to them.

triptick is the best tool for this job. It is installed on all our mail relay servers, but is not configured to run by default.

  1. Check the A record for fallback.relay.mayfirst.org. Ensure it is reporting all our fallback IP addresses and that each postfix instance using those IP addresses is properly configured with:
default_destination_recipient_limit: 2
default_destination_rate_delay: 30m

  1. Pick a mailrelay that is not currently running an instance of triptick. Edit /etc/systemd/system/triptick.service. For the --mx argument, add the last two parts of the MX server causing us the trouble (e.g. example.org). If every message is being blocked, then do not set a --interval. However, if some messages are getting through, set an interval like --interval 4h to we continue sending a trickle of messages to keep the IP warmed up. Lastly include --fallback fallback.relay.mayfirst.org so send all traffic to our fallback IPs (this helps ensure our good IPs don’t get blocked as well).

  2. In ansible, configure the Postfix instance with:

    check_recipient_mx_access: tcp:mailrelay001.mayfirst.cx:1724

The domain should match the mail relay you setup triptick to run on (note the .cx domain which resolves to our private IP range) and the port should match the port configured for triptick.

  1. Restart the postfix instance: systemctl restart postfix@postfix-bulkNNNN

  2. Submit a support request with the provider to request having the block removed. Either follow the directions in the defer/block message, or see our email-abuse page for details on contacting the big providers.

Dealing with a backlog of queued messages

If a postfix instance has a huge backlog of deferred email, you use mailq-manager to slowly move deferred messages to other postfix instances on the same server.