Mr. T: Querying elastic search for email related info in our logs

April 12, 2022

Mr. T provides a number of email reporting features based on mail log data collected by elastic search.

See the upstream README for documentation on usage - in particular how to get a list of senders and recipients to track down potential email problems.

General review and searching

mistert list and mistert stats print lists and statistics about email sent through our network. For example:

mistert --since 2h --until 1h list will print a line for every message sent between 2 hours ago and one hour ago, including who it was sent from and the message id.

You can get a list of recipients for a given message id with:

mistert --since 2h list --message-id 123@abc --list-recipients

You can also find all messages sent by a given sender with:

mistert --since 2h list --sender foo@bar.org

If a sender is an email list in which every from address is unique (due to VERP), you can specify a wildcard sender:

mistert --since 2h list --sender 'foo-list*lists.mayfirst.org'

If you want to see just one line of stats (sent, bounced, deferred, etc) for the list:

mistert --since 2h stats --sender 'foo-list*lists.mayfirst.org'

When we get blocked

mistert blocks will count how many messages bounced and how many of those had a well-known bounce message associated with spam or otherwise being blocked.

Typically, about 2 - 3% of bounce messages come with a spam/block-ish wording.

You can also limit by a postfix instance, an outgoing relay, and you can show a count of from addresses that caused the bounce and even see the bounce message itself.

mistert --since 3d blocks --relay outlook.com --instance postfix-bulk044 --show-from

Prebounce

mistert prebounce generates a list of email addresses that should be prebounced before entering your outgoing, relay email queue based on previous delivery errors.

Overview

Busy mail relay servers need to protect their reputations by keeping the number of invalid email addresses to a minimum. Furthermore, mail administrators need to review bounced and deferred messages to identify potential system-wide problems, but invalid email addresses muddy the waters, making it hard to parse through the bounce and deferred messages that could spell trouble.

Mr. T solves these problems by generating a list of email addresses with previous delivery errors (indicating that the address is invalid) so they can be blocked before entering your outgoing mail queue in the future.

Details

Mr. T is specifically designed to query, via mistert prebounce --mode collect, an elastic search database populated with postfix logs. It reads in a configuration file that specifies elastic search login information, a path to a sqlite database, and one or more invalid email patterns.

mistert prebounce --mode collect runs via a systemd timer (e.g. once an hour) to search for errors indicating invalid addresses over the previous time period and writes them to the sqlite database.

Meanwhile, mail relay servers ssh into the server running Mr. T and query the database using mistert prebounce --mode list, which produces a list of email addresses that should be blocked. You can run this command to verify if an email address is on the prebounce list.

Identify reason address was banned

You can get more details about why a specific address was banned by consulting the sqlite db.

sqlite3 /var/lib/mistert/db/mistert.db
SELECT * FROM email WHERE email = 'email@banned.org'

Removing addresses

You can remove an address from the list with:

sqlite3 /var/lib/mistert/db/mistert.db
DELETE FROM email WHERE email = 'email@toremove.org'

Then, you can either wait one hour for the mailrelay servers to update their list, or you can login to each of the mailrelay servers are manually run:

systemctl start mistert-retriever

More Details

Mr T uses an algorithm to decide when an email addresses should be dumped.

The first time an invalid email address is detected, it is entered in the sqlite database with the first and last columns populated with a timestamp representing the first time it was seen. If the same addresses is detected a second time, the last column is updated with the timestamp of the most recent event.

An email address is only added to the block list if the first and last columns are more then 24 hours apart. This approach avoids blocking email addresses that generate errors due to a temporary misconfiguration of the receiving server.

As an additional protection against incorrectly blocking an address, as long as the first and last columns are less then 24 hours apart, Mr T will search for any successfully sent message and, if found, will purge the address from the database.

All email addresses are purged after 6 months.

Audit relay users

mistert, via the mistert alert --host-type 'cf' command, can also review a list of all users who authenticate and relay email via our mailcf (client-facing) servers. These are the mail.mayfirst.org servers used by all email clients.

This command is run by pigeon and sends alerts when a username appears to be compromised.

By default, with no arguments, it will exit with exit status 0 and no output if, in the last 15 minutes, no username has exceeded the default threshholds. If a user exceeds the default threshhold then a brief report for each username will be printed along with the number of IPs they have connect via as well as counts of their sent, bounced and deferred email.

The defaults (and available arguments) are:

–login-limit - how many logins are allowed before we do a deeper analysis? Default: 15.
–location-limit - if we do a deeper analsysis, how many different locations (either different root domains if the IP address has a reverse domain name lookup OR how many different IPs if there is no reverse lookup) should trigger an alert. Default: 5
–delivery-failure-limit - if we do a deeper analysis, what percentage of failed deliveries should trigger an alert? Default: 10 percent.

Why Mr T

Mr T is an actor who was discovered by Sylvester Stallone after winning NBC’s “America’s Toughest Bouncer” competition.