How to respond to alerts

All alerts should come with a link to this page, which explains how to fix the problem.

elasticsearch health

The elasticsearch_health alert happens when accessing https://log.mayfirst.org/_cat/health does not return a 200 http response code. That might mean our elasticsearch database is under heavy load, is restarting or has failed. Login to log001.mayfirst.org and check /var/log/elasticsearch/elasticsearch.log and journalctl -u elasticsearch.

missing pigeon heartbeat

Every host running pigeon should send a heartbeat message once a minute. If this alert happens, it means we missed a heartbeat from the specified host. That might be because pigeon has crashed (try logging into the host and checking systemctl status pigeon). It might mean that our list of hosts to check is incorrect (the generated list in is monitor002 in /etc/simplemonitor/hosts.yml).

pigeon warning

A pigeon warning alert should come with additional information describing what triggered it.

Backup failed.

Log into the server and check /var/log/backupninja.log. You can test with:

    backupninja --test --now --debug --run /etc/backup.d/71-backup.borg

Disk space

certbot service failure

A common error is the certbot service failure error via key001.

  1. Login to key001
  2. Run: mf-cert-parse-failures
  3. For any failures, edit web conf in control panel and change from https to http.

Mail block alert

A mail block indicates one of the big providers is deferring or blocking our email.

Login to monitor002.mayfirst.org and run (replace S3150 with the search string reported):

   elastic-mail-status-message-by-instance --search S3150 --since 12h

This will give you a sense of how pervasive the problem is.

Then, use triptick to reduce the rate of messages going to the provider having trouble.

See mail queue management for more details.

pigeon critical

A pigeon critical alert should come with additional information describing what triggered it.

dovecot process limit

A dovecot process limit warning means we have seen a message like the following in the logs:

dovecot: master: Warning: service(imap-login): process_limit (100) reached, client connections are being dropped

This warning indicates that the dovecot server has too many connections.

Pay close attention to the server that is reporting the error, it might be a mailcf server - which are the servers that respond to requests to mail.mayfirst.org and proxy to the actual server, or it could be a mailstore server.

If we are reaching the limit due to normal usage, you can raise the limit by greping for ‘process_limit’ in the dovecot ansible role.

If we seem to be under attack, ensure that bruce-server.service and bruce-elastic.service are properly running on monitor002 (you may need to restart them).

postfix process limit

A postfix process limit might happen against mailcf servers or mailmx servers.

mailcf servers

This warning means we have a message like the following in the logs:

warning: service “submission” (587) has reached its process limit “100”: new clients may experience noticeable delays

This is a likely sign that we are experiencing a dictionary attack against our mailcf servers.

Ensure that bruce-server.service and bruce-elastic.service are properly running on monitor002 (you may need to restart them). Or, sometimes we just can’t keep with the dictionary attack and we might consider raising the limit.

You can search Elastic search for the message “SASL LOGIN authentication failed” to see the failures.

mailmx servers

If it’s against the mailmx servers, it might look like:

warning: service smtpd (private smtpd) has reached its process limit 100 : new clients may experience noticeable delays

This means we are getting hammered by a spam bot trying to deliver email to our users. It is likely being stopped by postfix but stopping it is consuming all available postfix connections. Unless this happenes for a sustained period or happens on all mailmx servers at the same time, it is not serious since it will only slightly delay receipt of email (mail clients will move to other maimx servers if this one is busy).

You can check the logs for entries like these:

Aug 06 12:27:04 mailmx002 postfix/smtpd[1183135]: warning: hostname mx.zeratte.or.ke does not resolve to address 141.98.10.114
Aug 06 12:27:04 mailmx002 postfix/smtpd[1183135]: connect from unknown[141.98.10.114]
Aug 06 12:27:04 mailmx002 postfix/smtpd[1183135]: warning: Connection concurrency limit exceeded: 51 from unknown[141.98.10.114] for service smtpd
Aug 06 12:27:04 mailmx002 postfix/smtpd[1183135]: disconnect from unknown[141.98.10.114] commands=0/0
Aug 06 12:27:04 mailmx002 postfix/smtpd[1183135]: warning: hostname mx.zeratte.or.ke does not resolve to address 141.98.10.114
Aug 06 12:27:04 mailmx002 postfix/smtpd[1183135]: connect from unknown[141.98.10.114]
Aug 06 12:27:04 mailmx002 postfix/smtpd[1183135]: warning: Connection concurrency limit exceeded: 51 from unknown[141.98.10.114] for service smtpd
Aug 06 12:27:04 mailmx002 postfix/smtpd[1183135]: disconnect from unknown[141.98.10.114] commands=0/0
Aug 06 12:27:04 mailmx002 postfix/smtpd[1183135]: warning: hostname mx.zeratte.or.ke does not resolve to address 141.98.10.114
Aug 06 12:27:04 mailmx002 postfix/smtpd[1183135]: connect from unknown[141.98.10.114]
Aug 06 12:27:04 mailmx002 postfix/smtpd[1183135]: warning: Connection concurrency limit exceeded: 51 from unknown[141.98.10.114] for service smtpd
Aug 06 12:27:04 mailmx002 postfix/smtpd[1183135]: disconnect from unknown[141.98.10.114] commands=0/0
Aug 06 12:27:04 mailmx002 postfix/smtpd[1183135]: warning: hostname mx.zeratte.or.ke does not resolve to address 141.98.10.114
Aug 06 12:27:04 mailmx002 postfix/smtpd[1183135]: connect from unknown[141.98.10.114]
Aug 06 12:27:04 mailmx002 postfix/smtpd[1183135]: warning: Connection concurrency limit exceeded: 51 from unknown[141.98.10.114] for service smtpd
Aug 06 12:27:04 mailmx002 postfix/smtpd[1183135]: disconnect from unknown[141.98.10.114] commands=0/0
Aug 06 12:27:04 mailmx002 postfix/smtpd[1183135]: warning: hostname mx.zeratte.or.ke does not resolve to address 141.98.10.114

MisterT Audit Relayers

If a email user has an unusually high number of bounces, we may get an alert along these lines:

monitor002-mistert_audit_relayers-critical: Possible compromised email relayers: sender: USER DOMAIN.ORG, connections: 361, ips: 13, failure rate: 5, messages: 712, sent: 675, bounced: 32, deferred: 5, expired: 0, spam virus: 0 0

The USER and DOMAIN.ORG will be replaced with the actual user and possibly the domain (notice that the @ sign is absent). Sometimes it will just be USER and not USER DOMAIN.ORG.

Some signs that the account is compromised include:

  • A high number of ip addresses
  • A high number of messages sent

Sometimes, however, it’s a legit message that just had a lot of bounces.

You can review the messages sent by logging into monitor002 and running:

mistert --since 1h list --sender USER@DOMAIN.ORG --list-recipients | less -R

If you are not sure if this is spam, you can find the queue id of any deferred messages. Then, lookup the queue id in kibana and find out which host and which postfix instance sent it.

Then, log in to the host and run:

postcat -c /etc/POSTFIX-INSTANCE -q QUEUEID | less

If it is spam, then:

  • Change the user account password
  • On each mailcf server, run systemctl restart saslauthd.service
  • On each mail relay server, run mf-mailq-delete SENDER
  • Notify the member
  • Confirm it has stopped with a kibana search: message: "sasl_username=USER@DOMAIN.ORG"

nginx connections

See below.

nginx worker connections

See below.

nginx open_files

All three of these messages mean the nginx server is getting pounded.

  1. Run mf-fastest-growing-logs to find out which site is getting pounded and disable the site and/or update the site’s DNS to point to a different web proxy (preferable one that is not being heavily used).
  2. Often, the requests will keep coming in becaues the DNS lookup is cached by the attackers. Edit /etc/nginx/site-enabled/001-placeholder.conf and add return 444 to the main location block. It’s faster for nginx to reject a request than server our default page.
  3. If nginx still can’t keep up, change the A record for the webproxy domain (e.g. {a,b,c,d}.webproxy.mayfirst.org) that is using the webproxy getting hammered. Change it to the IP of a different webproxy. That will trigger our nsstage002.mayfirst.org server to execute update-alias-records.service, which will update all the domains using that webproxy ALIAS. Be sure to disable the domain being attacked or first assign that domain to it’s own webproxy.