Simple Monitor

Simlemonitor is a python daemon that monitors a set of remote services and custom commands and sends alerts whenver it detects a problem.

Simplemonitor will also send a follow up alert when the problem is resolved.

The set of services to monitor is automatically generated based on the simplemonitor variable in the hosts.yaml ansible file.

For example:

simplemonitor:
  # The key will be used as the label to indentify the resource that is offline in the alert.
  server-name-ping:
    # Common types are 'host' (ping), 'http', or 'tcp'
    type: host
    host: domain.org
  server-name-http:
    type: http
    url: https://joe.was.org/
    # Path to certs to verify (might not be needed in production)
    verify_hostname: /etc/ssl/certs/ca-certificates.crt 
  server-name-smtp:
    type: tcp
    host: domain.org
    port: 25 

Checking elasticsearch for signs of trouble

Simple monitor also uses simple scripts to query our elasticsearch server for the presence of pigeon alerts and also any signs of trouble that may get logged via journald -> elasticsearch (like reaching the maximum number of postfix processes, or too many nginx files open).

The timing is imprecise. For example, pigeon might check for a problem every ten minutes resulting in an alert in journald every 10 minutes that the problem is present.

Simplemonitor will check about every 30 seconds for the presence of the pigeon alert in elastic search - with a search that goes back in time ten minutes.

That means we should always get an alert in a timely manner, but the alert may not be “cleared” immediately when the problem is resolved.

Alerts and re-alerts

Simplemonitor uses our alerter to send alerts. By default all monitor failures are “urgent” which means they set --priority critical to alerter which sends it both via signal and email. Some monitor failures have urgent = false which means they set --priority warning, which results in just an email.

Critical alerts will be re-sent approximately every 20 minutes during regular working hours (2:00 pm - 10:00 pm UTC). Warning alerts will be re-sent approximately once a day.

Note: If we get two pigeon critical alerts, one right after the other, the second one will not be sent until the re-alert time has passed because simplemonitor does not distinguish between different pigeon critical alerts. For this reason it is always important to fix critical alerts in a timely fashion.

Automatic monitoring of Lets Encrypt certificates

Simple monitor (on the monitor002 server) will also automatically monitor any certificate that is managed by our keyserver if you add the monitor port to the key server configuration as specified in the “Monitoring” section of the keyserver page. We should get an alert if a certificate is within 7 days of expiring.

Now what?

We also have a page describing what to do when you get an alert.

What If…

Just in case simplemonitor fails, we have a custom systemd timer called watch-the-watcher.timer on all monitor servers that will send us an alert if it detects that the simplemonitor service is not running.