Overview of Monitoring and Alerting

May First uses a variety of tools that work together to provide monitoring and alerting of our systems to warn us of impending problems or immediate outages.

There are three main jobs:

  • Detecting problems
  • Communicating problems to a central storage location
  • Notifying us of the problems

The heart of our system is our central elasticsearch database.

Information is sent from each individual server to elasticsearch via:

  • journalbeat, which sends select data from journald
  • metricbeat, which sends server health data,
  • filebeat, which sends selected log files,
  • pigeon, which detects anomolies and outputs them to journald
  • simplemonitor, which tests network connectivity and runs a command to ensure elasticsearch is properly working.

Admins are notified of problems via:

  • simplemonitor, the primary alerting system.
  • kibana, the web interface to the elasticsearch data, which has dashboards to display all info, including non-urgent information that can help us spot problems before they become urgent.
  • glance, a dead simple api-based web site that queries elastic search and provides a quick glance at whether or not we are experiencing problems.

Simplemonitor pushes alerts to administrators via alerter, which picks the right medium (email or signal) based on the priority of the alert.

Notifications

When, how and what should admins be notified about?

Every notification should fall into one of three priorities:

  • Critical
  • Warning
  • Info

Critical

These notifications are the most urgent and typically would be sent to cell phones (e.g. via signal).

There is an expectation that critical warnings will be fixed within 4 hours (and they typically re-alert every four hours).

Since they are intrusive and stress inducing, they should only be sent when there is a real expectation that the sysadmin getting the alert will stop what they are doing and address the problem.

Examples warranting a critical alert:

  1. When there is a problem users are noticing now or are about to notice.
  • Yes: Partition is 95% full, No: partition is 85% full
  • Yes: Connections to the web server are timing out, No: load average is getting high
  1. When there is a problem that immediate action can fix
  • Yes: a service has crashed, No: there’s a lot of disk i/o
  • Yes: a user account is probably compromised, No: a user is using lots of resources
  1. When a problem requires immediate action
  • Yes: a production server is offline, No: a backup failed
  • Yes: the central authentication server is failing, No: a dev or staging server crashed

Warning

Warnings are less urgent and typically would be sent by email.

Warnings should be resolved within about 24 hours (and they typically re-alert every four hours).

A critical alert is warranted when there is an impending problem that is not critical:

  • Yes: Partition is 85% full, No: a user just hit 100% of their disk capacity
  • Yes: RAID failure or disk errors detected

Info

Info notifications are not pushed out to administrators. Instead they are displayed in a dashboard. Warnings are designed to give us a heads up about something that is not a problem now, but could develop into a problem.

For example:

  • Repeated read errors in a backup
  • A service temporarily hits a max usage counter

What to do when you are alerted?

We mainain a what to do doc with instructions on what steps to take when an alert arrives.

Banning IPs

In addition to alerts sent to humans, our alert system can also trigger bruce, our IP banning software, to ban an IP.

Finding resource hogs

All servers have the sysstat package installed, which allows us to run sar to show a quick summary of CPU usage to track changes over time.

In addition, our Database, Web, and Mail servers are have a resourcehog script that breaks down CPU usage by user to identify any users that are significantly bigger users of our resources than others.

Both sar and resourcehog take a second argument to specify how many days back to go, e.g. resourcehog -1 to go back one day, or -2 to go back two days.