Pigeon
pigeon is our anomoly detection program. pigeon runs on all servers and periodically executes checks for known problem conditions. When a problem condition is found, it simply logs to journald.
pigeon reads a yml based configuration file located at /etc/pigon.yml to know
which tests should be run for a given server.
Additional tests can be added directly to pigeon as the need arises.
There pigeon alerts are picked up in three ways:
- An simplemonitor queries elasticsearch and picks up critical and warning reports and sends alerts
- A kibana dashboard can query for all alerts, including info alerts, and display them
- glance, a dead simple api-based web site that queries elastic search and provides a quick glance at whether or not we are experiencing problems.