Elasticsearch setup

Elasticsearch (es) is is a database designed to injest large amounts of data while providing an easy to use query language to search it.

It’s installed using es provided apt repos (not in Debian).

The es server(s) are dedicated to just running Elasticsearch. Other servers and services work together with Elasticsearch, including:

• Journalbeat - all servers run Journalbeat, which sends journald data to the es server.
• Metricbeat - all servers run metricbeat, which sends server metrics data to the es server.
• Filebeat - some servers run filebeat, which sends the contents of a given log file to the es server (e.g. nginx).
• Kibana - provides a user interface for querying the es server via https://report.mayfirst.org/.
• Simplemonitor - queries the data going into es and provides alerts when pre-defined conditions are met.

User management

We use basic auth with pre-defined usernames and passwords to grant access to the various programs that send and/or monitor our es server, including kibana_system, journalbeat_writer, and simplemonitor users.

Their passwords are set in a ansible vault file called secrets.yml kept in the inventory.

API helper

On the elasticsearch server, we have mf-elastic-api which is a helper for sending commands to the serveri curl. It auto-populates the user credentials, making it easier to experiment with the otherwise unweildy curl arguments.