Journalbeat

General Info

Journalbeat sends all journald logging to our elasticsearch server.

All servers (by default, not a requirement) run journalbeat so their logs can be analzed in aggregate. All servers share a journalbeat username and password authorizing them to connect to the es server.

Journalbeat doesn’t seem to work properly on /var/run/systemd/journalbeat (more testing necessary) so creating /var/log/journal seems to be required (which means more disk i/o)

To minimize information overload, we only send data on select services (ssh by default, but others can be added via the journalbeat_send ansible variable).

In addition, we use the dissect processor which allows us to cut up a journald message into separate fields. For example, we search for fields indicating login failures and use dissect to ensure that the client IP ends up in it’s own es field (dissect.client_ip).

Index Lifetime Management

Index Lifetime Management (ILM) is how elasticsearch manages log rotation and retention. It’s covered in it’s own ILM page.