Index Lifetime Management

Index Lifetime Management (ILM) is how elasticsearch manages log rotation.

In elasticsearch, the “index” is the log, so managing its lifetime allows us to control how long we retain data.

The primary indices we are concerned with are the ones generated by journalbeat and filebeat, since they contain the most sensitive information (mainly IP addresses). However, metricbeat is important because it generates so much data - if we don’t manage the indices the partition holding the elasticsearch data will fill up.

Not so easy

There are many moving parts to successful index management in elasticsearch. The best sources of information are:

• The general elasticsearch index management docs
• The metricbeat ilm docs
• The journalbeat ilm docs
• The filebeat ilm docs

There important parts of index management are:

• index template: this must be created on the elasticsearch server for each data source (metricbeat, journalbeat, filebeat etc). It controls, among other things, how the indices will be named (e.g. journalbeat-7.9.2-2020-10-21-00001). journalbeat, filebeat and metricbeat can be configured to auto create their own index templates, however, that requires that the metricbeat, filebeat and journalbeat writer users have elevated permissions, so we disabled this option via the setup.tempalte.enabled setting. Instead, we manually create the templates by running a command that executes with elevated permission.

  When a server is initially setup, this index template creation command is automatically run by ansible.

  It has to be manually run every time we upgrade the elasticsearch stack (see the elasticsearch-upgrade page.)

  Additionally, by default index templates are configured to store each index with at least one replica (i.e. one extra copy on a second elasticsearch server). Since we only run one elasticsearch server, this causes problems - specifically, each index is in a “yellow” status instead of a “green” status.

  Index rotation can only take place on “green” indices, so we fix this problem by adjusting the index template created by journalbeat, filebeat and metricbeat to specify 0 replicates (see the elasticsearch upgrade page for more info on how we do that.).

• ILM Enabled: In addition, each service has to have ILM enabled. With metricbeat, filebeat and journalbeat, this option is set to auto - which means it will be enabled if the elasticsearch server supports it.

• ILM Policy: Finally, we have the ILM policy. This is where you can specify how the indices are rotated. There are many options. In the case of metricbeat, filebeat and journalbeat, they already specify an index template that changes every day. So, those get rotated without an ILM policy. The ILM policy, therefore, controls when they get deleted. Our policy specifies that after one day, they go from “hot” to “delete”. Then after a specified period, they are deleted.

  This policy is updated via the secript that runs on upgrades.