Filebeat

General Info

Filebeat sends selected files to our elasticsearch server.

Filebeat can only send log files created by a service that already has a filebeat module. All the available modules are listed in /etc/filebeat/modules.d, with all modules disabled by default.

Filebeat can be enabled for a server via it’s yaml file, e.g.:

filebeat_enabled: true
filebeat_profile: service
filebeat_modules: [ "nginx" ]

Setup

All indexes and setup is carried out when you apply the ansible role to the elasticsearch server (see the index.yml file that is execute when filebeat_profile is set to index). The elasticsearch server should have all modules in use throughout the network listed in the filebeat_modules variable to ensure they are setup properly.

The only setup not handled automatically is the Kibana dashboards - which provide a useful visualization of the data collected by filebeat.

These must be created by hand. The following command will create all dashboards in Kibana:

filebeat setup --dashboards -E setup.kibana.host=KIBANA_HOSTNAME:443 -E setup.kibana.protocol=https -E output.elasticsearch.username=elastic -E output.elasticsearch.password=ELASTIC_PASSWORD

Whenever you add a new module, you may want to run that command again to ensure the relevant dashboards are created. However, since this command creates a lot of dashboards we don’t need or use, you should delete the extras after it is run.

Index Lifetime Management

Index Lifetime Management (ILM) is how elasticsearch manages log rotation and retention. It’s covered in it’s own ILM page.