Elasticsearch Upgrades

Elasticsearch has many different components that we use:

These three pieces of software are all released with the same version number at the same time. Because they are so tightly integrated, it’s a good idea to upgrade all of them together.

To facilitate this process, the elasticsearch version number is included as a variable in the hosts.yml file (m_elastic_version).

All servers use apt pinning to ensure we keep them at the right version.

And, when we change the version, ansible will automatically upgrade each server. However, these packages don’t trigger a restart after the upgrade, giving us yet more control of exactly how to manage it.

Steps to take when upgrading

  1. Bump the version number in the hosts.yml file and push out to just the log server. Use --tags elasticsearch_repos,elasticsearch,metricbeat,journalbeat,filebeat.
  2. Restart elasticsearch on the log server and ensure there are no errors.
  3. Run ELASTIC_PACKAGE=metricbeat mf-elastic-beat-setup, ELASTIC_PACKAGE=journalbeat mf-elastic-beat-setup, and ELASTIC_PACKAGE=filebeat mf-elastic-beat-setup- this will rebuild the indices and ensure ilm is setup properly for the new version. It deletes any existing indices for the current version so, it’s important to run this step before upgrading journalbeat and metricbeat on the other servers, otherwise we lose some logs.
  4. Restart metricbeat, filebeat and journalbeat on the es server.
  5. Push out via ansible to all servers (--tags elasticsearch_repos,metricbeat,journalbeat,filebeat,kibana ).
  6. Restart journalbeat, filebeat and metribeat on all servers via: ansible -i /path/to/hosts.yml 'all:!external' -m shell -a 'systemctl restart metricbeat && systemctl restart journalbeat' && systemctl cat filebeat.service >/dev/null && systemctl restart filebeat