After several weeks of intense testing, fixing configuration problems, re-indexing data and experiencing problems when upgrading our Kibana indices, we managed to upgrade our 36 Kibana instances and our Elasticsearch cluster in production from version 5.6.16 to 6.7.1 a couple of weeks ago.
We could not believe it but finally we had over 1.300 indices, 100TB of data, 102 000 000 000 documents and 18 Elasticsearch nodes running the last version of the elastic 6.x series at the moment.
I am writing this article with contradictory feelings, Am I having a bad day on top of The Oracle at Google not giving me answers, or is the documentation of the Keepalived software totally outdated and old?
If you have to administrate an Elasticsearch cluster, there are some common maintenance tasks that you will have to run to keep your data growth under control, backup your indexes and keep the cluster updated.
At the University of Oslo we have a 14 nodes Elasticsearch 5.x cluster (3 master + 2 clients + 4 SSD data+ 5 SAS data). We use it to manage, search, analyze, and explore our logs. It has around 100TB of total storage, around 1,300 indexes and we keep from 3 to 6-12 months of data per index type depending of the type of data they have.
One of the main plugins we were using with our 2.x Elasticsearch cluster was KOPF. This plugin was a web interface to the Elasticsearch API and it was an easy way of performing common tasks on our Elasticsearch cluster.
When we upgraded our Elasticsearch cluster to version 5.x, we could not continue using this plugin because it was not longer supported. The good thing was that the author of KOPF, Leonardo Menezes, had a new project called "CEREBRO" to offer an alternative to KOPF when running Elasticsearch 5.x.
We started using Zabbix to monitor the IT infrastructure at The University of Oslo in 2014. During all this time we have been running all our Zabbix servers on VMware virtual servers with an acceptable level of performance. This situation changed some months ago when the VMvare+Storage we were using did not have more available resources for us to grow and it was slowing down the future development of our monitoring system.
It has been a while since the last release of PgBackMan and with this version we hope to start a new period with more frequent releases. This new release implements some new features and fixes some bugs from version 1.1.0. Source files, RPM and DEB files are available at GitHub.
Add support for postgreSQL 9.6
Add dbname exception field to register_snapshot_definition
A new version of Zabbix-CLI has been released to the public. Source files, RPM and DEB files are available at GitHub.
This new version has new general improvements and new functionality to acknowledge events.
[Issue:#17]: Add a new command 'show_trigger_events' to show events
for a trigger.
Just before Christmas we launched a new version of the Zabbix-CLI client we have developed at The University of Oslo.
Zabbix-CLI 1.6.0 implements some new options for configuring the client and some new commands to handle maintenance periods in Zabbix via Zabbix-CLI.
To read about all the possibilities available to configure this client, you can use the Zabbix-CLI manual found at Github: https://github.com/usit-gd/zabbix-cli/blob/master/docs/manual.rst#configuration
A new version of Zabbix-CLI has been released. Source files, RPM and DEB files are available at GitHub.
We are working to get Zabbix-CLI in the official Debian repository and a repository for RPM packages will be available soon.
This new release of PgBackMan implements some new features and fixes some bugs from version 1.0.0. Read the"Migration to 1.1.0" section for important details before you upgrade to this new version.
