System Administration https://e-mc2.net/system-administration-posts en Behind the scenes of an ELK system https://e-mc2.net/behind-scenes-elk-system <span property="schema:name">Behind the scenes of an ELK system</span> <span rel="schema:author"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">rafaelma</span></span> <span property="schema:dateCreated" content="2020-01-20T07:48:01+00:00">Mon, 20/01/2020 - 08:48</span> <div class="field field--name-body field--type-text-with-summary field--label-above"> <div class="field__label">Body</div> <div property="schema:text" class="field__item"><p>Behind every security measure you take, you should have an information management system helping you take decisions. If you work with security, you need a way to collect, process, save and analyze huge amounts of data that should be used to control how your systems are behaving, find anomalies and evaluate the results of your actions.</p> <p>Have you ever wondered how to manage billions of logs and metrics from thousands of devices in your infrastructure? If you need high-availability and a resilient and stable system to process your data this is the tutorial for you.</p></div> </div> Mon, 20 Jan 2020 07:48:01 +0000 rafaelma 120 at https://e-mc2.net Elasticsearch in garbage collection hell https://e-mc2.net/elasticsearch-garbage-collection-hell <span property="schema:name">Elasticsearch in garbage collection hell</span> <span rel="schema:author"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">rafaelma</span></span> <span property="schema:dateCreated" content="2019-05-30T14:56:53+00:00">Thu, 30/05/2019 - 16:56</span> <div class="field field--name-body field--type-text-with-summary field--label-above"> <div class="field__label">Body</div> <div property="schema:text" class="field__item"><p>After several weeks of intense testing, fixing configuration problems, re-indexing data and experiencing problems when upgrading our Kibana indices, we managed to upgrade our 36 Kibana instances and our Elasticsearch cluster in production from version 5.6.16 to 6.7.1 a couple of weeks ago.</p> <p>We could not believe it but finally we had over 1.300 indices, 100TB of data, 102 000 000 000 documents and 18 Elasticsearch nodes running the last version of the elastic 6.x series at the moment.</p></div> </div> Thu, 30 May 2019 14:56:53 +0000 rafaelma 119 at https://e-mc2.net Keepalived - A documentation nightmare https://e-mc2.net/keepalived-documentation-nightmare <span property="schema:name">Keepalived - A documentation nightmare</span> <span rel="schema:author"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">rafaelma</span></span> <span property="schema:dateCreated" content="2018-03-06T20:40:47+00:00">Tue, 06/03/2018 - 21:40</span> <div class="field field--name-body field--type-text-with-summary field--label-above"> <div class="field__label">Body</div> <div property="schema:text" class="field__item"><p>I am writing this article with contradictory feelings, Am I having a bad day on top of The Oracle at Google not giving me answers, or is the documentation of the Keepalived software totally outdated and old?</p></div> </div> Tue, 06 Mar 2018 20:40:47 +0000 rafaelma 118 at https://e-mc2.net Elasticsearch - Common maintenance tasks https://e-mc2.net/elasticsearch-common-maintenance-tasks <span property="schema:name">Elasticsearch - Common maintenance tasks </span> <span rel="schema:author"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">rafaelma</span></span> <span property="schema:dateCreated" content="2017-07-10T11:23:53+00:00">Mon, 10/07/2017 - 13:23</span> <div class="field field--name-body field--type-text-with-summary field--label-above"> <div class="field__label">Body</div> <div property="schema:text" class="field__item"><p>If you have to administrate an Elasticsearch cluster, there are some common maintenance tasks that you will have to run to keep your data growth under control, backup your indexes and keep the cluster updated.</p> <p>At the University of Oslo we have a 14 nodes Elasticsearch 5.x cluster (3 master + 2 clients + 4 SSD data+ 5 SAS data). We use it to manage, search, analyze, and explore our logs. It has around 100TB of total storage, around 1,300 indexes and we keep from 3 to 6-12 months of data per index type depending of the type of data they have. </p></div> </div> Mon, 10 Jul 2017 11:23:53 +0000 rafaelma 111 at https://e-mc2.net Access to Elasticsearch with Cerebro via SSL+LDAP https://e-mc2.net/access-elasticsearch-cerebro-sslldap <span property="schema:name">Access to Elasticsearch with Cerebro via SSL+LDAP</span> <span rel="schema:author"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">rafaelma</span></span> <span property="schema:dateCreated" content="2017-06-25T13:53:42+00:00">Sun, 25/06/2017 - 15:53</span> <div class="field field--name-body field--type-text-with-summary field--label-above"> <div class="field__label">Body</div> <div property="schema:text" class="field__item"><p>One of the main plugins we were using with our 2.x Elasticsearch cluster was KOPF. This plugin was a web interface to the Elasticsearch API and it was an easy way of performing common tasks on our Elasticsearch cluster.</p> <p>When we upgraded our Elasticsearch cluster to version 5.x, we could not continue using this plugin because it was not longer supported. The good thing was that the author of KOPF, Leonardo Menezes, had a new project called "CEREBRO" to offer an alternative to KOPF when running Elasticsearch 5.x.</p></div> </div> Sun, 25 Jun 2017 13:53:42 +0000 rafaelma 99 at https://e-mc2.net Using Zabbix with PostgreSQL as the database backend https://e-mc2.net/using-zabbix-postgresql-database-backend <span property="schema:name">Using Zabbix with PostgreSQL as the database backend</span> <span rel="schema:author"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">rafaelma</span></span> <span property="schema:dateCreated" content="2017-06-19T19:22:34+00:00">Mon, 19/06/2017 - 21:22</span> <div class="field field--name-body field--type-text-with-summary field--label-above"> <div class="field__label">Body</div> <div property="schema:text" class="field__item"><p>We started using Zabbix to monitor the IT infrastructure at The University of Oslo in 2014. During all this time we have been running all our Zabbix servers on VMware virtual servers with an acceptable level of performance. This situation changed some months ago when the VMvare+Storage we were using did not have more available resources for us to grow and it was slowing down the future development of our monitoring system.</p></div> </div> Mon, 19 Jun 2017 19:22:34 +0000 rafaelma 97 at https://e-mc2.net ZABBIX automation @ UiO https://e-mc2.net/zabbix-automation-uio <span property="schema:name">ZABBIX automation @ UiO</span> <span rel="schema:author"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">rafaelma</span></span> <span property="schema:dateCreated" content="2017-05-02T12:59:14+00:00">Tue, 02/05/2017 - 14:59</span> <div class="field field--name-body field--type-text-with-summary field--label-above"> <div class="field__label">Body</div> <div property="schema:text" class="field__item"><p>A short presentation focused on how we automate the Zabbix configuration at the Universitity of Oslo</p></div> </div> Tue, 02 May 2017 12:59:14 +0000 rafaelma 114 at https://e-mc2.net Asegurando nuestros datos https://e-mc2.net/asegurando-nuestros-datos <span property="schema:name">Asegurando nuestros datos</span> <span rel="schema:author"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">rafaelma</span></span> <span property="schema:dateCreated" content="2011-01-31T23:00:00+00:00">Tue, 01/02/2011 - 00:00</span> <div class="field field--name-body field--type-text-with-summary field--label-above"> <div class="field__label">Body</div> <div property="schema:text" class="field__item"><p>A general presentation about different aspects and components we have to take care of when securing the data in a PostgreSQL database.</p></div> </div> Mon, 31 Jan 2011 23:00:00 +0000 rafaelma 56 at https://e-mc2.net