This article is a practical story about how we encountered the problem of separating the logs stored in Elasticsearch, because of which we had to change the approach to backups and index management.
It all started shortly after the production environment was raised. We had a “combat” Kubernetes cluster, all the logs from which fluentd collected and sent them directly to indexes logstash-yyy.mm.dd...
However, a request appeared to store some application logs for search for up to 90 days. At that time, we could not afford this: the storage of current indices for such a period would exceed all reasonable measures. Therefore, it was decided to create a separate index pattern for such applications and configure a separate retention for it.Log Separation
To accomplish such a task and to separate the “necessary” logs from the “unnecessary” ones, we used the matchfluentd parameter and created another section in output.confto search for matches on the necessary services in the namespace productionaccording to the tagging specified for the input plug-in (and also changing @idand logstash_prefixso send recording to different places).Fragments of the resulting output.conf:<match kubernetes.var.log.containers.**_production_**>
@id elasticsearch1
@type elasticsearch
logstash_format true
logstash_prefix log-prod
...
</match>
<match kubernetes.**>
@id elasticsearch
@type elasticsearch
logstash_format true
logstash_prefix logstash
...
</match>
Bottom line - we have two types of indexes ( log-prod-yyyy.mm.ddand logstash-yyyy.mm.dd):
Index Cleaning
At that time , curator was already configured in the cluster , which cleared indexes older than 14 days. It was described as a CronJob object to deploy directly to a Kubernetes cluster.Illustration in action_file.yml:-
actions:
1:
action: delete_indices
description: >-
Delete indices older than 14 days (based on index name), for logstash-
prefixed indices. Ignore the error if the filter does not result in an
actionable list of indices (ignore_empty_list) and exit cleanly.
options:
ignore_empty_list: True
timeout_override:
continue_if_exception: False
disable_action: False
allow_ilm_indices: True
filters:
- filtertype: pattern
kind: prefix
value: logstash
- filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 14
However, we decided that the option with a curator is redundant (it runs as a separate software, requires separate configuration and launch by crown) - after all, you can redo the cleaning using the index lifecycle policy .Elasticsearch since version 6.6 (released in January 2019) now has the ability to attach a policy to the index template that will track the retention time for the index. Policies can be used not only to control cleanup, but also other tasks that will simplify interaction with indexes (for example, aligning indexes by size rather than by day).To do this, it was only necessary to create two policies of this kind:PUT _ilm/policy/prod_retention
{
"policy" : {
"phases" : {
"delete" : {
"min_age" : "90d",
"actions" : {
"delete" : { }
}
}
}
}
}
PUT _ilm/policy/default_retention
{
"policy" : {
"phases" : {
"delete" : {
"min_age" : "14d",
"actions" : {
"delete" : { }
}
}
}
}
}
... and attach them to the required index templates:PUT _template/log_prod_template
{
"index_patterns": ["log-prod-*"],
"settings": {
"index.lifecycle.name": "prod_retention",
}
}
PUT _template/default_template
{
"index_patterns": ["logstash-*"],
"settings": {
"index.lifecycle.name": "default_retention",
}
}
Now Elasticsearch cluster will independently manage data storage. This means that all indexes that fall under the template in the index_patternsabove will be cleared after the number of days specified in the policy.But here we are faced with another problem ...Reindexing within a cluster and from remote clusters
The policies, patterns, and changes that we created in fluentd that we created will only have effect on newly created indexes . To put in order what we already have, you will have to start the reindexing process by contacting the Reindex API .It is required to select the “necessary” logs from already closed indexes and reindex them themselves so that policies are applied.Two simple queries are enough for this:POST _reindex
{
"source": {
"index": "logstash-2019.10.24",
"query": {
"match": {
"kubernetes.namespace_name": "production"
}
}
},
"dest": {
"index": "log-prod-2019.10.24"
}
}
POST _reindex
{
"source": {
"index": "logstash-2019.10.24"
"query": {
"bool": {
"must_not": {
"match": {
"kubernetes.namespace_name": "production"
}
}
}
}
},
"dest": {
"index": "logstash-2019.10.24-ri"
}
}
Now you can delete old indexes!However, there are other difficulties. As already known, curator was previously configured, which cleaned the cluster with a retention of 14 days. Thus, for services that are relevant to us, it will be necessary to restore data from what was once deleted. Backups will help here.In the simplest case, which applies to ours, backups are done by calling elasticdumpfor all indexes at once:/usr/bin/elasticdump --all $OPTIONS --input=http://localhost:9200 --output=$
Expand everything that was from the start of production was not possible, because there wasn’t so much space in the cluster. In addition, access to the logs from the backup was required now.The solution is to deploy a temporary cluster into which the backup is restored, from where we get the logs we need (in a manner similar to that already described). In parallel, we began to think about how to make removing backups a more convenient way - see below for more details.So, the following steps (on a temporary cluster):- Install another Elasticsearch on a separate server with a sufficiently large disk.
- Expand the backup from our dump file:
/usr/bin/elasticdump --bulk --input=dump.json --output=http://127.0.0.1:9200/
- Please note that indexes in the cluster will not become green, as we transfer the dump to a 1-node configuration. But worry about this is not worth it: the main thing is to pull out only primal-shards.
- Do not forget that to allow reindexing from remote clusters, you need to add them to whitelist in
elasticsearch.yaml:
reindex.remote.whitelist: 1.2.3.4:9200
- Next, we make a request for reindexing from a remote cluster in the current production cluster:
POST _reindex
{
"source": {
"remote": {
"host": "http://1.2.3.4:9200"
},
"size": 10000,
"index": "logstash-2019.10.24",
"query": {
"match": {
"kubernetes.namespace_name": "production"
}
}
},
"dest": {
"index": "log-prod-2019.10.24"
}
}
By this request, we obtain from the indices in the remote cluster documents that will be transmitted to the production cluster over the network. On the side of the temporary cluster, all documents that are not suitable for the request will be filtered.Parameters sizeand sliceare used to speed up the reindexing process:size - to increase the number of documents for indexation transferred in the package;slice- to divide this task into 6 subtasks, which will be engaged in reindexing in parallel. (The parameter does not work in reindexing from remote clusters).
At the host Elasticsearch, we tuned index templates for maximum performance.When all the necessary logs are in one place completely divided as required, the temporary cluster can be deleted:
Backups Elasticsearch
It's time to get back to backups: running elasticdumpon all indexes is far from the best solution. At least for the reason that recovery can take indecent time, and there are times when every hour is important.Such a task can be "given" to Elasticsearch itself - Snapshot Repository based on S3. And here are the main reasons why we ourselves chose this approach:- The convenience of creating and restoring from such backups, since the native syntax of queries to Elasticsearch is used.
- Snapshots are rolled incrementally, i.e. adding new data to existing ones.
- The ability to restore any index from a snapshot and even restore the global state of the cluster.
- S3 seems to be a more reliable place to store backups than just a file (including for the reason that we usually use S3 in HA modes).
- S3 portability: we are not tied to specific providers and can deploy S3 in a new location on our own.
Setting up backup in S3 requires additional installation of the plugin in Elasticsearch so that Snapshot Repository can communicate with S3. The whole configuration comes down to the following steps:- Install the plugin for S3 on all nodes:
bin/elasticsearch-plugin install repository-s3
... and at the same time add S3 to whitelist in elasticsearch.yml:
Repositories.url.allowed_urls:
- "https://example.com/*"
- Add keys
SECRETand ACCESSin Elasticsearch keystore. By default, the user is used to connect to S3 default:
bin/elasticsearch-keystore add s3.client.default.access_key
bin/elasticsearch-keystore add s3.client.default.secret_key
... and after that we restart the Elasticsearch service on all nodes one by one. - Create a repository for snapshots:
PUT /_snapshot/es_s3_repository
{
"type": "s3",
"settings": {
"bucket": "es-snapshot",
"region": "us-east-1",
"endpoint": "https://example.com"
}
}
In this case, to create a snapshot, there are enough examples from the documentation where the name of the snapshot will be in the format:, snapshot-2018.05.11- i.e.:
PUT /_snapshot/my_backup/%3Csnapshot-%7Bnow%2Fd%7D%3E
- It remains only to test the restoration of the index:
POST /_snapshot/es_s3_repository/snapshot-2019.12.30/_restore
{
"indices": "logstash-2019.12.29",
"rename_pattern": "logstash-(.+)",
"rename_replacement": "restored_index_$1"
}
So we restore the index “next to”, simply renaming it. However, you can restore it to the same index - only for this the index in the cluster must be closed and have the same number of shards as the index in the snapshot.
The snapshot removal status can be checked by name via the API, calling up the snapshot information and looking in the field state:GET /_snapshot/es_s3_repository/snapshot_2019.12.30
The recovery status can be checked from the cluster itself: at the beginning of recovery, the cluster will go into status red, because primal shards of your indices will be restored. As soon as the process is completed, the indexes and cluster will go into status yellow- until the specified number of replicas is created.You can also track the status with the help wait_for_completion=trueindicated directly in the query line.Result - we get an exact copy of the index from the snapshot made:green open restored_index_2019.12.29 eJ4wXqg9RAebo1Su7PftIg 1 1 1836257 0 1.9gb 1000.1mb
green open logstash-2019.12.29 vE8Fyb40QiedcW0k7vNQZQ 1 1 1836257 0 1.9gb 1000.1mb
Summary and disadvantages
The optimal solutions for us in the Elasticsearch cluster were:- Configuring the output plugin in fluentd, with which you can share logs directly at the exit from the cluster.
- Index lifecycle policy, which allows you not to worry about problems with the space occupied by indexes.
- snapshot repository ( ), S3, .
However, it will be useful to caution that changes in policies and templates due to changing needs will subsequently lead to the need to reindex all indexes in the cluster. And with backups, the picture is even further from the ideal ...Despite the fact that we “handed over” the backups to Elasticsearch itself, launching the snapshot removal and monitoring its implementation still remains our task: accessing the API, tracking statuses (through wait_for_completion) and by status we We will produce it ourselves using scripts. For all its convenience, the backup repository has another problem: too few plugins. For example, there is no way to work through WebDAV if instead of S3 you need something completely simple, but just as mobile.In general, this isolation of the system does not fit well with the use of a centralized approach to backups and we have not yet found (among Open Source tools) a universal tool that would allow this.PS
Read also in our blog: