This article is based on CentOS7 environment using rpm package installation.ELK's default logging can grow significantly, with the exception of Elastic Search, and long runs can have catastrophic consequences (such as node downtime).This is the main problem we are facing today.The main strategy is to limit the total number of logs: time + size, rotate one log file per day, or rotate a new log file every time the log file size exceeds 256M, and keep up to 7 days of log files.

ElasticSearch

Default configuration problem

ElasticSearch, by default, rolls one file a day, starts clearing the excess when it reaches 2G, and keeps accumulating when a file has only a few dozen K s.

Solution

Solve this by modifying the log4j2.properties file.The file is in the / etc/elasticsesarch directory, and the default configuration has the following settings

...
appender.rolling.strategy.action.condition.nested_condition.type = IfAccumulatedFileSize
appender.rolling.strategy.action.condition.nested_condition.exceeds = 2GB 
...

This configuration saves 2GB of logs and deletes old log files only if the cumulative log size exceeds 2GB.Suggested changes to

...
appender.rolling.strategy.action.condition.nested_condition.type = IfLastModified
appender.rolling.strategy.action.condition.nested_condition.age = 7D 
...

Keep logs for the last 7 days only.

Logstash

Default configuration problem

Logstash keeps growing gc files and rolling log files and does not delete them.

Solution

Increase the configuration by modifying the log4j2.properties file (/etc/logstash directory):

...
appender.rolling.strategy.type = DefaultRolloverStrategy
appender.rolling.strategy.action.type = Delete
appender.rolling.strategy.action.basepath = ${sys:ls.logs}
appender.rolling.strategy.action.condition.type = IfFileName
appender.rolling.strategy.action.condition.glob = ${sys:ls.logs}/logstash-${sys:ls.log.format}
appender.rolling.strategy.action.condition.nested_condition.type = IfLastModified
appender.rolling.strategy.action.condition.nested_condition.age = 7D 
...

Kibana

Default configuration problem

The log is exported to the kibana.out file, which becomes larger and larger.

Solution

In kibana's configuration file, there are only a few options:

logging.dest:
Default: stdout Enables you specify a file where Kibana stores log output.

logging.quiet:
Default: false Set the value of this setting to true to suppress all logging output other than error messages.

logging.silent:
Default: false Set the value of this setting to true to suppress all logging output.

logging.verbose:
Default: false Set the value of this setting to true to log all events, including system usage information and all requests. Supported on Elastic Cloud Enterprise.

logging.timezone
Default: UTC Set to the canonical timezone id (e.g. US/Pacific) to log events using that timezone. A list of timezones can be referenced at https://en.wikipedia.org/wiki/List_of_tz_database_time_zones.

We can specify the output log file and log content, but we cannot configure the rotate of the log.At this point, we need to use logrotate, a tool that linux installs by default.
First, we'll specify to generate a pid file in the configuration file:

pid.file: "pid.log"

Then, modify/etc/logrotate.conf:

/var/log/kibana {
    missingok
    notifempty
    shareds
    daily
    rotate 7
    copytruncate
    /bin/kill -HUP $(cat /usr/share/kibana/pid.log 2>/dev/null) 2>/dev/null
    end
}