Default image could have decent logratate options on

[heikkis]

Wish from a guy with 30GB logfile from logstash for example :)

[spujadas]

Fair point, and open to suggestions: any recommendations as to what options would be decent?

[heikkis]

Maybe something like this:

5 files with 100MB limit per service totally 500MB per service and totally over kibana, logstash, elasticsearch it would be 1500MB.

[heikkis]

Hmm. How logstash handle log writing? Is there log4j etc. handler to handle logrotating? Few examples just point out normal daily logrorate (which should be then hourly in this case?).

[spujadas]

OK thanks for the input, will look into this and update the image.
For Logstash at the very least, the logrotate configuration for Logstash that is included in the DEB/RPM/etc. packages looks promising.

[gvenka008c]

@spujadas How do you manage the logs that is sent from the client servers to ELK servers? How can we retain only 3 days worth of data in ELK servers that gets displayed in Kibana?

[spujadas]

@gvenka008c Sorry, not a Logstash expert so haven't got a definite answer to that one.
Deleting old indices as described here seems like a good idea, but don't take my word for it: you may want to check in with the Logstash community over at https://discuss.elastic.co/c/elasticsearch for a more solid answer to your question.

[spujadas]

Ended up going with a daily rotation + compression + deletion after a week, using logrotate for all three services.
Tried to configure Elasticsearch's logging.yml but that was very fiddly and ultimately dissatisfying: Elasticsearch's use of log4j 1.x does allow for log rotation but it appears to be much less flexible than logrotate (i.e. can't seem do to what logrotate easily does). This should change if/when log4j is upgraded to 2.x (see elastic/elasticsearch#17697 for upgrade plans), in the meantime will play it safe and use logrotate (tested, appears to work as intended).
Anyway, now has a "sensible" default, which can be overriden as needed.

[heikkis]

Thanks. Sound very good!

[reallistic]

Instead of using logrotate on logstash you may consider removing the following from 30-output.conf

stdout { codec => rubydebug }

See here: https://discuss.elastic.co/t/logstash-stdout-large-size/24939/4

Also, this logrotate setup did not work for me. After some debugging it seems as if the cron daemon is not running within the container (see issue #60)

[spujadas]

@reallistic Yep, sounds like a good idea, but I need to do a few tests first (see #60, will have to wait a couple of weeks for that).