Update log4j rollover to configure time retention

[andsel]

Docs PREVIEW: https://logstash_bk_16179.docs-preview.app.elstc.co/guide/en/logstash/master/logging.html#log4j2

Release notes

Update logging configuration to remove rolled files older than seven days.

What does this PR do?

Updates the plain, json and pipeline appenders in default config/log4j2.properties to define a delete rule executed during the rollover strategy, which deletes compressed log archives older than 7 days.
Updates the documentation that the describe the logging configuration to explain how the rollover file works, how to configure the strategy in particular how also update to have space limitation condition on the rollover.

Why is it important/What is the impact to the user?

With this commit the user doesn't have anymore problems of disk space exhaustion due to aged log files.

Checklist

• My code follows the style guidelines of this project
• [ ] I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files (and/or docker env variables)
• [ ] I have added tests that prove my fix is effective or that my feature works

Author's Checklist

• test locally.

How to test this PR locally

Change the rollover strategy to use a space limitation (for example 5MB) and use a pipeline that generates log lines, then check that during Logstash execution the files are properly deleted.

Log strategy definition

In log4j2.properties replace

appender.rolling.strategy.action.condition.nested_condition.type = IfLastModified
appender.rolling.strategy.action.condition.nested_condition.age = 7D

with:

appender.routing.pipeline.strategy.action.condition.nested_condition.type = IfAccumulatedFileSize
appender.routing.pipeline.strategy.action.condition.nested_condition.exceeds = 5MB

Pipeline to generate logs

Launch Logstash with following pipeline:

input {
  generator {
    message => '{"name": "John", "surname": "Doe"}'
    codec => json
  }
}

filter {
  ruby {
    code => 'logger.info "This is ia very long line logger by the ruby filter, there is one log line for each event and the only  goal is to generate log lines in the log file to test the log4j2 rollover startegy. In particular the size limit."'
  }	
}

output {
  sink {}
}

Then check the content of logs folder.

Related issues

• Closes [Docs] Document log4j2.properties default rolling strategy #15930
• Closes Rotate & Delete Logstash logs by default #7482

Use cases

As a user I don't want the archive log files clutter the local filesystem.

[jsvd]

LGTM

[karenzone]

@andsel, thanks for your work on this. Nice explanations.

I left some suggestions inline to break up the text a bit to make it easier to skim and parse. Please let me know what you think.

[andsel]

@karenzone thank's for all your suggestion, accepted in batch 🥇

[karenzone]

This is looking great. I left some questions about clarifying exactly what we're demonstrating to users here. When I'm clear on what we're doing, I can make some minor tweaks, and we should be able to wrap this up quickly.

[karenzone]

@andsel If I'm understanding we're saying that we're showing users how to override default settings, but in the example, we're showing default settings. Is that correct?

Can we offer any guidelines on appropriate settings other than recommending that people don't change them?

[andsel]

The intention is to describe what the default settings are and what they accomplish. The second example demonstrates how to switch the strategy to be size-based (the overall file sizes) instead of time-based (the default configuration that roll over after 7 days).

[karenzone]

Gotcha. Thank you.

[karenzone]

Please consider implementing the suggestion in https://github.com/elastic/logstash/pull/16179/files#r1626121129 to emphasize "maximum number of files." Otherwise, LGTM.

Thanks for your work on this.

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: c1e5c8f

History

• 💚 Build #1185 succeeded 9582ebd
• 💚 Build #1181 succeeded f373db9
• 💚 Build #1180 succeeded fff1776
• 💚 Build #1179 succeeded 1c03b65
• 💚 Build #1178 succeeded 308fcdb
• 💚 Build #1133 succeeded 0a6d4d2

cc @andsel