Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update log4j rollover to configure time retention #16179

Merged
merged 9 commits into from
Jun 5, 2024
Prev Previous commit
Next Next commit
Made wording greatly smoother
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
  • Loading branch information
andsel and karenzone authored Jun 4, 2024
commit 9582ebd0b76c27fb1e36bd97896abb623e497085
36 changes: 23 additions & 13 deletions docs/static/logging.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -48,20 +48,30 @@ The logger is usually identified by a Java class name, such as
path as in `org.logstash.dissect`. For Ruby classes, like `LogStash::Outputs::Elasticsearch`,
the logger name is obtained by lowercasing the full class name and replacing double colons with a single dot.

It's strongly recommended to use the default log4j configuration that's shipped with {ls}, however the
following section describe in detail how the rolling strategy works, so you have more control on these settings.
NOTE: Consider using the default log4j configuration that is shipped with {ls}, as it is configured to work well for most deployments.
The next section describes how the rolling strategy works in case you need to make adjustments.

In `log4j2.properties` file are defined three appenders to write on log files: one for plain text, one with json format,
and another to split log lines on per pipeline basis when using the `pipeline.separate_logs` setting.
These appenders define triggering policies and rollover strategy. The triggering policy determines if a rollover
should be performed, while the strategy defines how the rollover should be done.
By default definition two triggering policies are used: time and size. The time policy creates one file per day
while the size policy forces the creation of a new file once the file size surpasses 100 MB.
[[rollover]]
===== Rollover settings

The default strategy also performs file rollovers to a maximum number of 30 files and also deletes files older than seven days.
The rollover limit imposes that a maximum of 30 files is created. When the 31-st has to be created a rollover happen the first
is removed - all the other are renamed - so that 2nd becomes 1st and 30th becomes 29th, creating the space for the new file.
Each file will have a date and if the files are older than 7 days (by default) they are removed during rollover.
The `log4j2.properties` file has three appenders for writing to log files:
one for plain text, one with json format, and one to split log lines on per pipeline basis when you set the `pipeline.separate_logs` value.

These appenders define:

* **triggering policies** that determine _if_ a rollover should be performed, and
* **rollover strategy** to defines _how_ the rollover should be done.

By default, two triggering policies are defined--time and size.

* The **time** policy creates one file per day.
* The **size** policy forces the creation of a new file after the file size surpasses 100 MB.

The default strategy also performs file rollovers based on a maximum number files.
When the limit of 30 files has been reached, the first (oldest) file is removed to create space for the new file.
Subsequent files are renumbered accordingly.

Each file has a date, and files older than 7 days (default) are removed during rollover.
Comment on lines +71 to +74
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andsel If I'm understanding we're saying that we're showing users how to override default settings, but in the example, we're showing default settings. Is that correct?

Can we offer any guidelines on appropriate settings other than recommending that people don't change them?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The intention is to describe what the default settings are and what they accomplish. The second example demonstrates how to switch the strategy to be size-based (the overall file sizes) instead of time-based (the default configuration that roll over after 7 days).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gotcha. Thank you.


[source,text]
----------------------------------
Expand Down Expand Up @@ -107,7 +117,7 @@ appender.rolling.strategy.action.condition.glob = pipeline_${ctx:pipeline.id}.*.
appender.rolling.strategy.action.condition.nested_condition.type = IfAccumulatedFileSize
appender.rolling.strategy.action.condition.nested_condition.exceeds = 5MB <1>
----------------------------------
<1> Delete files if total accumulated compressed file size is over 5MB.
<1> Deletes files if total accumulated compressed file size is over 5MB.

==== Logging APIs

Expand Down
Loading