Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add troubleshooting section for the date filter #16549

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 57 additions & 0 deletions docs/static/troubleshoot/ts-date.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
[[ts-date]]
==== Date Filter issues and solutions

[discrete]
[[ts-date-locale]]
===== Date Parse Failures when using certain Locales

*Symptoms*

The {ls} Date Filter is unable to parse certain dates after upgrading to {ls} `8.15+`:

A date filter configured to parse dates as follows:

```
date { locale => "en_gb" match => [ "[date]" , "yyyy MMM dd HH:mm:ss" ] target => "[parsed_date]"
```

when parsing the date `2024 Sep 30 11:11:11` would, prior to `8.15.0`, have created the following entry in the event:

```
"parsed_date" => 2024-09-30T11:11:11.000Z,
```

After `8.15.0`, this would have failed to parse with the following added instead:

```
"tags" => [
[0] "_dateparsefailure"
]
```

as it now expects the date to be `2024 Sept 30 11:11:11` for that specific locale.


*Background*

Between JDK17 and JDK21, a change was made in the locale provider provided by the JVM internal.
JDK17 and JDK21 uses the https://cldr.unicode.org/[CLDR] locale provider by default, and between the release of JDK17 and JDK21, the version
of CLDR used by the default changed from version 37 to version 42.

To accommodate CLDR 38 in the JDK, https://github.com/openjdk/jdk/pull/1279/files#diff-97210acd6f77c4f4979c43445d60ba1c369f058230e41177dceca697800b1fa2R116[openjdk/jdk#1279] changed the abbreviated form of September from `Sep` to `Sept` in certain locales, including `en_GB` and `en_AU`.
This caused parse failures when there is a mismatch in the abbreviated form of date between Logstash and where the date was created.

For example, if the systems running Logstash are running with either an `en_AU` or `en_GB locale`, they will
expect September dates to use the `Sept` abbreviated form when the format states `MMM`.
If the systems generating these dates are creating dates with a Sep abbreviation, parsing will fail.

{ls} `8.15.0` upgraded the bundled JDK version to JDK21, which included the updated version of the `CLDR`, changing the
date parsing behavior for certain locales.

** Mitigations**

* Change the locale in the date filter to a locale that still uses `Sep` as the abbreviated form for September, such as `en_US`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Q: Can it be changed by using the ENV vars like https://www.baeldung.com/linux/locale-environment-variables ?

* Switch out the locale provider from CLDR to compat by setting `-Djava.locale.providers=COMPAT,SPI` in `jvm.options`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This option needs to be protected with a "Only for JDK < 23"

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Logstash only supports 11, 17 and 21.

Note that JDK23 drops support for the `compat` locale provider. A solutions is to add mutate filters to substitute `Sep` with `Sept` for all fields where the date is being parsed by the date filter.
Make sure these filters are placed *before* the date filter in the pipeline configuration.

1 change: 1 addition & 0 deletions docs/static/troubleshoot/ts-plugins.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,4 @@

include::ts-kafka.asciidoc[]
include::ts-azure.asciidoc[]
include::ts-date.asciidoc[]
Loading