[CI] CrossClusterQueryWithFiltersIT testTimestampFilterFromQuery failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic-platform-support #8626 / windows-2019_checkpart3_platform-support-windows
• elasticsearch-periodic-platform-support #8614 / sles-12_platform-support-unix

Reproduction Line:

gradlew ":x-pack:plugin:esql:internalClusterTest" --tests "org.elasticsearch.xpack.esql.action.CrossClusterQueryWithFiltersIT.testTimestampFilterFromQuery" -Dtests.seed=5F78E0A7BA976C68 -Dtests.locale=cgg-UG -Dtests.timezone=Atlantic/St_Helena -Druntime.java=24

Applicable branches:
8.19

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: 
Expected: <0>
     but: was <1>

Issue Reasons:

• [8.19] 2 failures in test testTimestampFilterFromQuery (1.4% fail rate in 138 executions)
• [8.19] 2 failures in pipeline elasticsearch-periodic-platform-support (50.0% fail rate in 4 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 2 failures in test testTimestampFilterFromQuery (1.1% fail rate in 180 executions)

Build Scans:

• elasticsearch-intake #22302 / part3
• elasticsearch-pull-request #68642 / part-3

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[fang-xing-esql]

@smalyshev Could you help take a look at this test? Thank you!

[smalyshev]

Something weird is happening here - the error basically says one shard is skipped where there shouldn't be any skipped shards, and I am not sure why exactly this happens. Seems to be very rare occurrence and is not reproducible locally. Generally having a skipped shard is no big deal unless it indicates some deeper problem - what worries me I still don't understand where it comes from. Let's see if I get more failures to see if there's any pattern.

[smalyshev]

One failure on 4th - looks like the fail is real, though very rare.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 2 failures in test testTimestampFilterFromQuery (0.3% fail rate in 705 executions)

Build Scans:

• elasticsearch-periodic-platform-support #8106 / ubuntu-2004-aarch64_checkpart3_platform-support-arm
• elasticsearch-pull-request #70417 / part-3