[TEST] RecoveryIT.testHistoryUUIDIsGenerated occasionally fails on 6.x

[dimitris-athanasiou]

Build: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+multijob-windows-compatibility/1119/console

Reproduce with:

gradlew :qa:rolling-upgrade:v5.6.10-SNAPSHOT#upgradedClusterTestRunner -Dtests.seed=CC9DDAFEF7C7362F -Dtests.class=org.elasticsearch.upgrades.RecoveryIT -Dtests.method="testHistoryUUIDIsGenerated" -Dtests.security.manager=true -Dtests.locale=en-IN -Dtests.timezone=Australia/Eucla

Failure:

RecoveryIT.testHistoryUUIDIsGenerated <<< FAILURES!
07:49:28    > Throwable #1: java.lang.AssertionError: different history uuid found for shard on BPdfH1zoREy4bR0ga9iksg
07:49:28    > Expected: "D9AAXTrJT7WOJsynoFB0LQ"
07:49:28    >      but: was "JsLgmYhvTEyjm_x9NWESGQ"
07:49:28    > 	at __randomizedtesting.SeedInfo.seed([CC9DDAFEF7C7362F:E43FE5891A4AA329]:0)
07:49:28    > 	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
07:49:28    > 	at org.elasticsearch.upgrades.RecoveryIT.testHistoryUUIDIsGenerated(RecoveryIT.java:79)
07:49:28    > 	at java.lang.Thread.run(Thread.java:748)

I could not reproduce locally.

[elasticmachine]

Pinging @elastic/es-distributed

[dimitris-athanasiou]

I've muted this test on 6.x.

[bleskes]

@dnhatn ping..

[dnhatn]

@bleskes Looking...

[dnhatn]

I opened #31542

[dnhatn]

I've enabled debug log and unmuted this test. I will surely keep an eye on it.

[dnhatn]

Another instance https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+default-distro-bwc-tests/141/console. CI log: history_uuid.txt.

Since this test does not fail too often, I won't mute it. I am looking at this.

[dnhatn]

I looked at this. Sadly, we still don't have enough information. I increased the logging level and wait for another failure.