org.elasticsearch.upgrades.RecoveryIT fails during upgrade tests

[przemekwitek]

Build scan:
[6.8.14] https://gradle-enterprise.elastic.co/s/zrc4fnpfqnwdu

Repro line:

REPRODUCE WITH: ./gradlew ':qa:rolling-upgrade:v6.8.14#upgradedClusterTest' \
  -Dtests.class="org.elasticsearch.upgrades.RecoveryIT" \
  -Dtests.method="testRecoveryWithConcurrentIndexing" \
  -Dtests.seed=9A9A65701543F0D \
  -Dtests.security.manager=true \
  -Dtests.bwc=true \
  -Dtests.locale=es-VE \
  -Dtests.timezone=Arctic/Longyearbyen \
  -Druntime.java=8

REPRODUCE WITH: ./gradlew ':qa:rolling-upgrade:v6.8.14#upgradedClusterTest' \
  -Dtests.class="org.elasticsearch.upgrades.RecoveryIT" \
  -Dtests.method="testRecoveryWithConcurrentIndexing" \
  -Dtests.seed=9A9A65701543F0D \
  -Dtests.security.manager=true \
  -Dtests.bwc=true \
  -Dtests.locale=es-VE \
  -Dtests.timezone=Arctic/Longyearbyen \
  -Druntime.java=8

Reproduces locally?:
Don't know yet.

Applicable branches:
7.x, 7.10

Failure history:

Failure excerpt:

12:46:20 org.elasticsearch.upgrades.RecoveryIT > testRecoveryWithConcurrentIndexing FAILED
12:46:20     org.elasticsearch.client.WarningFailureException: method [PUT], host [http://127.0.0.1:38975], URI [/recovery_with_concurrent_indexing/_settings], status line [HTTP/1.1 200 OK]
12:46:20     Warnings: ['Y' year-of-era should be replaced with 'y'. Use 'Y' for week-based-year.; 'Z' time zone offset/id fails when parsing 'Z' for Zulu timezone. Consider using 'X'. Use new java.time date format specifiers.]
12:46:20     {"acknowledged":true}
12:46:20         at __randomizedtesting.SeedInfo.seed([9A9A65701543F0D:89F8620483F0E1C7]:0)
12:46:20         at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:322)
12:46:20         at org.elasticsearch.client.RestClient.performRequest(RestClient.java:296)
12:46:20         at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270)
12:46:20         at org.elasticsearch.test.rest.ESRestTestCase.updateIndexSettings(ESRestTestCase.java:1208)
12:46:20         at org.elasticsearch.test.rest.ESRestTestCase.updateIndexSettings(ESRestTestCase.java:1202)
12:46:20         at org.elasticsearch.upgrades.RecoveryIT.testRecoveryWithConcurrentIndexing(RecoveryIT.java:174)

[przemekwitek]

There are more failures happening during upgrade tests:

Build scan: [6.8.14] https://gradle-enterprise.elastic.co/s/rb5rgbvydjkbo

REPRODUCE WITH: ./gradlew ':qa:mixed-cluster:v6.8.14#mixedClusterTest' \
  -Dtests.class="org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT" \
  -Dtests.method="test {p0=indices.stats/14_groups/Groups - search metric}" \
  -Dtests.seed=DD89C22B35BB4D0B \
  -Dtests.security.manager=true \
  -Dtests.bwc=true \
  -Dtests.locale=es-GT \
  -Dtests.timezone=Africa/Dar_es_Salaam \
  -Druntime.java=8

Build scan: [6.8.14] https://gradle-enterprise.elastic.co/s/kskphr4j4azxu

REPRODUCE WITH: ./gradlew ':qa:rolling-upgrade:v6.8.14#oneThirdUpgradedTest' \
  -Dtests.class="org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT" \
  -Dtests.method="test {p0=mixed_cluster/10_basic/Verify that we can still find things with the template}" \
  -Dtests.seed=1A03106CC59D4E56 \
  -Dtests.security.manager=true \
  -Dtests.bwc=true \
  -Dtests.locale=es-ES \
  -Dtests.timezone=Europe/Budapest \
  -Druntime.java=8 \
  -Dtests.rest.suite=mixed_cluster

Build scan: [6.8.14] https://gradle-enterprise.elastic.co/s/ltrxtandkmslu

REPRODUCE WITH: ./gradlew ':qa:rolling-upgrade:v6.8.14#oneThirdUpgradedTest' --tests "org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT.test {p0=mixed_cluster/10_basic/Verify custom cluster metadata still exists during upgrade}" \
  -Dtests.seed=D30DD654A166342 \
  -Dtests.security.manager=true \
  -Dtests.locale=fr \
  -Dtests.timezone=Asia/Yakutsk \
  -Druntime.java=8 \
  -Dtests.rest.suite=mixed_cluster

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[mark-vieira]

Those build scan links are for the wrong builds. Here's test history for these failures which include the actual failing builds:
https://gradle-enterprise.elastic.co/scans/tests?search.relativeStartTime=P7D&search.timeZoneId=America/Los_Angeles&tests.container=org.elasticsearch.upgrades.RecoveryIT&tests.sortField=FAILED&tests.test=testRelocationWithConcurrentIndexing&tests.unstableOnly=true

[rjernst]

@pgomulka I'm not sure why this would start failing now, but can you take a look? I can't find anywhere in RecoveryIT that the mappings are set explicitly, so perhaps this is due to a default somewhere in 6.x that gets propagated to the upgrade?

[mark-vieira]

First failiure I see for this particular test was about two weeks ago: https://gradle-enterprise.elastic.co/s/ixeausynaw6k6/tests/:qa:rolling-upgrade:v6.8.14%23upgradedClusterTest/org.elasticsearch.upgrades.RecoveryIT/testRecoveryWithConcurrentIndexing#1

That said, here's one from December, but for a different test. Perhaps this scenario is just really uncommon: https://gradle-enterprise.elastic.co/s/akacimfevowiu/tests/:qa:rolling-upgrade:v6.8.14%23upgradedClusterTest/org.elasticsearch.upgrades.RecoveryIT/testCloseIndexDuringRollingUpgrade#1

[pgomulka]

I took a look and also could not find anything that would create a custom joda pattern. We would not emit deprecations if default named pattern was used (like strict_date_optional_time or similar)

But what looks suspicious to me is that it always fail on

  case UPGRADED:
                updateIndexSettings(index, Settings.builder().put(INDEX_ROUTING_ALLOCATION_ENABLE_SETTING.getKey(), (String)null));

I would expect this fail earlier, in old cluster or mixed cluster step.

This test do not set a date field anywhere in the mappings nor in documents. I wonder if this is possible that when calling index settings update api, we trigger mapper service that would work on all indices and could trigger a deprecation warning?
JodaCompatibilityIT test is using joda_time index with date field with pattern YYYY-MM-dd'T'HH:mm:ssZZ.

[pgomulka]

I discussed this with @ywelsch and we suspect that ThreadContext is not properly stashed before we send out a response. I will try to investigate and come up with a fix.

[cbuescher]

This from today on 7.12 looks related to me https://gradle-enterprise.elastic.co/s/cwerwilfhxsta

[cbuescher]

Also this https://gradle-enterprise.elastic.co/s/z3mk5k2dz346o