[CI] MixedClusterClientYamlTestSuiteIT class failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic #5895 / 8.13.4_bwc
• elasticsearch-intake #15984 / 8.17.1_bwc-snapshots
• elasticsearch-periodic #5890 / 8.14.3_bwc
• elasticsearch-periodic #5890 / 8.13.4_bwc

Reproduction Line:

./gradlew ":qa:mixed-cluster:v8.13.4#mixedClusterTest" -Dtests.class="org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT" -Dtests.method="test {p0=search.vectors/160_knn_query_missing_params/kNN search used in nested field - missing num_candidates}" -Dtests.seed=BF9B1B70A38FCCCC -Dtests.bwc=true -Dtests.locale=mi-Latn-NZ -Dtests.timezone=Australia/North -Druntime.java=23

Applicable branches:
8.x

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: Failure at [search.vectors/160_knn_query_missing_params:171]: expected [2xx] status code but api [search] returned [400 Bad Request] [{"error":{"root_cause":[{"type":"x_content_parse_exception","reason":"[1:129] [knn] unknown field [k]","stack_trace":"org.elasticsearch.ElasticsearchException$1: [1:129] [knn] unknown field [k]\n\tat org.elasticsearch.server@8.13.4/org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:701)\n\tat org.elasticsearch.server@8.13.4/org.elasticsearch.ElasticsearchException.generateFailureXContent(ElasticsearchException.java:629)\n\tat org.elasticsearch.server@8.13.4/org.elasticsearch.rest.RestResponse.build(RestResponse.java:184)\n\tat org.elasticsearch.server@8.13.4/org.elasticsearch.rest.RestResponse.<init>(RestResponse.java:143)\n\tat org.elasticsearch.server@8.13.4/org.elasticsearch.rest.RestResponse.<init>(RestResponse.java:114)\n\tat org.elasticsearch.server@8.13.4/org.elasticsearch.rest.RestContro
[truncated]

Issue Reasons:

• [8.x] 2 consecutive failures in step 8.13.4_bwc
• [8.x] 4 failures in class org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT (1.2% fail rate in 335 executions)
• [8.x] 2 failures in step 8.13.4_bwc (50.0% fail rate in 4 executions)
• [8.x] 2 failures in pipeline elasticsearch-periodic (50.0% fail rate in 4 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch 8.x

Mute Reasons:

• [8.x] 4 consecutive failures in step 8.13.4_bwc
• [8.x] 4 consecutive failures in step 8.12.2_bwc
• [8.x] 3 consecutive failures in step 8.14.3_bwc
• [8.x] 12 failures in class org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT (3.4% fail rate in 356 executions)
• [8.x] 4 failures in step 8.13.4_bwc (100.0% fail rate in 4 executions)
• [8.x] 4 failures in step 8.12.2_bwc (100.0% fail rate in 4 executions)
• [8.x] 3 failures in step 8.14.3_bwc (100.0% fail rate in 3 executions)
• [8.x] 3 failures in pipeline elasticsearch-periodic (75.0% fail rate in 4 executions)
• [8.x] 2 failures in pipeline elasticsearch-pull-request (4.4% fail rate in 45 executions)

Build Scans:

• elasticsearch-periodic #5760 / 8.13.4_bwc
• elasticsearch-periodic #5760 / 8.12.2_bwc
• elasticsearch-pull-request #49432 / 8.17.1_bwc-snapshots
• elasticsearch-periodic #5755 / 8.14.3_bwc
• elasticsearch-periodic #5755 / 8.12.2_bwc
• elasticsearch-periodic #5755 / 8.13.4_bwc
• elasticsearch-pull-request #49316 / 8.14.3_bwc
• elasticsearch-pull-request #49316 / 8.13.4_bwc
• elasticsearch-pull-request #49316 / 8.12.2_bwc
• elasticsearch-periodic #5750 / 8.13.4_bwc

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)

[cbuescher]

We need to unmute this suite as it looks like one large spike in errors in a specific test might have muted the whole suite. We need to open more specific failure issues and mute selectively in order to not loose coverage of a while test suite.

[cbuescher]

Pinging @elastic/es-delivery for awareness, we were discussing if muting a whole rest test suite like in this case is the right thing to do. I spot-checked build scans on this and it seems mostly two knn-related yaml test suites were responsible for the peak in failures (see e.g. https://gradle-enterprise.elastic.co/s/i3f26xj4f72la/tests/overview?outcome=FAILED, https://gradle-enterprise.elastic.co/s/2oec47ob7mlke/tests/overview?outcome=FAILED). We would like to flag this muting as a false positive for future tuning of the bot automation but I'm not entirely sure who's in charge of this and where to direct this ask. Could you give me any directions?

[elasticsearchmachine]

This has been muted on branch 8.x

Mute Reasons:

• [8.x] 2 consecutive failures in step 8.13.4_bwc
• [8.x] 4 failures in class org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT (1.2% fail rate in 335 executions)
• [8.x] 2 failures in step 8.13.4_bwc (50.0% fail rate in 4 executions)
• [8.x] 2 failures in pipeline elasticsearch-periodic (50.0% fail rate in 4 executions)

Build Scans:

• elasticsearch-periodic #5895 / 8.13.4_bwc
• elasticsearch-intake #15984 / 8.17.1_bwc-snapshots
• elasticsearch-periodic #5890 / 8.14.3_bwc
• elasticsearch-periodic #5890 / 8.13.4_bwc