-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] Non-deterministic behavior of mixed cluster YAML tests #5257
Labels
bug
Something isn't working
Build
Build Tasks/Gradle Plugin, groovy scripts, build tools, Javadoc enforcement.
flaky-test
Random test failure that succeeds on second run
:test
Adding or fixing a test
Comments
andrross
added
bug
Something isn't working
:test
Adding or fixing a test
untriaged
distributed framework
labels
Nov 15, 2022
andrross
added
Build
Build Tasks/Gradle Plugin, groovy scripts, build tools, Javadoc enforcement.
and removed
distributed framework
labels
Feb 21, 2024
This was referenced Mar 12, 2024
8 tasks
This was referenced Apr 30, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
bug
Something isn't working
Build
Build Tasks/Gradle Plugin, groovy scripts, build tools, Javadoc enforcement.
flaky-test
Random test failure that succeeds on second run
:test
Adding or fixing a test
OpenSearch provides a mechanism for testing the REST API using a YAML-based DSL. All the YAML tests are run against the code at build time as a part of the
:rest-api-spec:yamlRestTest
. They are also run as a part of the:qa:mixed-cluster:v2.4.0#mixedClusterTest
task via MixedClusterClientYamlTestSuiteIT. When introducing a new feature to the REST API, the proper way to define a new test is to put askip <version>
section in the YAML test definition, which tells the mixed cluster test that this test should not run against an older version which doesn't support the feature being tested. However, if one forgets to add this skip section, the tests may still pass in the mixed cluster tests if the queries happen to hit only the nodes running the new version. See #4852 for a case where this happened. It would be better for the misconfigured test definition to deterministically fail. The obvious approach that stands out to me is to run these YAML tests against an exclusively "old version" cluster to ensure they have the proper guards in place so that any misconfiguration is guaranteed to fail. Are there any other approaches to solve this problem?The text was updated successfully, but these errors were encountered: