Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Non-deterministic behavior of mixed cluster YAML tests #5257

Open
andrross opened this issue Nov 15, 2022 · 0 comments
Open

[BUG] Non-deterministic behavior of mixed cluster YAML tests #5257

andrross opened this issue Nov 15, 2022 · 0 comments
Labels
bug Something isn't working Build Build Tasks/Gradle Plugin, groovy scripts, build tools, Javadoc enforcement. flaky-test Random test failure that succeeds on second run :test Adding or fixing a test

Comments

@andrross
Copy link
Member

OpenSearch provides a mechanism for testing the REST API using a YAML-based DSL. All the YAML tests are run against the code at build time as a part of the :rest-api-spec:yamlRestTest. They are also run as a part of the :qa:mixed-cluster:v2.4.0#mixedClusterTest task via MixedClusterClientYamlTestSuiteIT. When introducing a new feature to the REST API, the proper way to define a new test is to put a skip <version> section in the YAML test definition, which tells the mixed cluster test that this test should not run against an older version which doesn't support the feature being tested. However, if one forgets to add this skip section, the tests may still pass in the mixed cluster tests if the queries happen to hit only the nodes running the new version. See #4852 for a case where this happened. It would be better for the misconfigured test definition to deterministically fail. The obvious approach that stands out to me is to run these YAML tests against an exclusively "old version" cluster to ensure they have the proper guards in place so that any misconfiguration is guaranteed to fail. Are there any other approaches to solve this problem?

@andrross andrross added bug Something isn't working :test Adding or fixing a test untriaged distributed framework labels Nov 15, 2022
@peternied peternied added the flaky-test Random test failure that succeeds on second run label Feb 7, 2024
@andrross andrross added Build Build Tasks/Gradle Plugin, groovy scripts, build tools, Javadoc enforcement. and removed distributed framework labels Feb 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working Build Build Tasks/Gradle Plugin, groovy scripts, build tools, Javadoc enforcement. flaky-test Random test failure that succeeds on second run :test Adding or fixing a test
Projects
None yet
Development

No branches or pull requests

3 participants