Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution] Include only @serverlessQA Cypress tests to the 2nd quality gate #182600

Conversation

banderror
Copy link
Contributor

@banderror banderror commented May 3, 2024

Summary

Fixes a bug introduced in #181371 that included all Cypress tests tagged as @serverless to the 2nd quality gate (the QA stage in the release pipeline).

Current behavior: all tests tagged as @serverless run in the 2nd quality gate as part of the Kibana Serverless release process. Many of them are not ready for this and are not supposed to run there.

Correct behavior: include only those that were explicitly tagged as @serverlessQA.

Context

Tests that run in the 2nd quality gate can be found in https://buildkite.com/elastic?filter=security%20solution

For example, in this build we have a script:

$ yarn cypress:qa:serverless --spec './cypress/e2e/detection_response/rule_management/!(prebuilt_rules)/**/*.cy.ts'
--
  | $ TZ=UTC NODE_OPTIONS=--openssl-legacy-provider node ../../plugins/security_solution/scripts/start_cypress_parallel_serverless --config-file ../../test/security_solution_cypress/cypress/cypress_ci_serverless_qa.config.ts --spec './cypress/e2e/detection_response/rule_management/!(prebuilt_rules)/**/*.cy.ts'

that runs yarn cypress:qa:serverless which in turn uses the x-pack/test/security_solution_cypress/cypress/cypress_ci_serverless_qa.config.ts config which used incorrect tags to select Cypress tests.

@banderror banderror added bug Fixes for quality problems that affect the customer experience test impact:critical This issue should be addressed immediately due to a critical level of impact on the product. Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v8.14.0 v8.15.0 labels May 3, 2024
@banderror banderror self-assigned this May 3, 2024
@banderror banderror requested a review from a team as a code owner May 3, 2024 16:40
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@banderror banderror added the release_note:skip Skip the PR/issue when compiling release notes label May 3, 2024
@kibana-ci
Copy link
Collaborator

💚 Build Succeeded

Metrics [docs]

✅ unchanged

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @banderror

@MadameSheema
Copy link
Member

@banderror @serverless does not include all the tests in the second quality gate.

To minimize the number of Cypress configuration and scripts, inside the parallel script, we are overriding the tag configuration when the tests are triggered by the second quality gate.

I understand that the names used on buildkite when the periodic pipeline is triggered might be misleading if you are not familiar with the whole process. We'll work on that to improve it.

@banderror
Copy link
Contributor Author

Closing due to #182600 (comment). It's not a bug and there's nothing to fix, only to improve the naming of config files, scripts, pipelines, steps, etc. Thank you @MadameSheema!

@banderror banderror closed this May 6, 2024
@banderror banderror deleted the include-only-tests-marked-as-serverlessQA-to-2nd-quality-gate branch May 6, 2024 13:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Fixes for quality problems that affect the customer experience impact:critical This issue should be addressed immediately due to a critical level of impact on the product. release_note:skip Skip the PR/issue when compiling release notes Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. test v8.14.0 v8.15.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants