Add serverless emergency release quality gate pipeline

[pheyos]

Summary

This PR adds separately quality gate pipelines for the emergency release process.
This gives us the opportunity to run a different set of checks during an emergency release compared to a regular release.

Details

• Add new emergency quality gates pipeline definitions in .buildkite/pipelines/quality-gates/emergency. These are copies of the regular quality gates pipeline files with the following adjustments:
  • The entry point kibana-serverless-quality-gates-emergency.yml has an adjusted QG_PIPELINE_LOCATION and comment
  • The QA quality gates in pipeline.tests-qa.yaml is reduced to just the CP e2e tests
• Add new pipeline .buildkite/pipeline-resource-definitions/kibana-serverless-quality-gates-emergency.yml is added that will trigger the emergency version of the quality gates.

Other changes

In order to have things around the serverless quality gates and the emergency release consistent, I've taken the opportunity and moved the definitions of the following pipelines from catalog-info.yaml to .buildkite/pipeline-resource-definitions

• buildkite-pipeline-kibana-emergency-release -> .buildkite/pipeline-resource-definitions/kibana-serverless-emergency-release.yml
• kibana-tests-pipeline -> .buildkite/pipeline-resource-definitions/kibana-serverless-quality-gates.yml

[lukeelmers]

This LGTM. Since QA pipeline skips manual verification for an emergency release, the added time of running QA first before staging should be minimal as the RM won't be a bottleneck. If cp e2e test times ever start to creep up, we can reconsider.

My one question is whether we definitely want to include a 24h bake period by default for an emergency release. If it is a true emergency, I would expect us to never wait a full 24h before releasing to noncanary. But I suppose the consistency with the normal release process makes this easier to understand, and gives the RM more latitude to determine how long we want changes to sit in canary. Overall this is a question I wanted to raise -- not a concern -- so I'm fine with proceeding as-is.

[pheyos]

Thanks for your feedback @lukeelmers!

My one question is whether we definitely want to include a 24h bake period by default for an emergency release. If it is a true emergency, I would expect us to never wait a full 24h before releasing to noncanary.

I was thinking about the same. And I agree we will probably never use the full 24 hours in an emergency release.

gives the RM more latitude to determine how long we want changes to sit in canary

This was my main reason for keeping a long bake time here. Elasticsearch is running with 1 hour bake time in production-canary for emergency releases. This allows them to be more hands-off and just let the release roll out if automated checks are passing. However, if some investigation in canary takes more time (which, to be fair, didn't happen so far), it would require stopping the release and kicking the last stage off again. So I thought we'd start a bit more defensive here even though this will require one more manual RM step (cancel the bake time early).
Ultimately, I think we'd like to get to a place where this whole process is mostly automated. In such a world, we definitely need a shorter bake time (like 1 hour). So it basically comes down to the question: do we want to start closer to our goal (1 hour bake time) and adjust up if it doesn't work out or do we start with 24 hours bake time and run with it for some time, knowing that we'll bring it down in the future? I'm fine either way as both approaches have working escape hatches if needed.

[lukeelmers]

do we want to start closer to our goal (1 hour bake time) and adjust up if it doesn't work out or do we start with 24 hours bake time and run with it for some time, knowing that we'll bring it down in the future?

I agree with your logic of being more defensive and starting with a longer bake time, with the understanding that RMs will need to be (as they already have been) very involved in any emergency release process. We can document short-circuiting the bake time as the expected course of action, with the exact bake period determined by the RM based on their judgment of the situation.

We can revisit later and shorten the bake period once we get more comfortable with the new process.

[pheyos]

@elasticmachine merge upstream

[pheyos]

@elasticmachine merge upstream

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: d29c4da

Failed CI Steps

Metrics [docs]

✅ unchanged

History

• 💔 Build #218373 failed 4622343
• 💛 Build #217476 was flaky 0381d32

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @pheyos