[BK] Migrate es-forward (+add versions.json dependent triggering) #184018
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
delanni
force-pushed
the
add-pipeline-parametric-triggers
branch
from
8273a3a
to
b66422a
Compare
delanni
force-pushed
the
add-pipeline-parametric-triggers
branch
from
b66422a
to
07ac9cd
Compare
…led trigger through the new pipeline
delanni
force-pushed
the
add-pipeline-parametric-triggers
branch
from
bf75216
to
6442a19
Compare
delanni
commented
May 23, 2024
| @@ -21,7 +21,6 @@ spec: | |||
| env: | |||
| ELASTIC_SLACK_NOTIFICATIONS_ENABLED: 'true' | |||
| allow_rebuilds: true | |||
| branch_configuration: main 8.14 7.17 | |||
There was a problem hiding this comment.
needs to be removed, or replaced with a generic *.* + main type expression, because we don't want to keep updating this in every version switch
delanni
added
ci:cloud-deploy
|
/ci |
|
/ci |
💛 Build succeeded, but was flaky
Failed CI StepsMetrics [docs]Canvas Sharable Runtime
History
To update your PR or re-run it, just comment with: |
delanni
force-pushed
the
add-pipeline-parametric-triggers
branch
from
bf88a46
to
8c8770f
Compare
delanni
force-pushed
the
add-pipeline-parametric-triggers
branch
from
8c8770f
to
4cb0c47
Compare
delanni
changed the title
…ule on trigger-version-dependent-jobs
delanni
force-pushed
the
add-pipeline-parametric-triggers
branch
from
650b48c
to
5066bb2
Compare
delanni
removed
ci:cloud-deploy
@Ikuni17 - yes, I've updated the description with some indicative runs |
Ikuni17
approved these changes
Jul 9, 2024
.buildkite/scripts/pipelines/trigger_version_dependent_jobs/pipeline.test.ts
Show resolved
Hide resolved
.buildkite/scripts/pipelines/trigger_version_dependent_jobs/pipeline.ts
Outdated
Show resolved
Hide resolved
delanni
added a commit
that referenced
this pull request
Jul 10, 2024
## Summary The ES forward testing only happens from the direction of 7.17, so this file only exists on 7.17, thus it's not a backport, but a direct adjustment of these rules. Follows up: #184018
delanni
added
v8.15.0
backport:prev-minor
💔 All backports failed
Manual backportTo create the backport manually run: Questions ?Please refer to the Backport tool documentation |
delanni
added a commit
to delanni/kibana
that referenced
this pull request
Jul 10, 2024
…astic#184018) ## Goal We'd like to introduce a way to run pipelines that have a dependency on the currently active branch set (managed in [versions.json](./versions.json)). With this, we'd like to migrate over the `es-forward` pipelines (currently: [this](https://buildkite.com/elastic/kibana-7-dot-17-es-8-dot-15-forward-compatibility), and [this](https://buildkite.com/elastic/kibana-7-dot-17-es-8-dot-14-forward-compatibility)) to the new buildkite infra. ## Summary This PR introduces a new pipeline: https://buildkite.com/elastic/kibana-trigger-version-dependent-jobs (through [trigger-version-dependent-jobs.yml](.buildkite/pipeline-resource-definitions/trigger-version-dependent-jobs.yml)). The purpose of this new pipeline is to take the name of a "pipelineSet" that refers to a pipeline, and based on the `versions.json` file, work out what are the branches on which the referred pipeline should be triggered. ### Example: `Trigger ES forward compatibility tests` - a scheduled run on [kibana-trigger-version-dependent-jobs](https://buildkite.com/elastic/kibana-trigger-version-dependent-jobs) with the env var `TRIGGER_PIPELINE_SET=es-forward` runs - the pipeline implementation for `kibana-trigger-version-dependent-jobs` works out (looking at `versions.json`), that the `es-forward` set should trigger https://buildkite.com/elastic/kibana-es-forward (doesn't exist prior to the PR) for (7.17+8.14) and (7.17+8.15) - the pipeline implementation uploads two trigger steps, running https://buildkite.com/elastic/kibana-es-forward in two instances with the relevant parameterization. Since the trigger parameters are derived from the `versions.json` file, if we move on and close `8.14`, and open up `8.16`, this will follow, without having to update the pipeline resources or schedules. ## Changes - 2 pipelines created: [trigger-version-dependent-jobs.yml](.buildkite/pipeline-resource-definitions/trigger-version-dependent-jobs.yml), [kibana-es-forward.yml](.buildkite/pipeline-resource-definitions/kibana-es-forward.yml) - [x] add kibana-es-forward.yml - implementation for `trigger-version-dependent-jobs` added - branch configuration removed from pipelines (kibana-artifacts-staging, kibana-artifacts-snapshot, kibana-artifacts-trigger) - added a script for checking RREs validity (moved a few files) ## Verification I've used the migration staging pipeline (*) to run this: - https://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/130 - Env: `TRIGGER_PIPELINE_SET="artifacts-trigger"` - Result: [(success):](https://buildkite.com/elastic/kibana-artifacts-trigger/builds/10806) it triggered for 8.14 only (as expected) - https://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/131 - Env: `TRIGGER_PIPELINE_SET="es-forward"` - Result: (success): it generated 2 trigger steps, but since the es-forward pipeline doesn't exist, the upload step failed - https://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/132 - Env: `TRIGGER_PIPELINE_SET="artifacts-snapshot"` - Result: (success): it triggered jobs for all 3 open branches (main/8.14/7.17) - https://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/134 - Env: `TRIGGER_PIPELINE_SET="artifacts-staging"` - Result: (success): it triggered 8.14 / 7.14, but not for main (*note: this migration staging pipeline will come in handy even after the migration, to stage newly created pipelines without creating the resource up-front) (cherry picked from commit ea25099) # Conflicts: # .buildkite/pipeline-resource-definitions/kibana-artifacts-snapshot.yml # .buildkite/pipeline-resource-definitions/kibana-artifacts-staging.yml # .buildkite/pipeline-resource-definitions/kibana-artifacts-trigger.yml
delanni
added a commit
that referenced
this pull request
Jul 10, 2024
## Summary I've introduced a typo to the artifact publish script in #184018 - this PR fixes it.
delanni
added a commit
to delanni/kibana
that referenced
this pull request
Jul 10, 2024
…astic#184018) ## Goal We'd like to introduce a way to run pipelines that have a dependency on the currently active branch set (managed in [versions.json](./versions.json)). With this, we'd like to migrate over the `es-forward` pipelines (currently: [this](https://buildkite.com/elastic/kibana-7-dot-17-es-8-dot-15-forward-compatibility), and [this](https://buildkite.com/elastic/kibana-7-dot-17-es-8-dot-14-forward-compatibility)) to the new buildkite infra. ## Summary This PR introduces a new pipeline: https://buildkite.com/elastic/kibana-trigger-version-dependent-jobs (through [trigger-version-dependent-jobs.yml](.buildkite/pipeline-resource-definitions/trigger-version-dependent-jobs.yml)). The purpose of this new pipeline is to take the name of a "pipelineSet" that refers to a pipeline, and based on the `versions.json` file, work out what are the branches on which the referred pipeline should be triggered. ### Example: `Trigger ES forward compatibility tests` - a scheduled run on [kibana-trigger-version-dependent-jobs](https://buildkite.com/elastic/kibana-trigger-version-dependent-jobs) with the env var `TRIGGER_PIPELINE_SET=es-forward` runs - the pipeline implementation for `kibana-trigger-version-dependent-jobs` works out (looking at `versions.json`), that the `es-forward` set should trigger https://buildkite.com/elastic/kibana-es-forward (doesn't exist prior to the PR) for (7.17+8.14) and (7.17+8.15) - the pipeline implementation uploads two trigger steps, running https://buildkite.com/elastic/kibana-es-forward in two instances with the relevant parameterization. Since the trigger parameters are derived from the `versions.json` file, if we move on and close `8.14`, and open up `8.16`, this will follow, without having to update the pipeline resources or schedules. ## Changes - 2 pipelines created: [trigger-version-dependent-jobs.yml](.buildkite/pipeline-resource-definitions/trigger-version-dependent-jobs.yml), [kibana-es-forward.yml](.buildkite/pipeline-resource-definitions/kibana-es-forward.yml) - [x] add kibana-es-forward.yml - implementation for `trigger-version-dependent-jobs` added - branch configuration removed from pipelines (kibana-artifacts-staging, kibana-artifacts-snapshot, kibana-artifacts-trigger) - added a script for checking RREs validity (moved a few files) ## Verification I've used the migration staging pipeline (*) to run this: - https://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/130 - Env: `TRIGGER_PIPELINE_SET="artifacts-trigger"` - Result: [(success):](https://buildkite.com/elastic/kibana-artifacts-trigger/builds/10806) it triggered for 8.14 only (as expected) - https://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/131 - Env: `TRIGGER_PIPELINE_SET="es-forward"` - Result: (success): it generated 2 trigger steps, but since the es-forward pipeline doesn't exist, the upload step failed - https://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/132 - Env: `TRIGGER_PIPELINE_SET="artifacts-snapshot"` - Result: (success): it triggered jobs for all 3 open branches (main/8.14/7.17) - https://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/134 - Env: `TRIGGER_PIPELINE_SET="artifacts-staging"` - Result: (success): it triggered 8.14 / 7.14, but not for main (*note: this migration staging pipeline will come in handy even after the migration, to stage newly created pipelines without creating the resource up-front) (cherry picked from commit ea25099) # Conflicts: # .buildkite/pipeline-resource-definitions/_templates/_new_pipeline.yml # .buildkite/pipeline-resource-definitions/kibana-artifacts-snapshot.yml # .buildkite/pipeline-resource-definitions/kibana-artifacts-staging.yml # .buildkite/pipeline-resource-definitions/kibana-artifacts-trigger.yml # .buildkite/pipeline-resource-definitions/locations.yml # .buildkite/pipeline-resource-definitions/scripts/fix-location-collection.ts
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
pgayvallet
pushed a commit
to pgayvallet/kibana
that referenced
this pull request
Jul 11, 2024
…astic#184018) ## Goal We'd like to introduce a way to run pipelines that have a dependency on the currently active branch set (managed in [versions.json](./versions.json)). With this, we'd like to migrate over the `es-forward` pipelines (currently: [this](https://buildkite.com/elastic/kibana-7-dot-17-es-8-dot-15-forward-compatibility), and [this](https://buildkite.com/elastic/kibana-7-dot-17-es-8-dot-14-forward-compatibility)) to the new buildkite infra. ## Summary This PR introduces a new pipeline: https://buildkite.com/elastic/kibana-trigger-version-dependent-jobs (through [trigger-version-dependent-jobs.yml](.buildkite/pipeline-resource-definitions/trigger-version-dependent-jobs.yml)). The purpose of this new pipeline is to take the name of a "pipelineSet" that refers to a pipeline, and based on the `versions.json` file, work out what are the branches on which the referred pipeline should be triggered. ### Example: `Trigger ES forward compatibility tests` - a scheduled run on [kibana-trigger-version-dependent-jobs](https://buildkite.com/elastic/kibana-trigger-version-dependent-jobs) with the env var `TRIGGER_PIPELINE_SET=es-forward` runs - the pipeline implementation for `kibana-trigger-version-dependent-jobs` works out (looking at `versions.json`), that the `es-forward` set should trigger https://buildkite.com/elastic/kibana-es-forward (doesn't exist prior to the PR) for (7.17+8.14) and (7.17+8.15) - the pipeline implementation uploads two trigger steps, running https://buildkite.com/elastic/kibana-es-forward in two instances with the relevant parameterization. Since the trigger parameters are derived from the `versions.json` file, if we move on and close `8.14`, and open up `8.16`, this will follow, without having to update the pipeline resources or schedules. ## Changes - 2 pipelines created: [trigger-version-dependent-jobs.yml](.buildkite/pipeline-resource-definitions/trigger-version-dependent-jobs.yml), [kibana-es-forward.yml](.buildkite/pipeline-resource-definitions/kibana-es-forward.yml) - [x] add kibana-es-forward.yml - implementation for `trigger-version-dependent-jobs` added - branch configuration removed from pipelines (kibana-artifacts-staging, kibana-artifacts-snapshot, kibana-artifacts-trigger) - added a script for checking RREs validity (moved a few files) ## Verification I've used the migration staging pipeline (*) to run this: - https://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/130 - Env: `TRIGGER_PIPELINE_SET="artifacts-trigger"` - Result: [(success):](https://buildkite.com/elastic/kibana-artifacts-trigger/builds/10806) it triggered for 8.14 only (as expected) - https://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/131 - Env: `TRIGGER_PIPELINE_SET="es-forward"` - Result: (success): it generated 2 trigger steps, but since the es-forward pipeline doesn't exist, the upload step failed - https://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/132 - Env: `TRIGGER_PIPELINE_SET="artifacts-snapshot"` - Result: (success): it triggered jobs for all 3 open branches (main/8.14/7.17) - https://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/134 - Env: `TRIGGER_PIPELINE_SET="artifacts-staging"` - Result: (success): it triggered 8.14 / 7.14, but not for main (*note: this migration staging pipeline will come in handy even after the migration, to stage newly created pipelines without creating the resource up-front)
pgayvallet
pushed a commit
to pgayvallet/kibana
that referenced
this pull request
Jul 11, 2024
## Summary I've introduced a typo to the artifact publish script in elastic#184018 - this PR fixes it.
delanni
added a commit
that referenced
this pull request
Jul 15, 2024
…ng) (#184018) (#187987) # Backport This will backport the following commits from `main` to `8.14`: - [[BK] Migrate es-forward (+add versions.json dependent triggering) (#184018)](#184018) <!--- Backport version: 8.9.8 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Alex Szabo","email":"alex.szabo@elastic.co"},"sourceCommit":{"committedDate":"2024-07-10T10:08:16Z","message":"[BK] Migrate es-forward (+add versions.json dependent triggering) (#184018)\n\n## Goal\r\nWe'd like to introduce a way to run pipelines that have a dependency on\r\nthe currently active branch set (managed in\r\n[versions.json](./versions.json)).\r\n\r\nWith this, we'd like to migrate over the `es-forward` pipelines\r\n(currently:\r\n[this](https://buildkite.com/elastic/kibana-7-dot-17-es-8-dot-15-forward-compatibility),\r\nand\r\n[this](https://buildkite.com/elastic/kibana-7-dot-17-es-8-dot-14-forward-compatibility))\r\nto the new buildkite infra.\r\n\r\n## Summary\r\nThis PR introduces a new pipeline:\r\nhttps://buildkite.com/elastic/kibana-trigger-version-dependent-jobs\r\n(through\r\n[trigger-version-dependent-jobs.yml](.buildkite/pipeline-resource-definitions/trigger-version-dependent-jobs.yml)).\r\n\r\nThe purpose of this new pipeline is to take the name of a \"pipelineSet\"\r\nthat refers to a pipeline, and based on the `versions.json` file, work\r\nout what are the branches on which the referred pipeline should be\r\ntriggered.\r\n\r\n### Example: `Trigger ES forward compatibility tests`\r\n- a scheduled run on\r\n[kibana-trigger-version-dependent-jobs](https://buildkite.com/elastic/kibana-trigger-version-dependent-jobs)\r\nwith the env var `TRIGGER_PIPELINE_SET=es-forward` runs\r\n- the pipeline implementation for\r\n`kibana-trigger-version-dependent-jobs` works out (looking at\r\n`versions.json`), that the `es-forward` set should trigger\r\nhttps://buildkite.com/elastic/kibana-es-forward (doesn't exist prior to\r\nthe PR) for (7.17+8.14) and (7.17+8.15)\r\n- the pipeline implementation uploads two trigger steps, running\r\nhttps://buildkite.com/elastic/kibana-es-forward in two instances with\r\nthe relevant parameterization.\r\n\r\nSince the trigger parameters are derived from the `versions.json` file,\r\nif we move on and close `8.14`, and open up `8.16`, this will follow,\r\nwithout having to update the pipeline resources or schedules.\r\n\r\n## Changes\r\n- 2 pipelines created:\r\n[trigger-version-dependent-jobs.yml](.buildkite/pipeline-resource-definitions/trigger-version-dependent-jobs.yml),\r\n[kibana-es-forward.yml](.buildkite/pipeline-resource-definitions/kibana-es-forward.yml)\r\n - [x] add kibana-es-forward.yml\r\n - implementation for `trigger-version-dependent-jobs` added\r\n- branch configuration removed from pipelines (kibana-artifacts-staging,\r\nkibana-artifacts-snapshot, kibana-artifacts-trigger)\r\n - added a script for checking RREs validity (moved a few files)\r\n\r\n## Verification\r\nI've used the migration staging pipeline (*) to run this:\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/130\r\n - Env: `TRIGGER_PIPELINE_SET=\"artifacts-trigger\"`\r\n- Result:\r\n[(success):](https://buildkite.com/elastic/kibana-artifacts-trigger/builds/10806)\r\nit triggered for 8.14 only (as expected)\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/131\r\n - Env: `TRIGGER_PIPELINE_SET=\"es-forward\"`\r\n- Result: (success): it generated 2 trigger steps, but since the\r\nes-forward pipeline doesn't exist, the upload step failed\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/132\r\n - Env: `TRIGGER_PIPELINE_SET=\"artifacts-snapshot\"`\r\n- Result: (success): it triggered jobs for all 3 open branches\r\n(main/8.14/7.17)\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/134\r\n - Env: `TRIGGER_PIPELINE_SET=\"artifacts-staging\"`\r\n - Result: (success): it triggered 8.14 / 7.14, but not for main\r\n\r\n\r\n(*note: this migration staging pipeline will come in handy even after\r\nthe migration, to stage newly created pipelines without creating the\r\nresource up-front)","sha":"ea2509914f2238c646bb63eb5bbbb240365fd04d","branchLabelMapping":{"^v8.16.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Operations","release_note:skip","backport:prev-minor","v8.15.0","v8.16.0"],"number":184018,"url":"https://github.com/elastic/kibana/pull/184018","mergeCommit":{"message":"[BK] Migrate es-forward (+add versions.json dependent triggering) (#184018)\n\n## Goal\r\nWe'd like to introduce a way to run pipelines that have a dependency on\r\nthe currently active branch set (managed in\r\n[versions.json](./versions.json)).\r\n\r\nWith this, we'd like to migrate over the `es-forward` pipelines\r\n(currently:\r\n[this](https://buildkite.com/elastic/kibana-7-dot-17-es-8-dot-15-forward-compatibility),\r\nand\r\n[this](https://buildkite.com/elastic/kibana-7-dot-17-es-8-dot-14-forward-compatibility))\r\nto the new buildkite infra.\r\n\r\n## Summary\r\nThis PR introduces a new pipeline:\r\nhttps://buildkite.com/elastic/kibana-trigger-version-dependent-jobs\r\n(through\r\n[trigger-version-dependent-jobs.yml](.buildkite/pipeline-resource-definitions/trigger-version-dependent-jobs.yml)).\r\n\r\nThe purpose of this new pipeline is to take the name of a \"pipelineSet\"\r\nthat refers to a pipeline, and based on the `versions.json` file, work\r\nout what are the branches on which the referred pipeline should be\r\ntriggered.\r\n\r\n### Example: `Trigger ES forward compatibility tests`\r\n- a scheduled run on\r\n[kibana-trigger-version-dependent-jobs](https://buildkite.com/elastic/kibana-trigger-version-dependent-jobs)\r\nwith the env var `TRIGGER_PIPELINE_SET=es-forward` runs\r\n- the pipeline implementation for\r\n`kibana-trigger-version-dependent-jobs` works out (looking at\r\n`versions.json`), that the `es-forward` set should trigger\r\nhttps://buildkite.com/elastic/kibana-es-forward (doesn't exist prior to\r\nthe PR) for (7.17+8.14) and (7.17+8.15)\r\n- the pipeline implementation uploads two trigger steps, running\r\nhttps://buildkite.com/elastic/kibana-es-forward in two instances with\r\nthe relevant parameterization.\r\n\r\nSince the trigger parameters are derived from the `versions.json` file,\r\nif we move on and close `8.14`, and open up `8.16`, this will follow,\r\nwithout having to update the pipeline resources or schedules.\r\n\r\n## Changes\r\n- 2 pipelines created:\r\n[trigger-version-dependent-jobs.yml](.buildkite/pipeline-resource-definitions/trigger-version-dependent-jobs.yml),\r\n[kibana-es-forward.yml](.buildkite/pipeline-resource-definitions/kibana-es-forward.yml)\r\n - [x] add kibana-es-forward.yml\r\n - implementation for `trigger-version-dependent-jobs` added\r\n- branch configuration removed from pipelines (kibana-artifacts-staging,\r\nkibana-artifacts-snapshot, kibana-artifacts-trigger)\r\n - added a script for checking RREs validity (moved a few files)\r\n\r\n## Verification\r\nI've used the migration staging pipeline (*) to run this:\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/130\r\n - Env: `TRIGGER_PIPELINE_SET=\"artifacts-trigger\"`\r\n- Result:\r\n[(success):](https://buildkite.com/elastic/kibana-artifacts-trigger/builds/10806)\r\nit triggered for 8.14 only (as expected)\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/131\r\n - Env: `TRIGGER_PIPELINE_SET=\"es-forward\"`\r\n- Result: (success): it generated 2 trigger steps, but since the\r\nes-forward pipeline doesn't exist, the upload step failed\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/132\r\n - Env: `TRIGGER_PIPELINE_SET=\"artifacts-snapshot\"`\r\n- Result: (success): it triggered jobs for all 3 open branches\r\n(main/8.14/7.17)\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/134\r\n - Env: `TRIGGER_PIPELINE_SET=\"artifacts-staging\"`\r\n - Result: (success): it triggered 8.14 / 7.14, but not for main\r\n\r\n\r\n(*note: this migration staging pipeline will come in handy even after\r\nthe migration, to stage newly created pipelines without creating the\r\nresource up-front)","sha":"ea2509914f2238c646bb63eb5bbbb240365fd04d"}},"sourceBranch":"main","suggestedTargetBranches":["8.15"],"targetPullRequestStates":[{"branch":"8.15","label":"v8.15.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.16.0","labelRegex":"^v8.16.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/184018","number":184018,"mergeCommit":{"message":"[BK] Migrate es-forward (+add versions.json dependent triggering) (#184018)\n\n## Goal\r\nWe'd like to introduce a way to run pipelines that have a dependency on\r\nthe currently active branch set (managed in\r\n[versions.json](./versions.json)).\r\n\r\nWith this, we'd like to migrate over the `es-forward` pipelines\r\n(currently:\r\n[this](https://buildkite.com/elastic/kibana-7-dot-17-es-8-dot-15-forward-compatibility),\r\nand\r\n[this](https://buildkite.com/elastic/kibana-7-dot-17-es-8-dot-14-forward-compatibility))\r\nto the new buildkite infra.\r\n\r\n## Summary\r\nThis PR introduces a new pipeline:\r\nhttps://buildkite.com/elastic/kibana-trigger-version-dependent-jobs\r\n(through\r\n[trigger-version-dependent-jobs.yml](.buildkite/pipeline-resource-definitions/trigger-version-dependent-jobs.yml)).\r\n\r\nThe purpose of this new pipeline is to take the name of a \"pipelineSet\"\r\nthat refers to a pipeline, and based on the `versions.json` file, work\r\nout what are the branches on which the referred pipeline should be\r\ntriggered.\r\n\r\n### Example: `Trigger ES forward compatibility tests`\r\n- a scheduled run on\r\n[kibana-trigger-version-dependent-jobs](https://buildkite.com/elastic/kibana-trigger-version-dependent-jobs)\r\nwith the env var `TRIGGER_PIPELINE_SET=es-forward` runs\r\n- the pipeline implementation for\r\n`kibana-trigger-version-dependent-jobs` works out (looking at\r\n`versions.json`), that the `es-forward` set should trigger\r\nhttps://buildkite.com/elastic/kibana-es-forward (doesn't exist prior to\r\nthe PR) for (7.17+8.14) and (7.17+8.15)\r\n- the pipeline implementation uploads two trigger steps, running\r\nhttps://buildkite.com/elastic/kibana-es-forward in two instances with\r\nthe relevant parameterization.\r\n\r\nSince the trigger parameters are derived from the `versions.json` file,\r\nif we move on and close `8.14`, and open up `8.16`, this will follow,\r\nwithout having to update the pipeline resources or schedules.\r\n\r\n## Changes\r\n- 2 pipelines created:\r\n[trigger-version-dependent-jobs.yml](.buildkite/pipeline-resource-definitions/trigger-version-dependent-jobs.yml),\r\n[kibana-es-forward.yml](.buildkite/pipeline-resource-definitions/kibana-es-forward.yml)\r\n - [x] add kibana-es-forward.yml\r\n - implementation for `trigger-version-dependent-jobs` added\r\n- branch configuration removed from pipelines (kibana-artifacts-staging,\r\nkibana-artifacts-snapshot, kibana-artifacts-trigger)\r\n - added a script for checking RREs validity (moved a few files)\r\n\r\n## Verification\r\nI've used the migration staging pipeline (*) to run this:\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/130\r\n - Env: `TRIGGER_PIPELINE_SET=\"artifacts-trigger\"`\r\n- Result:\r\n[(success):](https://buildkite.com/elastic/kibana-artifacts-trigger/builds/10806)\r\nit triggered for 8.14 only (as expected)\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/131\r\n - Env: `TRIGGER_PIPELINE_SET=\"es-forward\"`\r\n- Result: (success): it generated 2 trigger steps, but since the\r\nes-forward pipeline doesn't exist, the upload step failed\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/132\r\n - Env: `TRIGGER_PIPELINE_SET=\"artifacts-snapshot\"`\r\n- Result: (success): it triggered jobs for all 3 open branches\r\n(main/8.14/7.17)\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/134\r\n - Env: `TRIGGER_PIPELINE_SET=\"artifacts-staging\"`\r\n - Result: (success): it triggered 8.14 / 7.14, but not for main\r\n\r\n\r\n(*note: this migration staging pipeline will come in handy even after\r\nthe migration, to stage newly created pipelines without creating the\r\nresource up-front)","sha":"ea2509914f2238c646bb63eb5bbbb240365fd04d"}}]}] BACKPORT-->
delanni
added a commit
that referenced
this pull request
Jul 15, 2024
…ng) (#184018) (#187983) # Backport This will backport the following commits from `main` to `8.15`: - [[BK] Migrate es-forward (+add versions.json dependent triggering) (#184018)](#184018) <!--- Backport version: 8.9.8 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Alex Szabo","email":"alex.szabo@elastic.co"},"sourceCommit":{"committedDate":"2024-07-10T10:08:16Z","message":"[BK] Migrate es-forward (+add versions.json dependent triggering) (#184018)\n\n## Goal\r\nWe'd like to introduce a way to run pipelines that have a dependency on\r\nthe currently active branch set (managed in\r\n[versions.json](./versions.json)).\r\n\r\nWith this, we'd like to migrate over the `es-forward` pipelines\r\n(currently:\r\n[this](https://buildkite.com/elastic/kibana-7-dot-17-es-8-dot-15-forward-compatibility),\r\nand\r\n[this](https://buildkite.com/elastic/kibana-7-dot-17-es-8-dot-14-forward-compatibility))\r\nto the new buildkite infra.\r\n\r\n## Summary\r\nThis PR introduces a new pipeline:\r\nhttps://buildkite.com/elastic/kibana-trigger-version-dependent-jobs\r\n(through\r\n[trigger-version-dependent-jobs.yml](.buildkite/pipeline-resource-definitions/trigger-version-dependent-jobs.yml)).\r\n\r\nThe purpose of this new pipeline is to take the name of a \"pipelineSet\"\r\nthat refers to a pipeline, and based on the `versions.json` file, work\r\nout what are the branches on which the referred pipeline should be\r\ntriggered.\r\n\r\n### Example: `Trigger ES forward compatibility tests`\r\n- a scheduled run on\r\n[kibana-trigger-version-dependent-jobs](https://buildkite.com/elastic/kibana-trigger-version-dependent-jobs)\r\nwith the env var `TRIGGER_PIPELINE_SET=es-forward` runs\r\n- the pipeline implementation for\r\n`kibana-trigger-version-dependent-jobs` works out (looking at\r\n`versions.json`), that the `es-forward` set should trigger\r\nhttps://buildkite.com/elastic/kibana-es-forward (doesn't exist prior to\r\nthe PR) for (7.17+8.14) and (7.17+8.15)\r\n- the pipeline implementation uploads two trigger steps, running\r\nhttps://buildkite.com/elastic/kibana-es-forward in two instances with\r\nthe relevant parameterization.\r\n\r\nSince the trigger parameters are derived from the `versions.json` file,\r\nif we move on and close `8.14`, and open up `8.16`, this will follow,\r\nwithout having to update the pipeline resources or schedules.\r\n\r\n## Changes\r\n- 2 pipelines created:\r\n[trigger-version-dependent-jobs.yml](.buildkite/pipeline-resource-definitions/trigger-version-dependent-jobs.yml),\r\n[kibana-es-forward.yml](.buildkite/pipeline-resource-definitions/kibana-es-forward.yml)\r\n - [x] add kibana-es-forward.yml\r\n - implementation for `trigger-version-dependent-jobs` added\r\n- branch configuration removed from pipelines (kibana-artifacts-staging,\r\nkibana-artifacts-snapshot, kibana-artifacts-trigger)\r\n - added a script for checking RREs validity (moved a few files)\r\n\r\n## Verification\r\nI've used the migration staging pipeline (*) to run this:\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/130\r\n - Env: `TRIGGER_PIPELINE_SET=\"artifacts-trigger\"`\r\n- Result:\r\n[(success):](https://buildkite.com/elastic/kibana-artifacts-trigger/builds/10806)\r\nit triggered for 8.14 only (as expected)\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/131\r\n - Env: `TRIGGER_PIPELINE_SET=\"es-forward\"`\r\n- Result: (success): it generated 2 trigger steps, but since the\r\nes-forward pipeline doesn't exist, the upload step failed\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/132\r\n - Env: `TRIGGER_PIPELINE_SET=\"artifacts-snapshot\"`\r\n- Result: (success): it triggered jobs for all 3 open branches\r\n(main/8.14/7.17)\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/134\r\n - Env: `TRIGGER_PIPELINE_SET=\"artifacts-staging\"`\r\n - Result: (success): it triggered 8.14 / 7.14, but not for main\r\n\r\n\r\n(*note: this migration staging pipeline will come in handy even after\r\nthe migration, to stage newly created pipelines without creating the\r\nresource up-front)","sha":"ea2509914f2238c646bb63eb5bbbb240365fd04d","branchLabelMapping":{"^v8.16.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Operations","release_note:skip","backport:prev-minor","v8.15.0","v8.16.0"],"number":184018,"url":"https://github.com/elastic/kibana/pull/184018","mergeCommit":{"message":"[BK] Migrate es-forward (+add versions.json dependent triggering) (#184018)\n\n## Goal\r\nWe'd like to introduce a way to run pipelines that have a dependency on\r\nthe currently active branch set (managed in\r\n[versions.json](./versions.json)).\r\n\r\nWith this, we'd like to migrate over the `es-forward` pipelines\r\n(currently:\r\n[this](https://buildkite.com/elastic/kibana-7-dot-17-es-8-dot-15-forward-compatibility),\r\nand\r\n[this](https://buildkite.com/elastic/kibana-7-dot-17-es-8-dot-14-forward-compatibility))\r\nto the new buildkite infra.\r\n\r\n## Summary\r\nThis PR introduces a new pipeline:\r\nhttps://buildkite.com/elastic/kibana-trigger-version-dependent-jobs\r\n(through\r\n[trigger-version-dependent-jobs.yml](.buildkite/pipeline-resource-definitions/trigger-version-dependent-jobs.yml)).\r\n\r\nThe purpose of this new pipeline is to take the name of a \"pipelineSet\"\r\nthat refers to a pipeline, and based on the `versions.json` file, work\r\nout what are the branches on which the referred pipeline should be\r\ntriggered.\r\n\r\n### Example: `Trigger ES forward compatibility tests`\r\n- a scheduled run on\r\n[kibana-trigger-version-dependent-jobs](https://buildkite.com/elastic/kibana-trigger-version-dependent-jobs)\r\nwith the env var `TRIGGER_PIPELINE_SET=es-forward` runs\r\n- the pipeline implementation for\r\n`kibana-trigger-version-dependent-jobs` works out (looking at\r\n`versions.json`), that the `es-forward` set should trigger\r\nhttps://buildkite.com/elastic/kibana-es-forward (doesn't exist prior to\r\nthe PR) for (7.17+8.14) and (7.17+8.15)\r\n- the pipeline implementation uploads two trigger steps, running\r\nhttps://buildkite.com/elastic/kibana-es-forward in two instances with\r\nthe relevant parameterization.\r\n\r\nSince the trigger parameters are derived from the `versions.json` file,\r\nif we move on and close `8.14`, and open up `8.16`, this will follow,\r\nwithout having to update the pipeline resources or schedules.\r\n\r\n## Changes\r\n- 2 pipelines created:\r\n[trigger-version-dependent-jobs.yml](.buildkite/pipeline-resource-definitions/trigger-version-dependent-jobs.yml),\r\n[kibana-es-forward.yml](.buildkite/pipeline-resource-definitions/kibana-es-forward.yml)\r\n - [x] add kibana-es-forward.yml\r\n - implementation for `trigger-version-dependent-jobs` added\r\n- branch configuration removed from pipelines (kibana-artifacts-staging,\r\nkibana-artifacts-snapshot, kibana-artifacts-trigger)\r\n - added a script for checking RREs validity (moved a few files)\r\n\r\n## Verification\r\nI've used the migration staging pipeline (*) to run this:\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/130\r\n - Env: `TRIGGER_PIPELINE_SET=\"artifacts-trigger\"`\r\n- Result:\r\n[(success):](https://buildkite.com/elastic/kibana-artifacts-trigger/builds/10806)\r\nit triggered for 8.14 only (as expected)\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/131\r\n - Env: `TRIGGER_PIPELINE_SET=\"es-forward\"`\r\n- Result: (success): it generated 2 trigger steps, but since the\r\nes-forward pipeline doesn't exist, the upload step failed\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/132\r\n - Env: `TRIGGER_PIPELINE_SET=\"artifacts-snapshot\"`\r\n- Result: (success): it triggered jobs for all 3 open branches\r\n(main/8.14/7.17)\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/134\r\n - Env: `TRIGGER_PIPELINE_SET=\"artifacts-staging\"`\r\n - Result: (success): it triggered 8.14 / 7.14, but not for main\r\n\r\n\r\n(*note: this migration staging pipeline will come in handy even after\r\nthe migration, to stage newly created pipelines without creating the\r\nresource up-front)","sha":"ea2509914f2238c646bb63eb5bbbb240365fd04d"}},"sourceBranch":"main","suggestedTargetBranches":["8.15"],"targetPullRequestStates":[{"branch":"8.15","label":"v8.15.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.16.0","labelRegex":"^v8.16.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/184018","number":184018,"mergeCommit":{"message":"[BK] Migrate es-forward (+add versions.json dependent triggering) (#184018)\n\n## Goal\r\nWe'd like to introduce a way to run pipelines that have a dependency on\r\nthe currently active branch set (managed in\r\n[versions.json](./versions.json)).\r\n\r\nWith this, we'd like to migrate over the `es-forward` pipelines\r\n(currently:\r\n[this](https://buildkite.com/elastic/kibana-7-dot-17-es-8-dot-15-forward-compatibility),\r\nand\r\n[this](https://buildkite.com/elastic/kibana-7-dot-17-es-8-dot-14-forward-compatibility))\r\nto the new buildkite infra.\r\n\r\n## Summary\r\nThis PR introduces a new pipeline:\r\nhttps://buildkite.com/elastic/kibana-trigger-version-dependent-jobs\r\n(through\r\n[trigger-version-dependent-jobs.yml](.buildkite/pipeline-resource-definitions/trigger-version-dependent-jobs.yml)).\r\n\r\nThe purpose of this new pipeline is to take the name of a \"pipelineSet\"\r\nthat refers to a pipeline, and based on the `versions.json` file, work\r\nout what are the branches on which the referred pipeline should be\r\ntriggered.\r\n\r\n### Example: `Trigger ES forward compatibility tests`\r\n- a scheduled run on\r\n[kibana-trigger-version-dependent-jobs](https://buildkite.com/elastic/kibana-trigger-version-dependent-jobs)\r\nwith the env var `TRIGGER_PIPELINE_SET=es-forward` runs\r\n- the pipeline implementation for\r\n`kibana-trigger-version-dependent-jobs` works out (looking at\r\n`versions.json`), that the `es-forward` set should trigger\r\nhttps://buildkite.com/elastic/kibana-es-forward (doesn't exist prior to\r\nthe PR) for (7.17+8.14) and (7.17+8.15)\r\n- the pipeline implementation uploads two trigger steps, running\r\nhttps://buildkite.com/elastic/kibana-es-forward in two instances with\r\nthe relevant parameterization.\r\n\r\nSince the trigger parameters are derived from the `versions.json` file,\r\nif we move on and close `8.14`, and open up `8.16`, this will follow,\r\nwithout having to update the pipeline resources or schedules.\r\n\r\n## Changes\r\n- 2 pipelines created:\r\n[trigger-version-dependent-jobs.yml](.buildkite/pipeline-resource-definitions/trigger-version-dependent-jobs.yml),\r\n[kibana-es-forward.yml](.buildkite/pipeline-resource-definitions/kibana-es-forward.yml)\r\n - [x] add kibana-es-forward.yml\r\n - implementation for `trigger-version-dependent-jobs` added\r\n- branch configuration removed from pipelines (kibana-artifacts-staging,\r\nkibana-artifacts-snapshot, kibana-artifacts-trigger)\r\n - added a script for checking RREs validity (moved a few files)\r\n\r\n## Verification\r\nI've used the migration staging pipeline (*) to run this:\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/130\r\n - Env: `TRIGGER_PIPELINE_SET=\"artifacts-trigger\"`\r\n- Result:\r\n[(success):](https://buildkite.com/elastic/kibana-artifacts-trigger/builds/10806)\r\nit triggered for 8.14 only (as expected)\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/131\r\n - Env: `TRIGGER_PIPELINE_SET=\"es-forward\"`\r\n- Result: (success): it generated 2 trigger steps, but since the\r\nes-forward pipeline doesn't exist, the upload step failed\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/132\r\n - Env: `TRIGGER_PIPELINE_SET=\"artifacts-snapshot\"`\r\n- Result: (success): it triggered jobs for all 3 open branches\r\n(main/8.14/7.17)\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/134\r\n - Env: `TRIGGER_PIPELINE_SET=\"artifacts-staging\"`\r\n - Result: (success): it triggered 8.14 / 7.14, but not for main\r\n\r\n\r\n(*note: this migration staging pipeline will come in handy even after\r\nthe migration, to stage newly created pipelines without creating the\r\nresource up-front)","sha":"ea2509914f2238c646bb63eb5bbbb240365fd04d"}}]}] BACKPORT-->
adelisle
pushed a commit
to Makila-AI/kibana
that referenced
this pull request
Aug 5, 2024
## Summary The ES forward testing only happens from the direction of 7.17, so this file only exists on 7.17, thus it's not a backport, but a direct adjustment of these rules. Follows up: elastic#184018
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Goal
We'd like to introduce a way to run pipelines that have a dependency on the currently active branch set (managed in versions.json).
With this, we'd like to migrate over the
es-forwardpipelines (currently: this, and this) to the new buildkite infra.Summary
This PR introduces a new pipeline: https://buildkite.com/elastic/kibana-trigger-version-dependent-jobs (through trigger-version-dependent-jobs.yml).
The purpose of this new pipeline is to take the name of a "pipelineSet" that refers to a pipeline, and based on the
versions.jsonfile, work out what are the branches on which the referred pipeline should be triggered.Example:
Trigger ES forward compatibility testsTRIGGER_PIPELINE_SET=es-forwardrunskibana-trigger-version-dependent-jobsworks out (looking atversions.json), that thees-forwardset should trigger https://buildkite.com/elastic/kibana-es-forward (doesn't exist prior to the PR) for (7.17+8.14) and (7.17+8.15)Since the trigger parameters are derived from the
versions.jsonfile, if we move on and close8.14, and open up8.16, this will follow, without having to update the pipeline resources or schedules.Changes
trigger-version-dependent-jobsaddedVerification
I've used the migration staging pipeline (*) to run this:
TRIGGER_PIPELINE_SET="artifacts-trigger"TRIGGER_PIPELINE_SET="es-forward"TRIGGER_PIPELINE_SET="artifacts-snapshot"TRIGGER_PIPELINE_SET="artifacts-staging"(*note: this migration staging pipeline will come in handy even after the migration, to stage newly created pipelines without creating the resource up-front)