-
Notifications
You must be signed in to change notification settings - Fork 8.2k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[BK] Migrate es-forward (+add versions.json dependent triggering) (#1…
…84018) ## Goal We'd like to introduce a way to run pipelines that have a dependency on the currently active branch set (managed in [versions.json](./versions.json)). With this, we'd like to migrate over the `es-forward` pipelines (currently: [this](https://buildkite.com/elastic/kibana-7-dot-17-es-8-dot-15-forward-compatibility), and [this](https://buildkite.com/elastic/kibana-7-dot-17-es-8-dot-14-forward-compatibility)) to the new buildkite infra. ## Summary This PR introduces a new pipeline: https://buildkite.com/elastic/kibana-trigger-version-dependent-jobs (through [trigger-version-dependent-jobs.yml](.buildkite/pipeline-resource-definitions/trigger-version-dependent-jobs.yml)). The purpose of this new pipeline is to take the name of a "pipelineSet" that refers to a pipeline, and based on the `versions.json` file, work out what are the branches on which the referred pipeline should be triggered. ### Example: `Trigger ES forward compatibility tests` - a scheduled run on [kibana-trigger-version-dependent-jobs](https://buildkite.com/elastic/kibana-trigger-version-dependent-jobs) with the env var `TRIGGER_PIPELINE_SET=es-forward` runs - the pipeline implementation for `kibana-trigger-version-dependent-jobs` works out (looking at `versions.json`), that the `es-forward` set should trigger https://buildkite.com/elastic/kibana-es-forward (doesn't exist prior to the PR) for (7.17+8.14) and (7.17+8.15) - the pipeline implementation uploads two trigger steps, running https://buildkite.com/elastic/kibana-es-forward in two instances with the relevant parameterization. Since the trigger parameters are derived from the `versions.json` file, if we move on and close `8.14`, and open up `8.16`, this will follow, without having to update the pipeline resources or schedules. ## Changes - 2 pipelines created: [trigger-version-dependent-jobs.yml](.buildkite/pipeline-resource-definitions/trigger-version-dependent-jobs.yml), [kibana-es-forward.yml](.buildkite/pipeline-resource-definitions/kibana-es-forward.yml) - [x] add kibana-es-forward.yml - implementation for `trigger-version-dependent-jobs` added - branch configuration removed from pipelines (kibana-artifacts-staging, kibana-artifacts-snapshot, kibana-artifacts-trigger) - added a script for checking RREs validity (moved a few files) ## Verification I've used the migration staging pipeline (*) to run this: - https://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/130 - Env: `TRIGGER_PIPELINE_SET="artifacts-trigger"` - Result: [(success):](https://buildkite.com/elastic/kibana-artifacts-trigger/builds/10806) it triggered for 8.14 only (as expected) - https://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/131 - Env: `TRIGGER_PIPELINE_SET="es-forward"` - Result: (success): it generated 2 trigger steps, but since the es-forward pipeline doesn't exist, the upload step failed - https://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/132 - Env: `TRIGGER_PIPELINE_SET="artifacts-snapshot"` - Result: (success): it triggered jobs for all 3 open branches (main/8.14/7.17) - https://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/134 - Env: `TRIGGER_PIPELINE_SET="artifacts-staging"` - Result: (success): it triggered 8.14 / 7.14, but not for main (*note: this migration staging pipeline will come in handy even after the migration, to stage newly created pipelines without creating the resource up-front) (cherry picked from commit ea25099) # Conflicts: # .buildkite/pipeline-resource-definitions/kibana-artifacts-snapshot.yml # .buildkite/pipeline-resource-definitions/kibana-artifacts-staging.yml # .buildkite/pipeline-resource-definitions/kibana-artifacts-trigger.yml
- Loading branch information
Showing
18 changed files
with
543 additions
and
37 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| # Buildkite pipeline resource definitions | ||
|
|
||
| ## Overview | ||
| The pipeline resources are "RRE" (real resource entities) that are used to create/maintain buildkite pipelines. | ||
|
|
||
| The resources described in these files are parsed and loaded to Backstage (https://backstage.elastic.dev). | ||
| From there, [Terrazzo](https://buildkite.com/elastic/terrazzo/) is generating and updating the buildkite pipelines. | ||
|
|
||
| These pipelines are referenced indirectly through the root's [`catalog-info.yaml`](../../catalog-info.yaml) file in order to reduce bloat in the main resources file. | ||
| There's a location file that collects files defined in this folder ([locations.yml](locations.yml)), this file needs to be updated in order to keep track of local files. | ||
|
|
||
| Available parameters and further help can be found here: https://docs.elastic.dev/ci/getting-started-with-buildkite-at-elastic | ||
|
|
||
| ## Creating a new pipeline resource definition | ||
| The easiest way to create a new pipeline is either by copying and editing a similar pipeline, | ||
| or by copying a blank template (see [_new_pipeline.yml](_templates/_new_pipeline.yml)) and editing that. | ||
|
|
||
| You can validate your pipeline's structural integrity, and it's conformity to baseline rules by running the following command: | ||
| ```bash | ||
| .buildkite/pipeline-resource-definitions/scripts/validate-pipeline-definition.sh <path_to_your_pipeline_file> | ||
| ``` | ||
|
|
||
| Once you've added the file, you should update the [locations.yml](locations.yml) file to include the new pipeline, or run the following command to update it: | ||
| ```bash | ||
| .buildkite/pipeline-resource-definitions/scripts/fix-location-collection.ts | ||
| ``` | ||
|
|
||
| Add your pipeline implementation, commit & push & merge. The pipeline resource will appear in Backstage within minutes, then the pipeline will be added to Buildkite within ~10 minutes. |
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
41 changes: 41 additions & 0 deletions
41
.buildkite/pipeline-resource-definitions/kibana-es-forward-testing.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| # yaml-language-server: $schema=https://gist.githubusercontent.com/elasticmachine/988b80dae436cafea07d9a4a460a011d/raw/rre.schema.json | ||
| apiVersion: backstage.io/v1alpha1 | ||
| kind: Resource | ||
| metadata: | ||
| name: bk-kibana-es-forward-compatibility-testing | ||
| description: Forward compatibility testing between Kibana 7.17 and ES 8+ | ||
| links: | ||
| - url: 'https://buildkite.com/elastic/kibana-es-forward-compatibility-testing' | ||
| title: Pipeline link | ||
| spec: | ||
| type: buildkite-pipeline | ||
| system: buildkite | ||
| owner: 'group:kibana-operations' | ||
| implementation: | ||
| apiVersion: buildkite.elastic.dev/v1 | ||
| kind: Pipeline | ||
| metadata: | ||
| name: kibana / ES Forward Compatibility Testing | ||
| description: Forward compatibility testing between Kibana 7.17 and ES 8+ | ||
| spec: | ||
| env: | ||
| SLACK_NOTIFICATIONS_CHANNEL: '#kibana-operations-alerts' | ||
| ELASTIC_SLACK_NOTIFICATIONS_ENABLED: 'true' | ||
| allow_rebuilds: false | ||
| branch_configuration: main | ||
| default_branch: main | ||
| repository: elastic/kibana | ||
| pipeline_file: .buildkite/pipelines/es_forward.yml # Note: this file exists in 7.17 only | ||
| skip_intermediate_builds: false | ||
| provider_settings: | ||
| prefix_pull_request_fork_branch_names: false | ||
| trigger_mode: none | ||
| teams: | ||
| kibana-operations: | ||
| access_level: MANAGE_BUILD_AND_READ | ||
| appex-qa: | ||
| access_level: MANAGE_BUILD_AND_READ | ||
| kibana-tech-leads: | ||
| access_level: MANAGE_BUILD_AND_READ | ||
| everyone: | ||
| access_level: BUILD_AND_READ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
28 changes: 28 additions & 0 deletions
28
.buildkite/pipeline-resource-definitions/scripts/validate-pipeline-definition.sh
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| # This script is used to validate a single RRE for a pipeline definition. | ||
|
|
||
| TARGET_FILE=$1 | ||
|
|
||
| if [ -z "$TARGET_FILE" ]; then | ||
| echo "Usage: $0 <path_to_your_pipeline_file>" | ||
| exit 1 | ||
| fi | ||
|
|
||
| echo "Validating $TARGET_FILE..." | ||
| ABSOLUTE_PATH=$(realpath "$TARGET_FILE") | ||
| FILE_NAME=$(basename "$ABSOLUTE_PATH") | ||
| FOLDER_NAME=$(dirname "$ABSOLUTE_PATH") | ||
|
|
||
| docker run -it \ | ||
| --mount type=bind,source="$FOLDER_NAME",target=/home/app/ \ | ||
| docker.elastic.co/ci-agent-images/pipelib \ | ||
| rre validate --backstage-entity-aware "/home/app/$FILE_NAME" | ||
|
|
||
| if [ $? -ne 0 ]; then | ||
| echo "$FILE_NAME invalid ❌" | ||
| exit 1 | ||
| else | ||
| echo "$FILE_NAME valid ✅" | ||
| exit 0 | ||
| fi |
71 changes: 71 additions & 0 deletions
71
.buildkite/pipeline-resource-definitions/trigger-version-dependent-jobs.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,71 @@ | ||
| ### | ||
| # For more information on authoring pipeline definitions, | ||
| # follow the guides at https://docs.elastic.dev/ci/getting-started-with-buildkite-at-elastic | ||
| ### | ||
| # yaml-language-server: $schema=https://gist.githubusercontent.com/elasticmachine/988b80dae436cafea07d9a4a460a011d/raw/rre.schema.json | ||
| apiVersion: backstage.io/v1alpha1 | ||
| kind: Resource | ||
| metadata: | ||
| name: bk-kibana-trigger-version-dependent-jobs | ||
| description: 'Trigger version-dependent jobs' | ||
| links: | ||
| - url: 'https://buildkite.com/elastic/kibana-trigger-version-dependent-jobs' | ||
| title: Pipeline link | ||
| spec: | ||
| type: buildkite-pipeline | ||
| system: buildkite | ||
| owner: 'group:kibana-operations' | ||
| implementation: | ||
| apiVersion: buildkite.elastic.dev/v1 | ||
| kind: Pipeline | ||
| metadata: | ||
| name: kibana / trigger version-dependent jobs | ||
| description: 'Trigger version-dependent jobs' | ||
| spec: | ||
| env: | ||
| SLACK_NOTIFICATIONS_CHANNEL: '#kibana-operations-alerts' | ||
| ELASTIC_SLACK_NOTIFICATIONS_ENABLED: 'true' | ||
|
|
||
| allow_rebuilds: false | ||
| branch_configuration: main | ||
| default_branch: main | ||
| repository: elastic/kibana | ||
| pipeline_file: .buildkite/scripts/pipelines/trigger_version_dependent_jobs/pipeline.sh | ||
| skip_intermediate_builds: false | ||
| provider_settings: | ||
| prefix_pull_request_fork_branch_names: false | ||
| skip_pull_request_builds_for_existing_commits: true | ||
| trigger_mode: none | ||
| teams: | ||
| kibana-operations: | ||
| access_level: MANAGE_BUILD_AND_READ | ||
| appex-qa: | ||
| access_level: MANAGE_BUILD_AND_READ | ||
| kibana-tech-leads: | ||
| access_level: MANAGE_BUILD_AND_READ | ||
| everyone: | ||
| access_level: BUILD_AND_READ | ||
| schedules: | ||
| Trigger ES forward compatibility tests: | ||
| cronline: 0 5 * * * | ||
| message: Trigger ES forward compatibility tests | ||
| env: | ||
| TRIGGER_PIPELINE_SET: es-forward | ||
| Trigger artifact staging builds: | ||
| cronline: 0 7 * * * America/New_York | ||
| message: Trigger artifact staging builds | ||
| env: | ||
| TRIGGER_PIPELINE_SET: artifacts-staging | ||
| MESSAGE: Daily staging build | ||
| Trigger artifact snapshot builds: | ||
| cronline: 0 7 * * * America/New_York | ||
| message: Trigger artifact snapshot builds | ||
| env: | ||
| TRIGGER_PIPELINE_SET: artifacts-snapshot | ||
| MESSAGE: Daily snapshot build | ||
| Run kibana-artifacts-trigger: | ||
| cronline: 0 */2 * * * America/New_York | ||
| message: Trigger 'kibana-artifacts-trigger' | ||
| env: | ||
| TRIGGER_PIPELINE_SET: artifacts-trigger | ||
| MESSAGE: Daily build |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,45 @@ | ||
| /* | ||
| * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
| * or more contributor license agreements. Licensed under the Elastic License | ||
| * 2.0 and the Server Side Public License, v 1; you may not use this file except | ||
| * in compliance with, at your election, the Elastic License 2.0 or the Server | ||
| * Side Public License, v 1. | ||
| */ | ||
|
|
||
| /* eslint-disable @typescript-eslint/no-unused-expressions */ | ||
|
|
||
| import { expect } from 'chai'; | ||
| import { getKibanaDir, getVersionsFile } from './utils'; | ||
| import fs from 'fs'; | ||
|
|
||
| // TODO: replace mocha with jest, and write tests that mock FS | ||
|
|
||
| describe('getKibanaDir', () => { | ||
| it('should return the kibana directory', () => { | ||
| const kibanaDir = getKibanaDir(); | ||
|
|
||
| expect(kibanaDir).to.be.ok; | ||
| expect(fs.existsSync(kibanaDir)).to.be.true; | ||
| }); | ||
| }); | ||
|
|
||
| describe('getVersionsFile', () => { | ||
| it('should return the versions file', () => { | ||
| const versionsFile = getVersionsFile(); | ||
|
|
||
| expect(versionsFile).to.be.ok; | ||
| expect(versionsFile.versions).to.be.an('array'); | ||
| }); | ||
|
|
||
| it('should correctly find prevMajor and prevMinor versions', () => { | ||
| const versionsFile = getVersionsFile(); | ||
|
|
||
| expect(versionsFile.prevMajors).to.be.an('array'); | ||
| expect(versionsFile.prevMajors.length).to.eql(1); | ||
| expect(versionsFile.prevMajors[0].branch).to.eql('7.17'); | ||
|
|
||
| expect(versionsFile.prevMinors).to.be.an('array'); | ||
| }); | ||
|
|
||
| // TODO: write more tests with mocking... | ||
| }); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
5 changes: 5 additions & 0 deletions
5
.buildkite/scripts/pipelines/trigger_version_dependent_jobs/pipeline.sh
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| #!/bin/bash | ||
|
|
||
| set -euo pipefail | ||
|
|
||
| ts-node .buildkite/scripts/pipelines/trigger_version_dependent_jobs/pipeline.ts |
Oops, something went wrong.