-
Notifications
You must be signed in to change notification settings - Fork 8.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BK] Migrate es-forward (+add versions.json dependent triggering) #184018
Merged
delanni
merged 23 commits into
elastic:main
from
delanni:add-pipeline-parametric-triggers
Jul 10, 2024
Merged
Changes from 18 commits
Commits
Show all changes
23 commits
Select commit
Hold shift + click to select a range
222cf87
chore: move files around, add README
delanni 07ac9cd
chore(CI): add pipeline for triggering version-dependent jobs
delanni 6442a19
chore(CI): replace version-dependent pipelines' schedules with schedu…
delanni ce4916f
Merge branch 'main' into add-pipeline-parametric-triggers
delanni 32a347d
Merge branch 'main' into add-pipeline-parametric-triggers
delanni 84cc679
Merge branch 'main' into add-pipeline-parametric-triggers
delanni dbba3c7
Merge branch 'main' into add-pipeline-parametric-triggers
delanni d576e62
Merge branch 'main' into add-pipeline-parametric-triggers
delanni 4cb0c47
adjust messaging
delanni 5066bb2
chore: move schedule from kibana-artifacts-trigger to a trigger sched…
delanni 98cbc48
Merge branch 'main' into add-pipeline-parametric-triggers
delanni c1490b3
chore: add pipeline for es-forward testing
delanni 3bf2b0f
chore: enhance artifacts publish with DRY_RUN forwarding
delanni 7ceae82
skip cloud testing if DRY_RUN is on
delanni 53521bd
add missing break
delanni f683a87
forward DRY_RUN if set
delanni 44fd428
Merge branch 'main' into add-pipeline-parametric-triggers
delanni 32ff31b
rename backstage entity
delanni 94b98bc
change staging trigger filter to be based on branches
delanni 91befca
Merge branch 'main' into add-pipeline-parametric-triggers
delanni 56b57bf
extract versions file, enrich model, simplify branch selection for pi…
delanni 440381a
add tests to ensure the triggered sets are the same
delanni fb35167
Merge branch 'main' into add-pipeline-parametric-triggers
delanni File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,28 @@ | ||
# Buildkite pipeline resource definitions | ||
|
||
## Overview | ||
The pipeline resources are "RRE" (real resource entities) that are used to create/maintain buildkite pipelines. | ||
|
||
The resources described in these files are parsed and loaded to Backstage (https://backstage.elastic.dev). | ||
From there, [Terrazzo](https://buildkite.com/elastic/terrazzo/) is generating and updating the buildkite pipelines. | ||
|
||
These pipelines are referenced indirectly through the root's [`catalog-info.yaml`](../../catalog-info.yaml) file in order to reduce bloat in the main resources file. | ||
There's a location file that collects files defined in this folder ([locations.yml](locations.yml)), this file needs to be updated in order to keep track of local files. | ||
|
||
Available parameters and further help can be found here: https://docs.elastic.dev/ci/getting-started-with-buildkite-at-elastic | ||
|
||
## Creating a new pipeline resource definition | ||
The easiest way to create a new pipeline is either by copying and editing a similar pipeline, | ||
or by copying a blank template (see [_new_pipeline.yml](_templates/_new_pipeline.yml)) and editing that. | ||
|
||
You can validate your pipeline's structural integrity, and it's conformity to baseline rules by running the following command: | ||
```bash | ||
.buildkite/pipeline-resource-definitions/scripts/validate-pipeline-definition.sh <path_to_your_pipeline_file> | ||
``` | ||
|
||
Once you've added the file, you should update the [locations.yml](locations.yml) file to include the new pipeline, or run the following command to update it: | ||
```bash | ||
.buildkite/pipeline-resource-definitions/scripts/fix-location-collection.ts | ||
``` | ||
|
||
Add your pipeline implementation, commit & push & merge. The pipeline resource will appear in Backstage within minutes, then the pipeline will be added to Buildkite within ~10 minutes. |
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
41 changes: 41 additions & 0 deletions
41
.buildkite/pipeline-resource-definitions/kibana-es-forward-testing.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,41 @@ | ||
# yaml-language-server: $schema=https://gist.githubusercontent.com/elasticmachine/988b80dae436cafea07d9a4a460a011d/raw/rre.schema.json | ||
apiVersion: backstage.io/v1alpha1 | ||
kind: Resource | ||
metadata: | ||
name: bk-kibana-es-forward-compatibility-testing | ||
description: Forward compatibility testing between Kibana 7.17 and ES 8+ | ||
links: | ||
- url: 'https://buildkite.com/elastic/kibana-es-forward-compatibility-testing' | ||
title: Pipeline link | ||
spec: | ||
type: buildkite-pipeline | ||
system: buildkite | ||
owner: 'group:kibana-operations' | ||
implementation: | ||
apiVersion: buildkite.elastic.dev/v1 | ||
kind: Pipeline | ||
metadata: | ||
name: kibana / ES Forward Compatibility Testing | ||
description: Forward compatibility testing between Kibana 7.17 and ES 8+ | ||
spec: | ||
env: | ||
SLACK_NOTIFICATIONS_CHANNEL: '#kibana-operations-alerts' | ||
ELASTIC_SLACK_NOTIFICATIONS_ENABLED: 'true' | ||
allow_rebuilds: false | ||
branch_configuration: main | ||
default_branch: main | ||
repository: elastic/kibana | ||
pipeline_file: .buildkite/pipelines/es_forward.yml # Note: this file exists in 7.17 only | ||
skip_intermediate_builds: false | ||
provider_settings: | ||
prefix_pull_request_fork_branch_names: false | ||
trigger_mode: none | ||
teams: | ||
kibana-operations: | ||
access_level: MANAGE_BUILD_AND_READ | ||
appex-qa: | ||
access_level: MANAGE_BUILD_AND_READ | ||
kibana-tech-leads: | ||
access_level: MANAGE_BUILD_AND_READ | ||
everyone: | ||
access_level: BUILD_AND_READ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
28 changes: 28 additions & 0 deletions
28
.buildkite/pipeline-resource-definitions/scripts/validate-pipeline-definition.sh
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,28 @@ | ||
#!/usr/bin/env bash | ||
|
||
# This script is used to validate a single RRE for a pipeline definition. | ||
|
||
TARGET_FILE=$1 | ||
|
||
if [ -z "$TARGET_FILE" ]; then | ||
echo "Usage: $0 <path_to_your_pipeline_file>" | ||
exit 1 | ||
fi | ||
|
||
echo "Validating $TARGET_FILE..." | ||
ABSOLUTE_PATH=$(realpath "$TARGET_FILE") | ||
FILE_NAME=$(basename "$ABSOLUTE_PATH") | ||
FOLDER_NAME=$(dirname "$ABSOLUTE_PATH") | ||
|
||
docker run -it \ | ||
--mount type=bind,source="$FOLDER_NAME",target=/home/app/ \ | ||
docker.elastic.co/ci-agent-images/pipelib \ | ||
rre validate --backstage-entity-aware "/home/app/$FILE_NAME" | ||
|
||
if [ $? -ne 0 ]; then | ||
echo "$FILE_NAME invalid ❌" | ||
exit 1 | ||
else | ||
echo "$FILE_NAME valid ✅" | ||
exit 0 | ||
fi |
71 changes: 71 additions & 0 deletions
71
.buildkite/pipeline-resource-definitions/trigger-version-dependent-jobs.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,71 @@ | ||
### | ||
# For more information on authoring pipeline definitions, | ||
# follow the guides at https://docs.elastic.dev/ci/getting-started-with-buildkite-at-elastic | ||
### | ||
# yaml-language-server: $schema=https://gist.githubusercontent.com/elasticmachine/988b80dae436cafea07d9a4a460a011d/raw/rre.schema.json | ||
apiVersion: backstage.io/v1alpha1 | ||
kind: Resource | ||
metadata: | ||
name: bk-kibana-trigger-version-dependent-jobs | ||
description: 'Trigger version-dependent jobs' | ||
links: | ||
- url: 'https://buildkite.com/elastic/kibana-trigger-version-dependent-jobs' | ||
title: Pipeline link | ||
spec: | ||
type: buildkite-pipeline | ||
system: buildkite | ||
owner: 'group:kibana-operations' | ||
implementation: | ||
apiVersion: buildkite.elastic.dev/v1 | ||
kind: Pipeline | ||
metadata: | ||
name: kibana / trigger version-dependent jobs | ||
description: 'Trigger version-dependent jobs' | ||
spec: | ||
env: | ||
SLACK_NOTIFICATIONS_CHANNEL: '#kibana-operations-alerts' | ||
ELASTIC_SLACK_NOTIFICATIONS_ENABLED: 'true' | ||
|
||
allow_rebuilds: false | ||
branch_configuration: main | ||
default_branch: main | ||
repository: elastic/kibana | ||
pipeline_file: .buildkite/scripts/pipelines/trigger_version_dependent_jobs/pipeline.sh | ||
skip_intermediate_builds: false | ||
provider_settings: | ||
prefix_pull_request_fork_branch_names: false | ||
skip_pull_request_builds_for_existing_commits: true | ||
trigger_mode: none | ||
teams: | ||
kibana-operations: | ||
access_level: MANAGE_BUILD_AND_READ | ||
appex-qa: | ||
access_level: MANAGE_BUILD_AND_READ | ||
kibana-tech-leads: | ||
access_level: MANAGE_BUILD_AND_READ | ||
everyone: | ||
access_level: BUILD_AND_READ | ||
schedules: | ||
Trigger ES forward compatibility tests: | ||
cronline: 0 5 * * * | ||
message: Trigger ES forward compatibility tests | ||
env: | ||
TRIGGER_PIPELINE_SET: es-forward | ||
Trigger artifact staging builds: | ||
cronline: 0 7 * * * America/New_York | ||
message: Trigger artifact staging builds | ||
env: | ||
TRIGGER_PIPELINE_SET: artifacts-staging | ||
MESSAGE: Daily staging build | ||
Trigger artifact snapshot builds: | ||
cronline: 0 7 * * * America/New_York | ||
message: Trigger artifact snapshot builds | ||
env: | ||
TRIGGER_PIPELINE_SET: artifacts-snapshot | ||
MESSAGE: Daily snapshot build | ||
Run kibana-artifacts-trigger: | ||
cronline: 0 */2 * * * America/New_York | ||
message: Trigger 'kibana-artifacts-trigger' | ||
env: | ||
TRIGGER_PIPELINE_SET: artifacts-trigger | ||
MESSAGE: Daily build |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
5 changes: 5 additions & 0 deletions
5
.buildkite/scripts/pipelines/trigger_version_dependent_jobs/pipeline.sh
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
#!/bin/bash | ||
|
||
set -euo pipefail | ||
|
||
ts-node .buildkite/scripts/pipelines/trigger_version_dependent_jobs/pipeline.ts |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
needs to be removed, or replaced with a generic
*.*
+main
type expression, because we don't want to keep updating this in every version switch