[BK] Migrate es-forward (+add versions.json dependent triggering) (el…

…astic#184018)

## Goal
We'd like to introduce a way to run pipelines that have a dependency on
the currently active branch set (managed in
[versions.json](./versions.json)).

With this, we'd like to migrate over the `es-forward` pipelines
(currently:
[this](https://buildkite.com/elastic/kibana-7-dot-17-es-8-dot-15-forward-compatibility),
and
[this](https://buildkite.com/elastic/kibana-7-dot-17-es-8-dot-14-forward-compatibility))
to the new buildkite infra.

## Summary
This PR introduces a new pipeline:
https://buildkite.com/elastic/kibana-trigger-version-dependent-jobs
(through
[trigger-version-dependent-jobs.yml](.buildkite/pipeline-resource-definitions/trigger-version-dependent-jobs.yml)).

The purpose of this new pipeline is to take the name of a "pipelineSet"
that refers to a pipeline, and based on the `versions.json` file, work
out what are the branches on which the referred pipeline should be
triggered.

### Example: `Trigger ES forward compatibility tests`
- a scheduled run on
[kibana-trigger-version-dependent-jobs](https://buildkite.com/elastic/kibana-trigger-version-dependent-jobs)
with the env var `TRIGGER_PIPELINE_SET=es-forward` runs
- the pipeline implementation for
`kibana-trigger-version-dependent-jobs` works out (looking at
`versions.json`), that the `es-forward` set should trigger
https://buildkite.com/elastic/kibana-es-forward (doesn't exist prior to
the PR) for (7.17+8.14) and (7.17+8.15)
- the pipeline implementation uploads two trigger steps, running
https://buildkite.com/elastic/kibana-es-forward in two instances with
the relevant parameterization.

Since the trigger parameters are derived from the `versions.json` file,
if we move on and close `8.14`, and open up `8.16`, this will follow,
without having to update the pipeline resources or schedules.

## Changes
- 2 pipelines created:
[trigger-version-dependent-jobs.yml](.buildkite/pipeline-resource-definitions/trigger-version-dependent-jobs.yml),
[kibana-es-forward.yml](.buildkite/pipeline-resource-definitions/kibana-es-forward.yml)
   - [x] add kibana-es-forward.yml
 - implementation for `trigger-version-dependent-jobs` added
- branch configuration removed from pipelines (kibana-artifacts-staging,
kibana-artifacts-snapshot, kibana-artifacts-trigger)
 - added a script for checking RREs validity (moved a few files)

## Verification
I've used the migration staging pipeline (*) to run this:
-
https://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/130
   - Env: `TRIGGER_PIPELINE_SET="artifacts-trigger"`
- Result:
[(success):](https://buildkite.com/elastic/kibana-artifacts-trigger/builds/10806)
it triggered for 8.14 only (as expected)
-
https://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/131
   - Env: `TRIGGER_PIPELINE_SET="es-forward"`
- Result: (success): it generated 2 trigger steps, but since the
es-forward pipeline doesn't exist, the upload step failed
-
https://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/132
   - Env: `TRIGGER_PIPELINE_SET="artifacts-snapshot"`
- Result: (success): it triggered jobs for all 3 open branches
(main/8.14/7.17)
-
https://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/134
   - Env: `TRIGGER_PIPELINE_SET="artifacts-staging"`
   - Result: (success): it triggered 8.14 / 7.14, but not for main

(*note: this migration staging pipeline will come in handy even after
the migration, to stage newly created pipelines without creating the
resource up-front)

(cherry picked from commit ea25099)

# Conflicts:
#	.buildkite/pipeline-resource-definitions/_templates/_new_pipeline.yml
#	.buildkite/pipeline-resource-definitions/kibana-artifacts-snapshot.yml
#	.buildkite/pipeline-resource-definitions/kibana-artifacts-staging.yml
#	.buildkite/pipeline-resource-definitions/kibana-artifacts-trigger.yml
#	.buildkite/pipeline-resource-definitions/locations.yml
#	.buildkite/pipeline-resource-definitions/scripts/fix-location-collection.ts

.buildkite/pipeline-resource-definitions/README.md

@@ -0,0 +1,28 @@
+# Buildkite pipeline resource definitions
+
+## Overview
+The pipeline resources are "RRE" (real resource entities) that are used to create/maintain buildkite pipelines.
+
+The resources described in these files are parsed and loaded to Backstage (https://backstage.elastic.dev).
+From there, [Terrazzo](https://buildkite.com/elastic/terrazzo/) is generating and updating the buildkite pipelines.
+
+These pipelines are referenced indirectly through the root's [`catalog-info.yaml`](../../catalog-info.yaml) file in order to reduce bloat in the main resources file.
+There's a location file that collects files defined in this folder ([locations.yml](locations.yml)), this file needs to be updated in order to keep track of local files.
+
+Available parameters and further help can be found here: https://docs.elastic.dev/ci/getting-started-with-buildkite-at-elastic
+
+## Creating a new pipeline resource definition
+The easiest way to create a new pipeline is either by copying and editing a similar pipeline, 
+or by copying a blank template (see [_new_pipeline.yml](_templates/_new_pipeline.yml)) and editing that.
+
+You can validate your pipeline's structural integrity, and it's conformity to baseline rules by running the following command:
+```bash
+.buildkite/pipeline-resource-definitions/scripts/validate-pipeline-definition.sh <path_to_your_pipeline_file>
+```
+
+Once you've added the file, you should update the [locations.yml](locations.yml) file to include the new pipeline, or run the following command to update it:
+```bash
+.buildkite/pipeline-resource-definitions/scripts/fix-location-collection.ts
+```
+
+Add your pipeline implementation, commit & push & merge. The pipeline resource will appear in Backstage within minutes, then the pipeline will be added to Buildkite within ~10 minutes.

.buildkite/pipeline-resource-definitions/kibana-es-forward-testing.yml

@@ -0,0 +1,41 @@
+# yaml-language-server: $schema=https://gist.githubusercontent.com/elasticmachine/988b80dae436cafea07d9a4a460a011d/raw/rre.schema.json
+apiVersion: backstage.io/v1alpha1
+kind: Resource
+metadata:
+  name: bk-kibana-es-forward-compatibility-testing
+  description: Forward compatibility testing between Kibana 7.17 and ES 8+
+  links:
+    - url: 'https://buildkite.com/elastic/kibana-es-forward-compatibility-testing'
+      title: Pipeline link
+spec:
+  type: buildkite-pipeline
+  system: buildkite
+  owner: 'group:kibana-operations'
+  implementation:
+    apiVersion: buildkite.elastic.dev/v1
+    kind: Pipeline
+    metadata:
+      name: kibana / ES Forward Compatibility Testing
+      description: Forward compatibility testing between Kibana 7.17 and ES 8+
+    spec:
+      env:
+        SLACK_NOTIFICATIONS_CHANNEL: '#kibana-operations-alerts'
+        ELASTIC_SLACK_NOTIFICATIONS_ENABLED: 'true'
+      allow_rebuilds: false
+      branch_configuration: main
+      default_branch: main
+      repository: elastic/kibana
+      pipeline_file: .buildkite/pipelines/es_forward.yml # Note: this file exists in 7.17 only
+      skip_intermediate_builds: false
+      provider_settings:
+        prefix_pull_request_fork_branch_names: false
+        trigger_mode: none
+      teams:
+        kibana-operations:
+          access_level: MANAGE_BUILD_AND_READ
+        appex-qa:
+          access_level: MANAGE_BUILD_AND_READ
+        kibana-tech-leads:
+          access_level: MANAGE_BUILD_AND_READ
+        everyone:
+          access_level: BUILD_AND_READ

.buildkite/pipeline-resource-definitions/scripts/validate-pipeline-definition.sh

@@ -0,0 +1,28 @@
+#!/usr/bin/env bash
+
+# This script is used to validate a single RRE for a pipeline definition.
+
+TARGET_FILE=$1
+
+if [ -z "$TARGET_FILE" ]; then
+  echo "Usage: $0 <path_to_your_pipeline_file>"
+  exit 1
+fi
+
+echo "Validating $TARGET_FILE..."
+ABSOLUTE_PATH=$(realpath "$TARGET_FILE")
+FILE_NAME=$(basename "$ABSOLUTE_PATH")
+FOLDER_NAME=$(dirname "$ABSOLUTE_PATH")
+
+docker run -it \
+ --mount type=bind,source="$FOLDER_NAME",target=/home/app/ \
+  docker.elastic.co/ci-agent-images/pipelib \
+  rre validate --backstage-entity-aware "/home/app/$FILE_NAME"
+
+if [ $? -ne 0 ]; then
+  echo "$FILE_NAME invalid ❌"
+  exit 1
+else
+  echo "$FILE_NAME valid ✅"
+  exit 0
+fi

.buildkite/pipeline-resource-definitions/trigger-version-dependent-jobs.yml

@@ -0,0 +1,71 @@
+###
+# For more information on authoring pipeline definitions,
+# follow the guides at https://docs.elastic.dev/ci/getting-started-with-buildkite-at-elastic
+###
+# yaml-language-server: $schema=https://gist.githubusercontent.com/elasticmachine/988b80dae436cafea07d9a4a460a011d/raw/rre.schema.json
+apiVersion: backstage.io/v1alpha1
+kind: Resource
+metadata:
+  name: bk-kibana-trigger-version-dependent-jobs
+  description: 'Trigger version-dependent jobs'
+  links:
+    - url: 'https://buildkite.com/elastic/kibana-trigger-version-dependent-jobs'
+      title: Pipeline link
+spec:
+  type: buildkite-pipeline
+  system: buildkite
+  owner: 'group:kibana-operations'
+  implementation:
+    apiVersion: buildkite.elastic.dev/v1
+    kind: Pipeline
+    metadata:
+      name: kibana / trigger version-dependent jobs
+      description: 'Trigger version-dependent jobs'
+    spec:
+      env:
+        SLACK_NOTIFICATIONS_CHANNEL: '#kibana-operations-alerts'
+        ELASTIC_SLACK_NOTIFICATIONS_ENABLED: 'true'
+
+      allow_rebuilds: false
+      branch_configuration: main
+      default_branch: main
+      repository: elastic/kibana
+      pipeline_file: .buildkite/scripts/pipelines/trigger_version_dependent_jobs/pipeline.sh
+      skip_intermediate_builds: false
+      provider_settings:
+        prefix_pull_request_fork_branch_names: false
+        skip_pull_request_builds_for_existing_commits: true
+        trigger_mode: none
+      teams:
+        kibana-operations:
+          access_level: MANAGE_BUILD_AND_READ
+        appex-qa:
+          access_level: MANAGE_BUILD_AND_READ
+        kibana-tech-leads:
+          access_level: MANAGE_BUILD_AND_READ
+        everyone:
+          access_level: BUILD_AND_READ
+      schedules:
+        Trigger ES forward compatibility tests:
+          cronline: 0 5 * * *
+          message: Trigger ES forward compatibility tests
+          env:
+            TRIGGER_PIPELINE_SET: es-forward
+        Trigger artifact staging builds:
+          cronline: 0 7 * * * America/New_York
+          message: Trigger artifact staging builds
+          env:
+            TRIGGER_PIPELINE_SET: artifacts-staging
+            MESSAGE: Daily staging build
+        Trigger artifact snapshot builds:
+          cronline: 0 7 * * * America/New_York
+          message: Trigger artifact snapshot builds
+          env:
+              TRIGGER_PIPELINE_SET: artifacts-snapshot
+              MESSAGE: Daily snapshot build
+        Run kibana-artifacts-trigger:
+          cronline: 0 */2 * * * America/New_York
+          message: Trigger 'kibana-artifacts-trigger'
+          env:
+            TRIGGER_PIPELINE_SET: artifacts-trigger
+            MESSAGE: Daily build

.buildkite/pipeline-utils/utils.test.ts

@@ -0,0 +1,45 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+/* eslint-disable @typescript-eslint/no-unused-expressions */
+
+import { expect } from 'chai';
+import { getKibanaDir, getVersionsFile } from './utils';
+import fs from 'fs';
+
+// TODO: replace mocha with jest, and write tests that mock FS
+
+describe('getKibanaDir', () => {
+  it('should return the kibana directory', () => {
+    const kibanaDir = getKibanaDir();
+
+    expect(kibanaDir).to.be.ok;
+    expect(fs.existsSync(kibanaDir)).to.be.true;
+  });
+});
+
+describe('getVersionsFile', () => {
+  it('should return the versions file', () => {
+    const versionsFile = getVersionsFile();
+
+    expect(versionsFile).to.be.ok;
+    expect(versionsFile.versions).to.be.an('array');
+  });
+
+  it('should correctly find prevMajor and prevMinor versions', () => {
+    const versionsFile = getVersionsFile();
+
+    expect(versionsFile.prevMajors).to.be.an('array');
+    expect(versionsFile.prevMajors.length).to.eql(1);
+    expect(versionsFile.prevMajors[0].branch).to.eql('7.17');
+
+    expect(versionsFile.prevMinors).to.be.an('array');
+  });
+
+  // TODO: write more tests with mocking...
+});

.buildkite/pipeline-utils/utils.ts

@@ -7,6 +7,8 @@
  */
 
 import { execSync } from 'child_process';
+import fs from 'fs';
+import path from 'path';
 
 const getKibanaDir = (() => {
   let kibanaDir: string | undefined;
@@ -21,4 +23,42 @@ const getKibanaDir = (() => {
   };
 })();
 
-export { getKibanaDir };
+export interface Version {
+  branch: string;
+  version: string;
+}
+export interface VersionsFile {
+  versions: Array<
+    {
+      previousMajor?: boolean;
+      previousMinor?: boolean;
+      currentMajor?: boolean;
+      currentMinor?: boolean;
+    } & Version
+  >;
+}
+const getVersionsFile = (() => {
+  let versions: VersionsFile & {
+    prevMinors: Version[];
+    prevMajors: Version[];
+    current: Version;
+  };
+  const versionsFileName = 'versions.json';
+  try {
+    const versionsJSON = JSON.parse(
+      fs.readFileSync(path.join(getKibanaDir(), versionsFileName)).toString()
+    );
+    versions = {
+      versions: versionsJSON.versions,
+      prevMinors: versionsJSON.versions.filter((v: any) => v.previousMinor),
+      prevMajors: versionsJSON.versions.filter((v: any) => v.previousMajor),
+      current: versionsJSON.versions.find((v: any) => v.currentMajor && v.currentMinor),
+    };
+  } catch (error) {
+    throw new Error(`Failed to read ${versionsFileName}: ${error}`);
+  }
+
+  return () => versions;
+})();
+
+export { getKibanaDir, getVersionsFile };

.buildkite/scripts/common/util.sh

@@ -172,3 +172,9 @@ npm_install_global() {
 download_artifact() {
   retry 3 1 timeout 3m buildkite-agent artifact download "$@"
 }
+
+print_if_dry_run() {
+  if [[ "${DRY_RUN:-}" =~ ^(1|true)$ ]]; then
+    echo "DRY_RUN is enabled."
+  fi
+}

.buildkite/scripts/pipelines/trigger_version_dependent_jobs/pipeline.sh

@@ -0,0 +1,5 @@
+#!/bin/bash
+
+set -euo pipefail
+
+ts-node .buildkite/scripts/pipelines/trigger_version_dependent_jobs/pipeline.ts

.buildkite/scripts/pipelines/trigger_version_dependent_jobs/pipeline.test.ts

@@ -0,0 +1,77 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+/* eslint-disable @typescript-eslint/no-unused-expressions */
+
+import { getVersionsFile } from '#pipeline-utils';
+import { expect } from 'chai';
+
+import {
+  getArtifactBuildTriggers,
+  getArtifactSnapshotPipelineTriggers,
+  getESForwardPipelineTriggers,
+  getArtifactStagingPipelineTriggers,
+} from './pipeline';
+
+const versionsFile = getVersionsFile();
+
+describe('pipeline trigger combinations', () => {
+  it('should trigger the correct pipelines for "es-forward"', () => {
+    const esForwardTriggers = getESForwardPipelineTriggers();
+    // tests 7.17 against 8.x versions
+    const targets = versionsFile.versions.filter((v) => v.currentMajor === true);
+
+    expect(esForwardTriggers.length).to.eql(targets.length);
+
+    expect(esForwardTriggers.every((trigger) => trigger.build?.branch === '7.17')).to.be.true;
+
+    const targetedManifests = esForwardTriggers.map((t) => t.build?.env?.ES_SNAPSHOT_MANIFEST);
+    targets.forEach((t) =>
+      expect(targetedManifests).to.include(
+        `https://storage.googleapis.com/kibana-ci-es-snapshots-daily/${t.version}/manifest-latest-verified.json`
+      )
+    );
+  });
+
+  it('should trigger the correct pipelines for "artifacts-snapshot"', () => {
+    const snapshotTriggers = getArtifactSnapshotPipelineTriggers();
+    // triggers for all open branches
+    const branches = versionsFile.versions.map((v) => v.branch);
+
+    expect(snapshotTriggers.length).to.eql(branches.length);
+
+    branches.forEach((b) => {
+      expect(snapshotTriggers.some((trigger) => trigger.build?.branch === b)).to.be.true;
+    });
+  });
+
+  it('should trigger the correct pipelines for "artifacts-trigger"', () => {
+    const triggerTriggers = getArtifactBuildTriggers();
+    // all currentMajor+prevMinor branches
+    const branches = versionsFile.versions
+      .filter((v) => v.currentMajor === true && v.previousMinor === true)
+      .map((v) => v.branch);
+
+    expect(triggerTriggers.length).to.eql(branches.length);
+    branches.forEach((b) => {
+      expect(triggerTriggers.some((trigger) => trigger.build?.branch === b)).to.be.true;
+    });
+  });
+
+  it('should trigger the correct pipelines for "artifacts-staging"', () => {
+    const stagingTriggers = getArtifactStagingPipelineTriggers();
+    // all branches that are not currentMajor+currentMinor
+    const branches = versionsFile.versions
+      .filter((v) => !v.currentMajor || !v.currentMinor)
+      .map((v) => v.branch);
+
+    expect(stagingTriggers.length).to.eql(branches.length);
+    branches.forEach((b) => {
+      expect(stagingTriggers.some((trigger) => trigger.build?.branch === b)).to.be.true;
+    });
+  });
+});