[CI][FIPS] Reconfigure pipeline to use Staging GovCloud/FRH ESS environment

[ycombinator]

What does this PR do?

This PR reconfigures the FIPS Buildkite pipeline to a) point it to the Staging GovCloud/FRH ESS environment and b) use an ESS API key from that environment for spinning up deployments.

Why is it important?

To run FIPS-related tests against the officially-configured FedRamp High (FRH) ESS environment.

[mergify]

This pull request does not have a backport label. Could you fix it @ycombinator? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label that automatically backports to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b bk-fips-redirect upstream/bk-fips-redirect
git merge upstream/main
git push upstream bk-fips-redirect

[ycombinator]

@pkoutsovasilis I could use your input here on how best to resolve the CI failure on this PR, since it pertains to the contents of the .package-version file.

The "Start ESS stack for FIPS integration tests" step is failing in CI like so:

│ Error: failed creating deployment
--
  | │
  | │   with ec_deployment.integration-testing,
  | │   on deployment.tf line 100, in resource "ec_deployment" "integration-testing":
  | │  100: resource "ec_deployment" "integration-testing" {
  | │
  | │ api error: 1 error occurred:
  | │ 	* clusters.cluster_plan_change_prohibited: The requested cluster configuration change is not permitted. Value of field 'elasticsearch.docker_image' cannot
  | │ be changed to [docker.elastic.co/cloud-release/elasticsearch-cloud-ess-fips:9.2.0-0c2a1cac-SNAPSHOT]. (resources.elasticsearch[0].elasticsearch.docker_image)
  | │

This is happening because there is no Docker image with this tag (9.2.0-0c2a1cac-SNAPSHOT) in the Staging GovCloud ESS environment, which is the environment that we run our FIPS tests against. The tag comes from

elastic-agent/.package-version

Line 7 in c1de40f

"stack_build_id": "9.2.0-0c2a1cac-SNAPSHOT"

That field is being read in the terraform file used for creating the ESS deployment over here:

elastic-agent/test_infra/ess/deployment.tf

Line 82 in c1de40f

images_version = coalesce(var.stack_build_id, var.stack_version)

Just for testing, I temporarily hardcoded the tag as 9.2.0-SNAPSHOT in the terraform file and it worked. So I know the issue is isolated to the tag not existing in the environment, and everything else in our configuration is OK.

Do you have any thoughts on how we could accommodate the Staging GovCloud environment in our testing configuration?

Here is one idea I have; let me know if you have a better suggestion:

1. In https://github.com/elastic/elastic-agent/blob/main/.package-version, create a new field, stack_version, which will have the same value as stack_build_id but without the SHA. So, today, the value of this new field would be 9.2.0-SNAPSHOT. The automation that keeps that file updated will set this new field as well.
2. We pass this new field value to the Terraform file in addition to passing stack_build_id today.
3. In the Terraform file, we introduce a new boolean variable fips. The FIPS Buildkite pipeline will pass this value as true to the Terraform file.
4. In the Terraform file, if fips is set to true, we use the value of stack_version otherwise we use the value of stack_build_id, to create the image tag.

Another solution, similar to the above, but needing more automation would be to store something like a frh_stack_build_id field in the .package-version file and have automation set that based on trying to find the stack build ID actually available in the ESS Staging GovCloud environment that matches stack_build_id the closest.

Keep in mind that we have a deadline to finish FIPS testing by the end of this week so we may need to go with a less-ideal solution for now and then revisit it to make it better.

[pkoutsovasilis]

Hey @ycombinator 👋 I do like the following proposal

In https://github.com/elastic/elastic-agent/blob/main/.package-version, create a new field, stack_version, which will have the same value as stack_build_id but without the SHA. So, today, the value of this new field would be 9.2.0-SNAPSHOT. The automation that keeps that file updated will set this new field as well.
We pass this new field value to the Terraform file in addition to passing stack_build_id today.
In the Terraform file, we introduce a new boolean variable fips. The FIPS Buildkite pipeline will pass this value as true to the Terraform file.
In the Terraform file, if fips is set to true, we use the value of stack_version otherwise we use the value of stack_build_id, to create the image tag.

but let's move fast to get this up and running and we figure the details of how .package-version can be utilised here afterwards. What do you think of the following?

1. here do a change in the fashion of if FIPS=TRUE then STACK_BUILD_ID=""
2. similar to the above here
3. windows you don't care at the moment, but it's similar to the above here

this will result using only the STACK_VERSION and not the STACK_BUILD_ID

[ycombinator]

The "Start ESS stack for FIPS integration tests" step in CI is failing with the error:

ec_deployment.integration-testing: Creating...
--
  | ╷
  | │ Error: failed creating deployment
  | │
  | │   with ec_deployment.integration-testing,
  | │   on deployment.tf line 100, in resource "ec_deployment" "integration-testing":
  | │  100: resource "ec_deployment" "integration-testing" {
  | │
  | │ api error: 1 error occurred:
  | │ 	* clusters.cluster_plan_change_prohibited: The requested cluster configuration change is not permitted. Value of field 'elasticsearch.docker_image' cannot
  | │ be changed to [docker.elastic.co/cloud-release/elasticsearch-cloud-ess-fips:9.2.0-SNAPSHOT]. (resources.elasticsearch[0].elasticsearch.docker_image)
  | │
  | │
  | ╵
  | ╷
  | │ Error: failed creating deployment
  | │
  | │   with ec_deployment.integration-testing,
  | │   on deployment.tf line 100, in resource "ec_deployment" "integration-testing":
  | │  100: resource "ec_deployment" "integration-testing" {

I'm confused because the same image tag succeeded in the previous CI build: https://buildkite.com/elastic/elastic-agent/builds/25276#0198a930-20fb-4c9e-93c3-80392012a236/136-180

[EDIT] I've compared the Terraform execution plans from both CI builds (previous - successful vs. current - failed) and they are identical except for the docker.elastic.co/beats-ci/elastic-agent-cloud-fips image's SHA (which is expected to be different since we're building different commits) and the buildkite_id tag (which is also expected to be different as it's different builds).

[EDIT] I've also run terraform apply manually with the same terraform configuration that's being used in CI like so:

TF_VAR_docker_images_name_suffix=-fips \
TF_VAR_stack_version=9.2.0-SNAPSHOT \
TF_VAR_integration_server_docker_image="docker.elastic.co/cloud-release/elastic-agent-cloud-fips:9.2.0-SNAPSHOT" \
TF_VAR_deployment_template_id="aws-general-purpose" \
TF_VAR_ess_region="us-gov-east-1" \
EC_API_KEY="XXXXXXX" \
EC_ENDPOINT="https://api.staging.elastic-gov.com" \
terraform apply

I've run this three times so far, one after the other (with a terraform destroy in between). The first attempt succeeded, the second attempt failed with the same error that we're seeing in CI, and the third attempt succeeded as well. So I'm starting to think something about the Staging GovCloud environment might be flaky?

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 83b3d82

History

• 💔 Build #25278 failed ea6fc40
• 💔 Build #25257 failed 7c721f6
• 💔 Build #25251 failed 3212ff2
• 💔 Build #25239 failed 4914875
• 💔 Build #25231 failed 20af3e9

cc @ycombinator

[oakrizan]

lgtm for CI part