Add fips mode to the Agent

[Kaderinho]

WIP, still a rough draft because I have some questions

Questions

• Do we want to run a nightly build + tests with upstream GO. If yes, how can we do that
• Do we need some naming convention to differentiate package (ms go vs upstream go) ?
• How can we cache openssl omnibus software ?
• Should we mention the toolchain somewhere in the package or in the agent version subcommand ?
• Should we test most of the code with FIPS enabled or only a small subset and consider FIPS disabled as the main mode
• Who should own what ?
• Do we need to add anything on the delivery side before merging this PR or is the current pipeline enough ?

TODOs

• Windows
• Option to crash if FIPS is not enabled
• Add e2e tests at the end

[pr-commenter]

Regression Detector

Regression Detector Results

Run ID: 8f8f2793-eb24-48cb-b96a-4e5386b84d52 Metrics dashboard Target profiles

Baseline: 22686e5
Comparison: 98faafe

Performance changes are noted in the perf column of each table:

• ✅ = significantly better comparison variant performance
• ❌ = significantly worse comparison variant performance
• ➖ = no significant change in performance

No significant changes in experiment optimization goals

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

There were no significant changes in experiment optimization goals at this confidence level and effect size tolerance.

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI	trials	links
➖	file_tree	memory utilization	+4.08	[+4.00, +4.17]	1	Logs
➖	idle	memory utilization	+2.96	[+2.91, +3.01]	1	Logs
➖	otel_to_otel_logs	ingress throughput	+1.10	[+0.29, +1.91]	1	Logs
➖	pycheck_lots_of_tags	% cpu utilization	+0.45	[-2.03, +2.93]	1	Logs
➖	tcp_dd_logs_filter_exclude	ingress throughput	+0.00	[-0.01, +0.01]	1	Logs
➖	uds_dogstatsd_to_api	ingress throughput	-0.00	[-0.09, +0.09]	1	Logs
➖	uds_dogstatsd_to_api_cpu	% cpu utilization	-0.03	[-0.77, +0.72]	1	Logs
➖	tcp_syslog_to_blackhole	ingress throughput	-0.38	[-0.44, -0.33]	1	Logs
➖	basic_py_check	% cpu utilization	-1.22	[-3.99, +1.55]	1	Logs

Bounds Checks

perf	experiment	bounds_check_name	replicates_passed
✅	idle	memory_usage	10/10

Explanation

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

1. Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.

2. Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.

3. Its configuration does not mark it "erratic".

[pr-commenter]

Gitlab CI Configuration Changes

Modified Jobs

stages (configuration)

  stages:
  - .pre
  - setup
  - maintenance_jobs
  - deps_build
  - deps_fetch
  - source_test
  - source_test_stats
  - software_composition_analysis
  - binary_build
  - package_deps_build
  - kernel_matrix_testing_prepare
  - kernel_matrix_testing_system_probe
  - kernel_matrix_testing_security_agent
  - kernel_matrix_testing_cleanup
  - integration_test
  - benchmarks
  - package_build
  - packaging
  - pkg_metrics
  - kitchen_deploy
  - kitchen_testing
  - container_build
  - container_scan
  - check_deploy
  - dev_container_deploy
  - deploy_containers
  - deploy_packages
  - deploy_cws_instrumentation
  - deploy_dca
+ - check_fips_compliance
  - trigger_release
  - choco_build
  - choco_deploy
  - internal_image_deploy
  - install_script_testing
  - e2e_pre_test
  - e2e
  - e2e_k8s
  - e2e_install_packages
  - kitchen_cleanup
  - functional_test
  - functional_test_cleanup
  - junit_upload
  - internal_kubernetes_deploy
  - post_rc_build
  - check_merge
  - notify
  - .post

variables (configuration)

  variables:
    AGENT_BINARIES_DIR: bin/agent
    AGENT_GITHUB_APP_ID: ci.datadog-agent.platform-github-app-id
    AGENT_GITHUB_INSTALLATION_ID: ci.datadog-agent.platform-github-app-installation-id
    AGENT_GITHUB_KEY: ci.datadog-agent.platform-github-app-key
    AGENT_QA_PROFILE: ci.datadog-agent.agent-qa-profile
    API_KEY_DDDEV: ci.datadog-agent.datadog_api_key
    API_KEY_ORG2: ci.datadog-agent.datadog_api_key_org2
    APP_KEY_ORG2: ci.datadog-agent.datadog_app_key_org2
    ARTIFACT_DOWNLOAD_ATTEMPTS: 2
    BTFHUB_ARCHIVE_BRANCH: main
    BUCKET_BRANCH: dev
    CHANGELOG_COMMIT_SHA: ci.datadog-agent.gitlab_changelog_commit_sha
    CHOCOLATEY_API_KEY: ci.datadog-agent.chocolatey_api_key
    CLANG_LLVM_VER: 12.0.1
    CLUSTER_AGENT_BINARIES_DIR: bin/datadog-cluster-agent
    CLUSTER_AGENT_CLOUDFOUNDRY_BINARIES_DIR: bin/datadog-cluster-agent-cloudfoundry
    CODECOV_TOKEN: ci.datadog-agent.codecov_token
    CWS_INSTRUMENTATION_BINARIES_DIR: bin/cws-instrumentation
-   DATADOG_AGENT_ARMBUILDIMAGES: v44534774-f5cc3e24
+   DATADOG_AGENT_ARMBUILDIMAGES: v45066542-a714c2f1
-   DATADOG_AGENT_ARMBUILDIMAGES_SUFFIX: ''
?                                        ^^
+   DATADOG_AGENT_ARMBUILDIMAGES_SUFFIX: _test_only
?                                        ^^^^^^^^^^
-   DATADOG_AGENT_BTF_GEN_BUILDIMAGES: v44534774-f5cc3e24
?                                         ^^ ^^^^ ^^^^^^^
+   DATADOG_AGENT_BTF_GEN_BUILDIMAGES: v45066542-a714c2f1
?                                        +++++ ^^^^^ ^^ ^
-   DATADOG_AGENT_BTF_GEN_BUILDIMAGES_SUFFIX: ''
?                                             ^^
+   DATADOG_AGENT_BTF_GEN_BUILDIMAGES_SUFFIX: _test_only
?                                             ^^^^^^^^^^
-   DATADOG_AGENT_BUILDIMAGES: v44534774-f5cc3e24
+   DATADOG_AGENT_BUILDIMAGES: v45066542-a714c2f1
-   DATADOG_AGENT_BUILDIMAGES_SUFFIX: ''
?                                     ^^
+   DATADOG_AGENT_BUILDIMAGES_SUFFIX: _test_only
?                                     ^^^^^^^^^^
    DATADOG_AGENT_EMBEDDED_PATH: /opt/datadog-agent/embedded
-   DATADOG_AGENT_SYSPROBE_BUILDIMAGES: v44534774-f5cc3e24
?                                          ^^ ^^^^ ^^^^^^^
+   DATADOG_AGENT_SYSPROBE_BUILDIMAGES: v45066542-a714c2f1
?                                         +++++ ^^^^^ ^^ ^
-   DATADOG_AGENT_SYSPROBE_BUILDIMAGES_SUFFIX: ''
?                                              ^^
+   DATADOG_AGENT_SYSPROBE_BUILDIMAGES_SUFFIX: _test_only
?                                              ^^^^^^^^^^
-   DATADOG_AGENT_WINBUILDIMAGES: v44534774-f5cc3e24
+   DATADOG_AGENT_WINBUILDIMAGES: v45066542-a714c2f1
-   DATADOG_AGENT_WINBUILDIMAGES_SUFFIX: ''
?                                        ^^
+   DATADOG_AGENT_WINBUILDIMAGES_SUFFIX: _test_only
?                                        ^^^^^^^^^^
    DD_AGENT_TESTING_DIR: $CI_PROJECT_DIR/test/kitchen
    DD_PKG_VERSION: latest
    DEB_GPG_KEY: ci.datadog-agent.deb_signing_private_key_${DEB_GPG_KEY_ID}
    DEB_GPG_KEY_ID: c0962c7d
    DEB_GPG_KEY_NAME: Datadog, Inc. APT key
    DEB_RPM_TESTING_BUCKET_BRANCH: testing
    DEB_S3_BUCKET: apt.datad0g.com
    DEB_SIGNING_PASSPHRASE: ci.datadog-agent.deb_signing_key_passphrase_${DEB_GPG_KEY_ID}
    DEB_TESTING_S3_BUCKET: apttesting.datad0g.com
    DOCKER_REGISTRY_LOGIN: ci.datadog-agent.docker_hub_login
    DOCKER_REGISTRY_PWD: ci.datadog-agent.docker_hub_pwd
    DOCKER_REGISTRY_URL: docker.io
    DOGSTATSD_BINARIES_DIR: bin/dogstatsd
    E2E_TESTS_API_KEY: ci.datadog-agent.e2e_tests_api_key
    E2E_TESTS_APP_KEY: ci.datadog-agent.e2e_tests_app_key
    E2E_TESTS_AZURE_CLIENT_ID: ci.datadog-agent.e2e_tests_azure_client_id
    E2E_TESTS_AZURE_CLIENT_SECRET: ci.datadog-agent.e2e_tests_azure_client_secret
    E2E_TESTS_AZURE_SUBSCRIPTION_ID: ci.datadog-agent.e2e_tests_azure_subscription_id
    E2E_TESTS_AZURE_TENANT_ID: ci.datadog-agent.e2e_tests_azure_tenant_id
    E2E_TESTS_GCP_CREDENTIALS: ci.datadog-agent.e2e_tests_gcp_credentials
    E2E_TESTS_RC_KEY: ci.datadog-agent.e2e_tests_rc_key
    EXECUTOR_JOB_SECTION_ATTEMPTS: 2
    FF_KUBERNETES_HONOR_ENTRYPOINT: true
    FF_SCRIPT_SECTIONS: 1
    GENERAL_ARTIFACTS_CACHE_BUCKET_URL: https://dd-agent-omnibus.s3.amazonaws.com
    GET_SOURCES_ATTEMPTS: 2
    GITHUB_PR_COMMENTER_APP_KEY: pr-commenter.github_app_key
    GITHUB_PR_COMMENTER_INSTALLATION_ID: pr-commenter.github_installation_id
    GITHUB_PR_COMMENTER_INTEGRATION_ID: pr-commenter.github_integration_id
    GITLAB_FULL_API_TOKEN: ci.datadog-agent.gitlab_full_api_token
    GITLAB_READ_API_TOKEN: ci.datadog-agent.gitlab_read_api_token
    GITLAB_SCHEDULER_TOKEN: ci.datadog-agent.gitlab_pipelines_scheduler_token
    GO_TEST_SKIP_FLAKE: 'true'
    INSTALL_SCRIPT_API_KEY: ci.agent-linux-install-script.datadog_api_key_2
    INTEGRATION_WHEELS_CACHE_BUCKET: dd-agent-omnibus
    JIRA_READ_API_TOKEN: ci.datadog-agent.jira_read_api_token
    KERNEL_MATRIX_TESTING_ARM_AMI_ID: ami-021f04c00ecfa8590
    KERNEL_MATRIX_TESTING_X86_AMI_ID: ami-0c54d42f8f4180b0c
    KITCHEN_AZURE_CLIENT_ID: ci.datadog-agent.azure_kitchen_client_id
    KITCHEN_AZURE_CLIENT_SECRET: ci.datadog-agent.azure_kitchen_client_secret
    KITCHEN_AZURE_SUBSCRIPTION_ID: ci.datadog-agent.azure_kitchen_subscription_id
    KITCHEN_AZURE_TENANT_ID: ci.datadog-agent.azure_kitchen_tenant_id
    KITCHEN_EC2_SSH_KEY: ci.datadog-agent.aws_ec2_kitchen_ssh_key
    KITCHEN_INFRASTRUCTURE_FLAKES_RETRY: 2
    MACOS_GITHUB_APP_ID: ci.datadog-agent.macos_github_app_id
    MACOS_GITHUB_APP_ID_2: ci.datadog-agent.macos_github_app_id_2
    MACOS_GITHUB_INSTALLATION_ID: ci.datadog-agent.macos_github_installation_id
    MACOS_GITHUB_INSTALLATION_ID_2: ci.datadog-agent.macos_github_installation_id_2
    MACOS_GITHUB_KEY: ci.datadog-agent.macos_github_key_b64
    MACOS_GITHUB_KEY_2: ci.datadog-agent.macos_github_key_b64_2
    MACOS_S3_BUCKET: dd-agent-macostesting
    OMNIBUS_BASE_DIR: /omnibus
    OMNIBUS_GIT_CACHE_DIR: /tmp/omnibus-git-cache
    OMNIBUS_PACKAGE_DIR: $CI_PROJECT_DIR/omnibus/pkg/
    OMNIBUS_PACKAGE_DIR_SUSE: $CI_PROJECT_DIR/omnibus/suse/pkg
    PROCESS_S3_BUCKET: datad0g-process-agent
    RELEASE_VERSION_6: nightly
    RELEASE_VERSION_7: nightly-a7
    RESTORE_CACHE_ATTEMPTS: 2
    RPM_GPG_KEY: ci.datadog-agent.rpm_signing_private_key_${RPM_GPG_KEY_ID}
    RPM_GPG_KEY_ID: b01082d3
    RPM_GPG_KEY_NAME: Datadog, Inc. RPM key
    RPM_S3_BUCKET: yum.datad0g.com
    RPM_SIGNING_PASSPHRASE: ci.datadog-agent.rpm_signing_key_passphrase_${RPM_GPG_KEY_ID}
    RPM_TESTING_S3_BUCKET: yumtesting.datad0g.com
    RUN_E2E_TESTS: auto
    RUN_KMT_TESTS: auto
    RUN_UNIT_TESTS: auto
    S3_ARTIFACTS_URI: s3://dd-ci-artefacts-build-stable/$CI_PROJECT_NAME/$CI_PIPELINE_ID
    S3_CP_CMD: aws s3 cp $S3_CP_OPTIONS
    S3_CP_OPTIONS: --no-progress --region us-east-1 --sse AES256
    S3_DD_AGENT_OMNIBUS_BTFS_URI: s3://dd-agent-omnibus/btfs
    S3_DD_AGENT_OMNIBUS_LLVM_URI: s3://dd-agent-omnibus/llvm
    S3_DSD6_URI: s3://dsd6-staging
    S3_OMNIBUS_CACHE_BUCKET: dd-ci-datadog-agent-omnibus-cache-build-stable
    S3_PERMANENT_ARTIFACTS_URI: s3://dd-ci-persistent-artefacts-build-stable/$CI_PROJECT_NAME
    S3_PROJECT_ARTIFACTS_URI: s3://dd-ci-artefacts-build-stable/$CI_PROJECT_NAME
    S3_RELEASE_ARTIFACTS_URI: s3://dd-release-artifacts/$CI_PROJECT_NAME/$CI_PIPELINE_ID
    S3_RELEASE_INSTALLER_ARTIFACTS_URI: s3://dd-release-artifacts/datadog-installer/$CI_PIPELINE_ID
    S3_SBOM_STORAGE_URI: s3://sbom-root-us1-ddbuild-io/$CI_PROJECT_NAME/$CI_PIPELINE_ID
    SLACK_AGENT_CI_TOKEN: ci.datadog-agent.slack_agent_ci_token
    SMP_ACCOUNT_ID: ci.datadog-agent.single-machine-performance-account-id
    SMP_AGENT_TEAM_ID: ci.datadog-agent.single-machine-performance-agent-team-id
    SMP_API: ci.datadog-agent.single-machine-performance-api
    SMP_BOT_ACCESS_KEY: ci.datadog-agent.single-machine-performance-bot-access-key
    SMP_BOT_ACCESS_KEY_ID: ci.datadog-agent.single-machine-performance-bot-access-key-id
    SSH_KEY: ci.datadog-agent.ssh_key
    SSH_KEY_RSA: ci.datadog-agent.ssh_key_rsa
    SSH_PUBLIC_KEY_RSA: ci.datadog-agent.ssh_public_key_rsa
    STATIC_BINARIES_DIR: bin/static
    SYSTEM_PROBE_BINARIES_DIR: bin/system-probe
    USE_S3_CACHING: --omnibus-s3-cache
    VCPKG_BLOB_SAS_URL: ci.datadog-agent-buildimages.vcpkg_blob_sas_url
    WINDOWS_BUILDS_S3_BUCKET: $WIN_S3_BUCKET/builds
    WINDOWS_TESTING_S3_BUCKET_A6: pipelines/A6/$CI_PIPELINE_ID
    WINDOWS_TESTING_S3_BUCKET_A7: pipelines/A7/$CI_PIPELINE_ID
    WINGET_PAT: ci.datadog-agent.winget_pat
    WIN_S3_BUCKET: dd-agent-mstesting

.agent_build_common

  .agent_build_common:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - echo "About to build for $RELEASE_VERSION"
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - rm -rf $OMNIBUS_PACKAGE_DIR/*
    - tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
    - mkdir -p /tmp/system-probe
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
    - cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
    - chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
-   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
?                                                                              -----------------------
+   - inv -e omnibus.build --fips-mode --release-version "$RELEASE_VERSION" --major-version
?                         ++++++++++++
-     --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING}
+     "$AGENT_MAJOR_VERSION" --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING}
?    +++++++++++++++++++++++
      --skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
      --flavor "$FLAVOR"
    - ls -la $OMNIBUS_PACKAGE_DIR
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
    stage: package_build
    variables:
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi

.kmt_setup_env

  .kmt_setup_env:
    after_script:
    - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $API_KEY_ORG2) || exit $?;
      export DD_API_KEY
    - export AWS_PROFILE=agent-qa-ci
    - FILTER_TEAM="Name=tag:team,Values=ebpf-platform"
    - FILTER_MANAGED="Name=tag:managed-by,Values=pulumi"
    - FILTER_STATE="Name=instance-state-name,Values=running"
    - FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}"
    - FILTER_ARCH="Name=tag:arch,Values=${ARCH}"
    - FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}"
    - FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}"
    - QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId'
    - QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress'
    - mkdir -p $CI_PROJECT_DIR/libvirt/log/$ARCH $CI_PROJECT_DIR/libvirt/xml $CI_PROJECT_DIR/libvirt/qemu
      $CI_PROJECT_DIR/libvirt/dnsmasq
    - INSTANCE_IP=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED
      $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE --output
      text --query $QUERY_PRIVATE_IPS)
    - echo "$ARCH-instance-ip" $INSTANCE_IP
    - ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP"
      "sudo virsh list --name | grep -v -E '^$' | xargs -I '{}' sh -c \"sudo virsh dumpxml
      '{}' > /tmp/ddvm-xml-'{}'.txt\""
    - ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP"
      "sudo virsh list --name | xargs -I '{}' sh -c \"sudo cp /var/log/libvirt/qemu/'{}'.log
      /tmp/qemu-ddvm-'{}'.log && sudo chown 1000:1000 /tmp/qemu-ddvm*\""
    - ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP"
      "mkdir /tmp/dnsmasq && sudo cp /var/lib/libvirt/dnsmasq/* /tmp/dnsmasq/ && sudo
      chown 1000:1000 /tmp/dnsmasq/*"
    - scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/ddvm-*.log"
      $CI_PROJECT_DIR/libvirt/log
    - scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/ddvm-xml-*"
      $CI_PROJECT_DIR/libvirt/xml
    - scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/qemu-ddvm-*.log"
      $CI_PROJECT_DIR/libvirt/qemu
    - scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/dnsmasq/*"
      $CI_PROJECT_DIR/libvirt/dnsmasq
    - "GO_ARCH=$ARCH\nif [ \"${ARCH}\" == \"x86_64\" ]; then\n  GO_ARCH=amd64\nfi\n"
    - cd test/new-e2e && GOOS=linux GOARCH="${GO_ARCH}" go build system-probe/vm-metrics/vm-metrics.go
    - scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE $CI_PROJECT_DIR/test/new-e2e/vm-metrics
      "ubuntu@$INSTANCE_IP:/home/ubuntu/vm-metrics"
    - ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP"
      "/home/ubuntu/vm-metrics -statsd-host=127.0.0.1 -statsd-port=8125 -libvirt-uri=/var/run/libvirt/libvirt-sock-ro
      --tag \"arch:${ARCH}\" --tag \"test-component:${TEST_COMPONENT}\" --tag \"ci-pipeline-id:${CI_PIPELINE_ID}\"
      --daemon -log-file /home/ubuntu/daemon.log"
    - inv -e kmt.tag-ci-job
    artifacts:
      paths:
      - $CI_PROJECT_DIR/stack.output
      - $CI_PROJECT_DIR/libvirt
      - $VMCONFIG_FILE
      when: always
    before_script:
+   - source /root/.bashrc
    - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $API_KEY_ORG2) || exit $?;
      export DD_API_KEY
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - mkdir -p ~/.aws
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_PROFILE >> ~/.aws/config ||
      exit $?
    - export AWS_PROFILE=agent-qa-ci
    - touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SSH_KEY > $AWS_EC2_SSH_KEY_FILE || exit
      $?
    - echo "" >> $AWS_EC2_SSH_KEY_FILE
    - chmod 600 $AWS_EC2_SSH_KEY_FILE
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/test-infra-definitions/runner$TEST_INFRA_DEFINITIONS_BUILDIMAGES_SUFFIX:$TEST_INFRA_DEFINITIONS_BUILDIMAGES
    needs:
    - go_deps
    - go_tools_deps
    script:
    - echo "s3://dd-pulumi-state?region=us-east-1&awssdk=v2&profile=$AWS_PROFILE" >
      $STACK_DIR
    - pulumi login $(cat $STACK_DIR | tr -d '\n')
    - inv -e kmt.gen-config --ci --arch=$ARCH --output-file=$VMCONFIG_FILE --sets=$TEST_SETS
      --vmconfig-template=$TEST_COMPONENT --memory=12288
    - inv -e system-probe.start-microvms --provision-instance --provision-microvms --vmconfig=$VMCONFIG_FILE
      $INSTANCE_TYPE_ARG $AMI_ID_ARG --ssh-key-name=$AWS_EC2_SSH_KEY_NAME --ssh-key-path=$AWS_EC2_SSH_KEY_FILE
      --infra-env=$INFRA_ENV --stack-name=kernel-matrix-testing-${TEST_COMPONENT}-${ARCH}-${CI_PIPELINE_ID}
      --run-agent
    - jq "." $CI_PROJECT_DIR/stack.output
    - pulumi logout
    stage: kernel_matrix_testing_prepare
    tags:
    - arch:amd64
    variables:
      AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key
      AWS_EC2_SSH_KEY_NAME: datadog-agent-ci
      AWS_REGION: us-east-1
      INFRA_ENV: aws/agent-qa
      KITCHEN_EC2_REGION: us-east-1
      KITCHEN_EC2_SG_IDS: sg-019917348cb0eb7e7
      KITCHEN_EC2_SUBNET: subnet-05d7c6b1b5cfea811
      KUBERNETES_MEMORY_LIMIT: 16Gi
      KUBERNETES_MEMORY_REQUEST: 12Gi
      PIPELINE_ID: $CI_PIPELINE_ID
      RESOURCE_TAGS: instance-type:${INSTANCE_TYPE},arch:${ARCH},test-component:${TEST_COMPONENT},git-branch:${CI_COMMIT_REF_NAME}
      STACK_DIR: $CI_PROJECT_DIR/stack.dir
      TEAM: ebpf-platform
      VMCONFIG_FILE: ${CI_PROJECT_DIR}/vmconfig-${CI_PIPELINE_ID}-${ARCH}.json

.package_deb_common

  .package_deb_common:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    script:
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - echo "About to package for $RELEASE_VERSION"
    - set +x
    - printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_GPG_KEY)" | gpg --import
      --batch
    - EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
      needs gitlab runner restart"; exit $EXIT; fi
    - DEB_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_SIGNING_PASSPHRASE)
      || exit $?; export DEB_SIGNING_PASSPHRASE
-   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
?                                                                              -----------------------
+   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
?                                                                 ++++++++++++
-     --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project ${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
+     "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project
+     ${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
    - curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
      | tar -xz -C /usr/local/bin dd-pkg
    - find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
    - "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n  find $OMNIBUS_PACKAGE_DIR\
      \ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
      \ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/datadog-${DD_PROJECT}_*_${PACKAGE_ARCH}.deb $S3_ARTIFACTS_URI/$DESTINATION_DEB
    stage: packaging
    variables:
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
      PACKAGE_REQUIRED_FILES_LIST: test/required_files/agent-deb.txt

.package_rpm_common

  .package_rpm_common:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    before_script: null
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - echo "About to build for $RELEASE_VERSION"
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_GPG_KEY)" | gpg --import
      --batch
    - EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
      needs gitlab runner restart"; exit $EXIT; fi
    - RPM_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_SIGNING_PASSPHRASE)
      || exit $?; export RPM_SIGNING_PASSPHRASE
-   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
?                                                                              -----------------------
+   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
?                                                                 ++++++++++++
-     --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
?                                                                            ----------------------
+     "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT}
?    +++++++++++++++++++++++
+     ${OMNIBUS_EXTRA_ARGS}
    - ls -la $OMNIBUS_PACKAGE_DIR/
    - curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
      | tar -xz -C /usr/local/bin dd-pkg
    - find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
    - "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n  find $OMNIBUS_PACKAGE_DIR\
      \ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
      \ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
    stage: packaging
    variables:
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
      PACKAGE_REQUIRED_FILES_LIST: test/required_files/agent-rpm.txt

.package_suse_rpm_common

  .package_suse_rpm_common:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR_SUSE
    before_script: null
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - echo "About to build for $RELEASE_VERSION"
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_GPG_KEY)" | gpg --import
      --batch
    - EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
      needs gitlab runner restart"; exit $EXIT; fi
    - RPM_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_SIGNING_PASSPHRASE)
      || exit $?; export RPM_SIGNING_PASSPHRASE
-   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
?                                                                              -----------------------
+   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
?                                                                 ++++++++++++
-     --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
?                                                                            ----------------------
+     "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT}
?    +++++++++++++++++++++++
+     ${OMNIBUS_EXTRA_ARGS}
    - ls -la $OMNIBUS_PACKAGE_DIR/
    - curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
      | tar -xz -C /usr/local/bin dd-pkg
    - find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
    - "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n  find $OMNIBUS_PACKAGE_DIR\
      \ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
      \ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
    - mkdir -p $OMNIBUS_PACKAGE_DIR_SUSE && mv $OMNIBUS_PACKAGE_DIR/*.rpm $OMNIBUS_PACKAGE_DIR_SUSE/
    stage: packaging
    variables:
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      OMNIBUS_EXTRA_ARGS: --host-distribution=suse
      OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
      PACKAGE_REQUIRED_FILES_LIST: test/required_files/agent-rpm.txt

.prepare_secagent_ebpf_functional_tests

  .prepare_secagent_ebpf_functional_tests:
    artifacts:
      paths:
      - $CI_PROJECT_DIR/kmt-deps
      - $DD_AGENT_TESTING_DIR/site-cookbooks/dd-security-agent-check/files
      when: always
    before_script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache_tools.tar.xz
+   - source /root/.bashrc
    - inv -e install-tools
    - mkdir -p $DATADOG_AGENT_EMBEDDED_PATH/bin
    - mkdir -p $DATADOG_AGENT_EMBEDDED_PATH/include
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.$ARCH /tmp/clang-bpf
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.$ARCH /tmp/llc-bpf
    needs:
    - go_deps
    - go_tools_deps
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - inv -e kmt.prepare --ci --component="security-agent"
    - mkdir -p /opt/datadog-agent/embedded/bin
    - cp /tmp/clang-bpf /opt/datadog-agent/embedded/bin/clang-bpf
    - cp /tmp/llc-bpf /opt/datadog-agent/embedded/bin/llc-bpf
    - invoke -e security-agent.kitchen-prepare --skip-linters
    stage: source_test

.prepare_sysprobe_ebpf_functional_tests

  .prepare_sysprobe_ebpf_functional_tests:
    artifacts:
      paths:
      - $CI_PROJECT_DIR/kmt-deps
      when: always
    before_script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache_tools.tar.xz
+   - source /root/.bashrc
    - inv -e install-tools
    - mkdir -p $DATADOG_AGENT_EMBEDDED_PATH/bin
    - mkdir -p $DATADOG_AGENT_EMBEDDED_PATH/include
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.$ARCH /tmp/clang-bpf
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.$ARCH /tmp/llc-bpf
    needs:
    - go_deps
    - go_tools_deps
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - inv -e kmt.prepare --ci --component="system-probe"
    stage: source_test
    variables:
      KUBERNETES_CPU_REQUEST: 4

.system-probe_build_common

  .system-probe_build_common:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
      - $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz.sum
    before_script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - find "$CI_BUILDS_DIR" ! -path '*DataDog/datadog-agent*' -depth
    - find "$CI_BUILDS_DIR" ! -path '*DataDog/datadog-agent*' -delete || true
+   - source /root/.bashrc
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - inv check-go-version
    - inv -e system-probe.build --strip-object-files --no-bundle
    - objdump -p $CI_PROJECT_DIR/$SYSTEM_PROBE_BINARIES_DIR/system-probe | egrep 'GLIBC_2\.(1[8-9]|[2-9][0-9])'
      && exit 1
    - inv -e system-probe.save-build-outputs $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
    variables:
      KUBERNETES_CPU_REQUEST: 6
      KUBERNETES_MEMORY_LIMIT: 12Gi
      KUBERNETES_MEMORY_REQUEST: 6Gi

.tests_linux_ebpf

  .tests_linux_ebpf:
    before_script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache_tools.tar.xz
+   - source /root/.bashrc
    needs:
    - go_deps
    - go_tools_deps
    script:
    - inv -e install-tools
    - inv -e system-probe.object-files
    - invoke -e linter.go --build system-probe-unit-tests --cpus 4 --targets ./pkg
    - invoke -e security-agent.run-ebpf-unit-tests --verbose
    - invoke -e linter.go --targets=./pkg/security/tests --cpus 4 --build-tags="functionaltests
      stresstests trivy containerd linux_bpf ebpf_bindata"
    stage: source_test
    variables:
      KUBERNETES_CPU_REQUEST: 6
      KUBERNETES_MEMORY_LIMIT: 16Gi
      KUBERNETES_MEMORY_REQUEST: 16Gi

.upload_secagent_tests

  .upload_secagent_tests:
    after_script:
    - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $API_KEY_ORG2) || exit $?;
      export DD_API_KEY
    - inv -e kmt.tag-ci-job
    allow_failure: true
    artifacts:
      paths:
      - $CI_PROJECT_DIR/connector-${ARCH}
      when: always
    before_script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - mkdir -p ~/.aws
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_PROFILE >> ~/.aws/config ||
      exit $?
    - export AWS_PROFILE=agent-qa-ci
    - touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SSH_KEY > $AWS_EC2_SSH_KEY_FILE || exit
      $?
    - echo "" >> $AWS_EC2_SSH_KEY_FILE
    - chmod 600 $AWS_EC2_SSH_KEY_FILE
    rules:
    - allow_failure: true
      if: $CI_COMMIT_BRANCH == "main"
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - if: $RUN_KMT_TESTS == 'on'
    - changes:
        compare_to: main
        paths:
        - pkg/ebpf/**/*
        - pkg/security/**/*
        - pkg/eventmonitor/**/*
        - test/kitchen/site-cookbooks/dd-security-agent-check/**/*
        - test/kitchen/test/integration/security-agent-test/**/*
        - test/kitchen/test/integration/security-agent-stress/**/*
        - .gitlab/functional_test/security_agent.yml
        - .gitlab/kernel_matrix_testing/security_agent.yml
        - .gitlab/kernel_matrix_testing/common.yml
        - .gitlab/source_test/ebpf.yml
        - test/new-e2e/system-probe/**/*
        - test/new-e2e/scenarios/system-probe/**/*
        - test/new-e2e/pkg/runner/**/*
        - test/new-e2e/pkg/utils/**/*
        - test/new-e2e/go.mod
        - tasks/security_agent.py
        - tasks/kmt.py
        - tasks/kernel_matrix_testing/*
    - allow_failure: true
      when: manual
    script:
+   - source /root/.bashrc
    - pushd $CI_PROJECT_DIR/kmt-deps/ci/$ARCH
    - tar czvf $TEST_ARCHIVE_NAME opt
    - popd
    - FILTER_TEAM="Name=tag:team,Values=ebpf-platform"
    - FILTER_MANAGED="Name=tag:managed-by,Values=pulumi"
    - FILTER_STATE="Name=instance-state-name,Values=running"
    - FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}"
    - FILTER_ARCH="Name=tag:arch,Values=${ARCH}"
    - FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}"
    - FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}"
    - QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId'
    - QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress'
    - "COUNTER=0\nwhile [[ $(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED\
      \ $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE\
      \ --output text --query $QUERY_INSTANCE_IDS  | wc -l ) != \"1\" && $COUNTER -le\
      \ 80 ]]; do COUNTER=$[$COUNTER +1]; echo \"[${COUNTER}] Waiting for instance\"\
      ; sleep 30; done\n# check that instance is ready, or fail\nif [ $(aws ec2 describe-instances\
      \ --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT\
      \ $FILTER_INSTANCE_TYPE --output text --query $QUERY_INSTANCE_IDS | wc -l) -ne\
      \ \"1\" ]; then\n    echo \"Instance NOT found\"\n    touch ${CI_PROJECT_DIR}/instance_not_found\n\
      \    \"false\"\nfi\necho \"Instance found\"\nINSTANCE_ID=$(aws ec2 describe-instances\
      \ --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT\
      \ $FILTER_INSTANCE_TYPE --output text --query $QUERY_INSTANCE_IDS)\naws ec2 wait\
      \ instance-status-ok --instance-ids $INSTANCE_ID\nsleep 10\n"
    - INSTANCE_IP=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED
      $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE --output
      text --query $QUERY_PRIVATE_IPS)
    - echo "$ARCH-instance-ip" $INSTANCE_IP
    - mkdir -p ~/.ssh && chmod 700 ~/.ssh
    - echo -e "Host metal_instance\nHostname $INSTANCE_IP\nUser ubuntu\nStrictHostKeyChecking
      no\nIdentityFile $AWS_EC2_SSH_KEY_FILE\n" | tee -a ~/.ssh/config
    - chmod 600 ~/.ssh/config
    - scp $CI_PROJECT_DIR/kmt-deps/ci/$ARCH/$TEST_ARCHIVE_NAME metal_instance:/opt/kernel-version-testing/
    - pushd $CI_PROJECT_DIR/test/new-e2e
    - go build -o $CI_PROJECT_DIR/connector-${ARCH} $CI_PROJECT_DIR/test/new-e2e/system-probe/connector/main.go
    - popd
    - scp $CI_PROJECT_DIR/connector-${ARCH} metal_instance:/home/ubuntu/connector
    stage: kernel_matrix_testing_prepare
    variables:
      AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key
      TEST_ARCHIVE_NAME: tests-$ARCH.tar.gz
      TEST_COMPONENT: security-agent

.upload_sysprobe_tests

  .upload_sysprobe_tests:
    after_script:
    - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $API_KEY_ORG2) || exit $?;
      export DD_API_KEY
    - inv -e kmt.tag-ci-job
    artifacts:
      paths:
      - $CI_PROJECT_DIR/connector-${ARCH}
      when: always
    before_script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - mkdir -p ~/.aws
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_PROFILE >> ~/.aws/config ||
      exit $?
    - export AWS_PROFILE=agent-qa-ci
    - touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SSH_KEY > $AWS_EC2_SSH_KEY_FILE || exit
      $?
    - echo "" >> $AWS_EC2_SSH_KEY_FILE
    - chmod 600 $AWS_EC2_SSH_KEY_FILE
    rules:
    - if: $CI_COMMIT_BRANCH == "main"
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - if: $RUN_KMT_TESTS == 'on'
    - changes:
        compare_to: main
        paths:
        - pkg/collector/corechecks/ebpf/**/*
        - pkg/collector/corechecks/servicediscovery/module/*
        - pkg/ebpf/**/*
        - pkg/network/**/*
        - pkg/process/monitor/*
        - pkg/util/kernel/**/*
        - .gitlab/kernel_matrix_testing/system_probe.yml
        - .gitlab/kernel_matrix_testing/common.yml
        - .gitlab/source_test/ebpf.yml
        - test/new-e2e/system-probe/**/*
        - test/new-e2e/scenarios/system-probe/**/*
        - test/new-e2e/pkg/runner/**/*
        - test/new-e2e/pkg/utils/**/*
        - test/new-e2e/go.mod
        - tasks/system_probe.py
        - tasks/kmt.py
        - tasks/kernel_matrix_testing/*
    script:
+   - source /root/.bashrc
    - pushd $CI_PROJECT_DIR/kmt-deps/ci/$ARCH
    - tar czvf $TEST_ARCHIVE_NAME opt
    - popd
    - FILTER_TEAM="Name=tag:team,Values=ebpf-platform"
    - FILTER_MANAGED="Name=tag:managed-by,Values=pulumi"
    - FILTER_STATE="Name=instance-state-name,Values=running"
    - FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}"
    - FILTER_ARCH="Name=tag:arch,Values=${ARCH}"
    - FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}"
    - FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}"
    - QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId'
    - QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress'
    - "COUNTER=0\nwhile [[ $(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED\
      \ $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE\
      \ --output text --query $QUERY_INSTANCE_IDS  | wc -l ) != \"1\" && $COUNTER -le\
      \ 80 ]]; do COUNTER=$[$COUNTER +1]; echo \"[${COUNTER}] Waiting for instance\"\
      ; sleep 30; done\n# check that instance is ready, or fail\nif [ $(aws ec2 describe-instances\
      \ --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT\
      \ $FILTER_INSTANCE_TYPE --output text --query $QUERY_INSTANCE_IDS | wc -l) -ne\
      \ \"1\" ]; then\n    echo \"Instance NOT found\"\n    touch ${CI_PROJECT_DIR}/instance_not_found\n\
      \    \"false\"\nfi\necho \"Instance found\"\nINSTANCE_ID=$(aws ec2 describe-instances\
      \ --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT\
      \ $FILTER_INSTANCE_TYPE --output text --query $QUERY_INSTANCE_IDS)\naws ec2 wait\
      \ instance-status-ok --instance-ids $INSTANCE_ID\nsleep 10\n"
    - INSTANCE_IP=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED
      $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE --output
      text --query $QUERY_PRIVATE_IPS)
    - echo "$ARCH-instance-ip" $INSTANCE_IP
    - mkdir -p ~/.ssh && chmod 700 ~/.ssh
    - echo -e "Host metal_instance\nHostname $INSTANCE_IP\nUser ubuntu\nStrictHostKeyChecking
      no\nIdentityFile $AWS_EC2_SSH_KEY_FILE\n" | tee -a ~/.ssh/config
    - chmod 600 ~/.ssh/config
    - scp $CI_PROJECT_DIR/kmt-deps/ci/$ARCH/$TEST_ARCHIVE_NAME metal_instance:/opt/kernel-version-testing/
    - pushd $CI_PROJECT_DIR/test/new-e2e
    - go build -o $CI_PROJECT_DIR/connector-${ARCH} $CI_PROJECT_DIR/test/new-e2e/system-probe/connector/main.go
    - popd
    - scp $CI_PROJECT_DIR/connector-${ARCH} metal_instance:/home/ubuntu/connector
    stage: kernel_matrix_testing_prepare
    variables:
      AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key
      TEST_ARCHIVE_NAME: tests-$ARCH.tar.gz
      TEST_COMPONENT: system-probe

agent_deb-arm64-a6

  agent_deb-arm64-a6:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_arm64$DATADOG_AGENT_ARMBUILDIMAGES_SUFFIX:$DATADOG_AGENT_ARMBUILDIMAGES
    needs:
    - datadog-agent-6-arm64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - echo "About to package for $RELEASE_VERSION"
    - set +x
    - printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_GPG_KEY)" | gpg --import
      --batch
    - EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
      needs gitlab runner restart"; exit $EXIT; fi
    - DEB_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_SIGNING_PASSPHRASE)
      || exit $?; export DEB_SIGNING_PASSPHRASE
-   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
?                                                                              -----------------------
+   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
?                                                                 ++++++++++++
-     --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project ${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
+     "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project
+     ${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
    - curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
      | tar -xz -C /usr/local/bin dd-pkg
    - find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
    - "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n  find $OMNIBUS_PACKAGE_DIR\
      \ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
      \ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/datadog-${DD_PROJECT}_*_${PACKAGE_ARCH}.deb $S3_ARTIFACTS_URI/$DESTINATION_DEB
    stage: packaging
    tags:
    - arch:arm64
    variables:
      AGENT_MAJOR_VERSION: 6
      DD_PKG_ARCH: arm64
      DD_PROJECT: agent
      DESTINATION_DEB: datadog-agent_6_arm64.deb
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
      PACKAGE_ARCH: arm64
      PACKAGE_REQUIRED_FILES_LIST: test/required_files/agent-deb.txt
      RELEASE_VERSION: $RELEASE_VERSION_6

agent_deb-arm64-a7

  agent_deb-arm64-a7:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_arm64$DATADOG_AGENT_ARMBUILDIMAGES_SUFFIX:$DATADOG_AGENT_ARMBUILDIMAGES
    needs:
    - datadog-agent-7-arm64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - echo "About to package for $RELEASE_VERSION"
    - set +x
    - printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_GPG_KEY)" | gpg --import
      --batch
    - EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
      needs gitlab runner restart"; exit $EXIT; fi
    - DEB_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_SIGNING_PASSPHRASE)
      || exit $?; export DEB_SIGNING_PASSPHRASE
-   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
?                                                                              -----------------------
+   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
?                                                                 ++++++++++++
-     --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project ${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
+     "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project
+     ${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
    - curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
      | tar -xz -C /usr/local/bin dd-pkg
    - find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
    - "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n  find $OMNIBUS_PACKAGE_DIR\
      \ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
      \ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/datadog-${DD_PROJECT}_*_${PACKAGE_ARCH}.deb $S3_ARTIFACTS_URI/$DESTINATION_DEB
    stage: packaging
    tags:
    - arch:arm64
    variables:
      AGENT_MAJOR_VERSION: 7
      DD_PKG_ARCH: arm64
      DD_PROJECT: agent
      DESTINATION_DEB: datadog-agent_7_arm64.deb
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
      PACKAGE_ARCH: arm64
      PACKAGE_REQUIRED_FILES_LIST: test/required_files/agent-deb.txt
      RELEASE_VERSION: $RELEASE_VERSION_7

agent_deb-x64-a6

  agent_deb-x64-a6:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - datadog-agent-6-x64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - echo "About to package for $RELEASE_VERSION"
    - set +x
    - printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_GPG_KEY)" | gpg --import
      --batch
    - EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
      needs gitlab runner restart"; exit $EXIT; fi
    - DEB_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_SIGNING_PASSPHRASE)
      || exit $?; export DEB_SIGNING_PASSPHRASE
-   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
?                                                                              -----------------------
+   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
?                                                                 ++++++++++++
-     --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project ${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
+     "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project
+     ${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
    - curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
      | tar -xz -C /usr/local/bin dd-pkg
    - find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
    - "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n  find $OMNIBUS_PACKAGE_DIR\
      \ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
      \ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/datadog-${DD_PROJECT}_*_${PACKAGE_ARCH}.deb $S3_ARTIFACTS_URI/$DESTINATION_DEB
    stage: packaging
    tags:
    - arch:amd64
    variables:
      AGENT_MAJOR_VERSION: 6
      DD_PKG_ARCH: x86_64
      DD_PROJECT: agent
      DESTINATION_DEB: datadog-agent_6_amd64.deb
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
      PACKAGE_ARCH: amd64
      PACKAGE_REQUIRED_FILES_LIST: test/required_files/agent-deb.txt
      RELEASE_VERSION: $RELEASE_VERSION_6

agent_deb-x64-a7

  agent_deb-x64-a7:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - datadog-agent-7-x64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - echo "About to package for $RELEASE_VERSION"
    - set +x
    - printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_GPG_KEY)" | gpg --import
      --batch
    - EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
      needs gitlab runner restart"; exit $EXIT; fi
    - DEB_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_SIGNING_PASSPHRASE)
      || exit $?; export DEB_SIGNING_PASSPHRASE
-   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
?                                                                              -----------------------
+   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
?                                                                 ++++++++++++
-     --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project ${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
+     "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project
+     ${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
    - curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
      | tar -xz -C /usr/local/bin dd-pkg
    - find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
    - "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n  find $OMNIBUS_PACKAGE_DIR\
      \ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
      \ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/datadog-${DD_PROJECT}_*_${PACKAGE_ARCH}.deb $S3_ARTIFACTS_URI/$DESTINATION_DEB
    stage: packaging
    tags:
    - arch:amd64
    variables:
      AGENT_MAJOR_VERSION: 7
      DD_PKG_ARCH: x86_64
      DD_PROJECT: agent
      DESTINATION_DEB: datadog-agent_7_amd64.deb
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
      PACKAGE_ARCH: amd64
      PACKAGE_REQUIRED_FILES_LIST: test/required_files/agent-deb.txt
      RELEASE_VERSION: $RELEASE_VERSION_7

agent_rpm-arm64-a6

  agent_rpm-arm64-a6:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    before_script: null
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/rpm_arm64$DATADOG_AGENT_ARMBUILDIMAGES_SUFFIX:$DATADOG_AGENT_ARMBUILDIMAGES
    needs:
    - datadog-agent-6-arm64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - echo "About to build for $RELEASE_VERSION"
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_GPG_KEY)" | gpg --import
      --batch
    - EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
      needs gitlab runner restart"; exit $EXIT; fi
    - RPM_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_SIGNING_PASSPHRASE)
      || exit $?; export RPM_SIGNING_PASSPHRASE
-   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
?                                                                              -----------------------
+   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
?                                                                 ++++++++++++
-     --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
?                                                                            ----------------------
+     "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT}
?    +++++++++++++++++++++++
+     ${OMNIBUS_EXTRA_ARGS}
    - ls -la $OMNIBUS_PACKAGE_DIR/
    - curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
      | tar -xz -C /usr/local/bin dd-pkg
    - find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
    - "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n  find $OMNIBUS_PACKAGE_DIR\
      \ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
      \ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
    stage: packaging
    tags:
    - arch:arm64
    variables:
      AGENT_MAJOR_VERSION: 6
      DD_PKG_ARCH: arm64
      DD_PROJECT: agent
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
      PACKAGE_ARCH: arm64
      PACKAGE_REQUIRED_FILES_LIST: test/required_files/agent-rpm.txt
      RELEASE_VERSION: $RELEASE_VERSION_6

agent_rpm-arm64-a7

  agent_rpm-arm64-a7:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    before_script: null
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/rpm_arm64$DATADOG_AGENT_ARMBUILDIMAGES_SUFFIX:$DATADOG_AGENT_ARMBUILDIMAGES
    needs:
    - datadog-agent-7-arm64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - echo "About to build for $RELEASE_VERSION"
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_GPG_KEY)" | gpg --import
      --batch
    - EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
      needs gitlab runner restart"; exit $EXIT; fi
    - RPM_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_SIGNING_PASSPHRASE)
      || exit $?; export RPM_SIGNING_PASSPHRASE
-   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
?                                                                              -----------------------
+   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
?                                                                 ++++++++++++
-     --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
?                                                                            ----------------------
+     "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT}
?    +++++++++++++++++++++++
+     ${OMNIBUS_EXTRA_ARGS}
    - ls -la $OMNIBUS_PACKAGE_DIR/
    - curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
      | tar -xz -C /usr/local/bin dd-pkg
    - find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
    - "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n  find $OMNIBUS_PACKAGE_DIR\
      \ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
      \ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
    stage: packaging
    tags:
    - arch:arm64
    variables:
      AGENT_MAJOR_VERSION: 7
      DD_PKG_ARCH: arm64
      DD_PROJECT: agent
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
      PACKAGE_ARCH: arm64
      PACKAGE_REQUIRED_FILES_LIST: test/required_files/agent-rpm.txt
      RELEASE_VERSION: $RELEASE_VERSION_7

agent_rpm-x64-a6

  agent_rpm-x64-a6:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    before_script: null
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/rpm_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - datadog-agent-6-x64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - echo "About to build for $RELEASE_VERSION"
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_GPG_KEY)" | gpg --import
      --batch
    - EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
      needs gitlab runner restart"; exit $EXIT; fi
    - RPM_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_SIGNING_PASSPHRASE)
      || exit $?; export RPM_SIGNING_PASSPHRASE
-   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
?                                                                              -----------------------
+   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
?                                                                 ++++++++++++
-     --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
?                                                                            ----------------------
+     "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT}
?    +++++++++++++++++++++++
+     ${OMNIBUS_EXTRA_ARGS}
    - ls -la $OMNIBUS_PACKAGE_DIR/
    - curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
      | tar -xz -C /usr/local/bin dd-pkg
    - find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
    - "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n  find $OMNIBUS_PACKAGE_DIR\
      \ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
      \ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
    stage: packaging
    tags:
    - arch:amd64
    variables:
      AGENT_MAJOR_VERSION: 6
      DD_PKG_ARCH: x86_64
      DD_PROJECT: agent
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
      PACKAGE_ARCH: amd64
      PACKAGE_REQUIRED_FILES_LIST: test/required_files/agent-rpm.txt
      RELEASE_VERSION: $RELEASE_VERSION_6

agent_rpm-x64-a7

  agent_rpm-x64-a7:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    before_script: null
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/rpm_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - datadog-agent-7-x64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - echo "About to build for $RELEASE_VERSION"
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_GPG_KEY)" | gpg --import
      --batch
    - EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
      needs gitlab runner restart"; exit $EXIT; fi
    - RPM_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_SIGNING_PASSPHRASE)
      || exit $?; export RPM_SIGNING_PASSPHRASE
-   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
?                                                                              -----------------------
+   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
?                                                                 ++++++++++++
-     --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
?                                                                            ----------------------
+     "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT}
?    +++++++++++++++++++++++
+     ${OMNIBUS_EXTRA_ARGS}
    - ls -la $OMNIBUS_PACKAGE_DIR/
    - curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
      | tar -xz -C /usr/local/bin dd-pkg
    - find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
    - "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n  find $OMNIBUS_PACKAGE_DIR\
      \ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
      \ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
    stage: packaging
    tags:
    - arch:amd64
    variables:
      AGENT_MAJOR_VERSION: 7
      DD_PKG_ARCH: x86_64
      DD_PROJECT: agent
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
      PACKAGE_ARCH: amd64
      PACKAGE_REQUIRED_FILES_LIST: test/required_files/agent-rpm.txt
      RELEASE_VERSION: $RELEASE_VERSION_7

agent_suse-arm64-a7

  agent_suse-arm64-a7:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR_SUSE
    before_script: null
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/rpm_arm64$DATADOG_AGENT_ARMBUILDIMAGES_SUFFIX:$DATADOG_AGENT_ARMBUILDIMAGES
    needs:
    - datadog-agent-7-arm64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - echo "About to build for $RELEASE_VERSION"
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_GPG_KEY)" | gpg --import
      --batch
    - EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
      needs gitlab runner restart"; exit $EXIT; fi
    - RPM_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_SIGNING_PASSPHRASE)
      || exit $?; export RPM_SIGNING_PASSPHRASE
-   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
?                                                                              -----------------------
+   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
?                                                                 ++++++++++++
-     --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
?                                                                            ----------------------
+     "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT}
?    +++++++++++++++++++++++
+     ${OMNIBUS_EXTRA_ARGS}
    - ls -la $OMNIBUS_PACKAGE_DIR/
    - curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
      | tar -xz -C /usr/local/bin dd-pkg
    - find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
    - "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n  find $OMNIBUS_PACKAGE_DIR\
      \ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
      \ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
    - mkdir -p $OMNIBUS_PACKAGE_DIR_SUSE && mv $OMNIBUS_PACKAGE_DIR/*.rpm $OMNIBUS_PACKAGE_DIR_SUSE/
    stage: packaging
    tags:
    - arch:arm64
    variables:
      AGENT_MAJOR_VERSION: 7
      DD_PKG_ARCH: arm64
      DD_PRODUCT: agent
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      OMNIBUS_EXTRA_ARGS: --host-distribution=suse
      OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
      PACKAGE_ARCH: arm64
      PACKAGE_REQUIRED_FILES_LIST: test/required_files/agent-rpm.txt
      RELEASE_VERSION: $RELEASE_VERSION_7

agent_suse-x64-a6

  agent_suse-x64-a6:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR_SUSE
    before_script: null
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/rpm_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - datadog-agent-6-x64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - echo "About to build for $RELEASE_VERSION"
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_GPG_KEY)" | gpg --import
      --batch
    - EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
      needs gitlab runner restart"; exit $EXIT; fi
    - RPM_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_SIGNING_PASSPHRASE)
      || exit $?; export RPM_SIGNING_PASSPHRASE
-   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
?                                                                              -----------------------
+   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
?                                                                 ++++++++++++
-     --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
?                                                                            ----------------------
+     "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT}
?    +++++++++++++++++++++++
+     ${OMNIBUS_EXTRA_ARGS}
    - ls -la $OMNIBUS_PACKAGE_DIR/
    - curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
      | tar -xz -C /usr/local/bin dd-pkg
    - find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
    - "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n  find $OMNIBUS_PACKAGE_DIR\
      \ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
      \ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
    - mkdir -p $OMNIBUS_PACKAGE_DIR_SUSE && mv $OMNIBUS_PACKAGE_DIR/*.rpm $OMNIBUS_PACKAGE_DIR_SUSE/
    stage: packaging
    tags:
    - arch:amd64
    variables:
      AGENT_MAJOR_VERSION: 6
      DD_PKG_ARCH: x86_64
      DD_PRODUCT: agent
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      OMNIBUS_EXTRA_ARGS: --host-distribution=suse
      OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
      PACKAGE_ARCH: amd64
      PACKAGE_REQUIRED_FILES_LIST: test/required_files/agent-rpm.txt
      RELEASE_VERSION: $RELEASE_VERSION_6

agent_suse-x64-a7

  agent_suse-x64-a7:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR_SUSE
    before_script: null
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/rpm_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - datadog-agent-7-x64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - echo "About to build for $RELEASE_VERSION"
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_GPG_KEY)" | gpg --import
      --batch
    - EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
      needs gitlab runner restart"; exit $EXIT; fi
    - RPM_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_SIGNING_PASSPHRASE)
      || exit $?; export RPM_SIGNING_PASSPHRASE
-   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
?                                                                              -----------------------
+   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
?                                                                 ++++++++++++
-     --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
?                                                                            ----------------------
+     "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT}
?    +++++++++++++++++++++++
+     ${OMNIBUS_EXTRA_ARGS}
    - ls -la $OMNIBUS_PACKAGE_DIR/
    - curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
      | tar -xz -C /usr/local/bin dd-pkg
    - find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
    - "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n  find $OMNIBUS_PACKAGE_DIR\
      \ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
      \ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
    - mkdir -p $OMNIBUS_PACKAGE_DIR_SUSE && mv $OMNIBUS_PACKAGE_DIR/*.rpm $OMNIBUS_PACKAGE_DIR_SUSE/
    stage: packaging
    tags:
    - arch:amd64
    variables:
      AGENT_MAJOR_VERSION: 7
      DD_PKG_ARCH: x86_64
      DD_PRODUCT: agent
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      OMNIBUS_EXTRA_ARGS: --host-distribution=suse
      OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
      PACKAGE_ARCH: amd64
      PACKAGE_REQUIRED_FILES_LIST: test/required_files/agent-rpm.txt
      RELEASE_VERSION: $RELEASE_VERSION_7

build_system-probe-arm64

  build_system-probe-arm64:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
      - $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz.sum
    before_script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - find "$CI_BUILDS_DIR" ! -path '*DataDog/datadog-agent*' -depth
    - find "$CI_BUILDS_DIR" ! -path '*DataDog/datadog-agent*' -delete || true
+   - source /root/.bashrc
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/system-probe_arm64$DATADOG_AGENT_SYSPROBE_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_SYSPROBE_BUILDIMAGES
    needs:
    - go_deps
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - inv check-go-version
    - inv -e system-probe.build --strip-object-files --no-bundle
    - objdump -p $CI_PROJECT_DIR/$SYSTEM_PROBE_BINARIES_DIR/system-probe | egrep 'GLIBC_2\.(1[8-9]|[2-9][0-9])'
      && exit 1
    - inv -e system-probe.save-build-outputs $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
    stage: binary_build
    tags:
    - arch:arm64
    variables:
      ARCH: arm64
      KUBERNETES_CPU_REQUEST: 6
      KUBERNETES_MEMORY_LIMIT: 12Gi
      KUBERNETES_MEMORY_REQUEST: 6Gi

build_system-probe-x64

  build_system-probe-x64:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
      - $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz.sum
    before_script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - find "$CI_BUILDS_DIR" ! -path '*DataDog/datadog-agent*' -depth
    - find "$CI_BUILDS_DIR" ! -path '*DataDog/datadog-agent*' -delete || true
+   - source /root/.bashrc
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/system-probe_x64$DATADOG_AGENT_SYSPROBE_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_SYSPROBE_BUILDIMAGES
    needs:
    - go_deps
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - inv check-go-version
    - inv -e system-probe.build --strip-object-files --no-bundle
    - objdump -p $CI_PROJECT_DIR/$SYSTEM_PROBE_BINARIES_DIR/system-probe | egrep 'GLIBC_2\.(1[8-9]|[2-9][0-9])'
      && exit 1
    - inv -e system-probe.save-build-outputs $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
    stage: binary_build
    tags:
    - arch:amd64
    variables:
      ARCH: amd64
      KUBERNETES_CPU_REQUEST: 6
      KUBERNETES_MEMORY_LIMIT: 12Gi
      KUBERNETES_MEMORY_REQUEST: 6Gi

datadog-agent-6-arm64

  datadog-agent-6-arm64:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    before_script:
    - export RELEASE_VERSION=$RELEASE_VERSION_6
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_arm64$DATADOG_AGENT_ARMBUILDIMAGES_SUFFIX:$DATADOG_AGENT_ARMBUILDIMAGES
    needs:
    - go_mod_tidy_check
    - build_system-probe-arm64
    - go_deps
    - generate_minimized_btfs_arm64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - echo "About to build for $RELEASE_VERSION"
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - rm -rf $OMNIBUS_PACKAGE_DIR/*
    - tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
    - mkdir -p /tmp/system-probe
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
    - cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
    - chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
-   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
?                                                                              -----------------------
+   - inv -e omnibus.build --fips-mode --release-version "$RELEASE_VERSION" --major-version
?                         ++++++++++++
-     --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING}
+     "$AGENT_MAJOR_VERSION" --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING}
?    +++++++++++++++++++++++
      --skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
      --flavor "$FLAVOR"
    - ls -la $OMNIBUS_PACKAGE_DIR
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
    stage: package_build
    tags:
    - arch:arm64
    variables:
      AGENT_MAJOR_VERSION: 6
      FLAVOR: base
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      PACKAGE_ARCH: arm64
      PYTHON_RUNTIMES: '2,3'

datadog-agent-6-x64

  datadog-agent-6-x64:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    before_script:
    - export RELEASE_VERSION=$RELEASE_VERSION_6
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/rpm_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - go_mod_tidy_check
    - build_system-probe-x64
    - go_deps
    - generate_minimized_btfs_x64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - echo "About to build for $RELEASE_VERSION"
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - rm -rf $OMNIBUS_PACKAGE_DIR/*
    - tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
    - mkdir -p /tmp/system-probe
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
    - cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
    - chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
-   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
?                                                                              -----------------------
+   - inv -e omnibus.build --fips-mode --release-version "$RELEASE_VERSION" --major-version
?                         ++++++++++++
-     --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING}
+     "$AGENT_MAJOR_VERSION" --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING}
?    +++++++++++++++++++++++
      --skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
      --flavor "$FLAVOR"
    - ls -la $OMNIBUS_PACKAGE_DIR
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
    stage: package_build
    tags:
    - arch:amd64
    variables:
      AGENT_MAJOR_VERSION: 6
      FLAVOR: base
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      PACKAGE_ARCH: amd64
      PYTHON_RUNTIMES: '2,3'

datadog-agent-7-arm64

  datadog-agent-7-arm64:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    before_script:
    - export RELEASE_VERSION=$RELEASE_VERSION_7
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_arm64$DATADOG_AGENT_ARMBUILDIMAGES_SUFFIX:$DATADOG_AGENT_ARMBUILDIMAGES
    needs:
    - go_mod_tidy_check
    - build_system-probe-arm64
    - go_deps
    - generate_minimized_btfs_arm64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - echo "About to build for $RELEASE_VERSION"
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - rm -rf $OMNIBUS_PACKAGE_DIR/*
    - tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
    - mkdir -p /tmp/system-probe
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
    - cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
    - chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
-   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
?                                                                              -----------------------
+   - inv -e omnibus.build --fips-mode --release-version "$RELEASE_VERSION" --major-version
?                         ++++++++++++
-     --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING}
+     "$AGENT_MAJOR_VERSION" --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING}
?    +++++++++++++++++++++++
      --skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
      --flavor "$FLAVOR"
    - ls -la $OMNIBUS_PACKAGE_DIR
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
    stage: package_build
    tags:
    - arch:arm64
    variables:
      AGENT_MAJOR_VERSION: 7
      FLAVOR: base
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      PACKAGE_ARCH: arm64
      PYTHON_RUNTIMES: '3'

datadog-agent-7-x64

  datadog-agent-7-x64:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    before_script:
    - export RELEASE_VERSION=$RELEASE_VERSION_7
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/rpm_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - go_mod_tidy_check
    - build_system-probe-x64
    - go_deps
    - generate_minimized_btfs_x64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - echo "About to build for $RELEASE_VERSION"
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - rm -rf $OMNIBUS_PACKAGE_DIR/*
    - tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
    - mkdir -p /tmp/system-probe
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
    - cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
    - chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
-   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
?                                                                              -----------------------
+   - inv -e omnibus.build --fips-mode --release-version "$RELEASE_VERSION" --major-version
?                         ++++++++++++
-     --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING}
+     "$AGENT_MAJOR_VERSION" --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING}
?    +++++++++++++++++++++++
      --skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
      --flavor "$FLAVOR"
    - ls -la $OMNIBUS_PACKAGE_DIR
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
    stage: package_build
    tags:
    - arch:amd64
    variables:
      AGENT_MAJOR_VERSION: 7
      FLAVOR: base
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      PACKAGE_ARCH: amd64
      PYTHON_RUNTIMES: '3'

datadog-ot-agent-7-arm64

  datadog-ot-agent-7-arm64:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    before_script:
    - export RELEASE_VERSION=$RELEASE_VERSION_7
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_arm64$DATADOG_AGENT_ARMBUILDIMAGES_SUFFIX:$DATADOG_AGENT_ARMBUILDIMAGES
    needs:
    - go_mod_tidy_check
    - build_system-probe-arm64
    - go_deps
    - generate_minimized_btfs_arm64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - echo "About to build for $RELEASE_VERSION"
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - rm -rf $OMNIBUS_PACKAGE_DIR/*
    - tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
    - mkdir -p /tmp/system-probe
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
    - cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
    - chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
-   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
?                                                                              -----------------------
+   - inv -e omnibus.build --fips-mode --release-version "$RELEASE_VERSION" --major-version
?                         ++++++++++++
-     --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING}
+     "$AGENT_MAJOR_VERSION" --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING}
?    +++++++++++++++++++++++
      --skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
      --flavor "$FLAVOR"
    - ls -la $OMNIBUS_PACKAGE_DIR
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
    stage: package_build
    tags:
    - arch:arm64
    variables:
      AGENT_MAJOR_VERSION: 7
      FLAVOR: ot
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      PACKAGE_ARCH: arm64
      PYTHON_RUNTIMES: '3'

datadog-ot-agent-7-x64

  datadog-ot-agent-7-x64:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    before_script:
    - export RELEASE_VERSION=$RELEASE_VERSION_7
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/rpm_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - go_mod_tidy_check
    - build_system-probe-x64
    - go_deps
    - generate_minimized_btfs_x64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - echo "About to build for $RELEASE_VERSION"
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - rm -rf $OMNIBUS_PACKAGE_DIR/*
    - tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
    - mkdir -p /tmp/system-probe
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
    - cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
    - chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
-   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
?                                                                              -----------------------
+   - inv -e omnibus.build --fips-mode --release-version "$RELEASE_VERSION" --major-version
?                         ++++++++++++
-     --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING}
+     "$AGENT_MAJOR_VERSION" --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING}
?    +++++++++++++++++++++++
      --skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
      --flavor "$FLAVOR"
    - ls -la $OMNIBUS_PACKAGE_DIR
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
    stage: package_build
    tags:
    - arch:amd64
    variables:
      AGENT_MAJOR_VERSION: 7
      FLAVOR: ot
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      PACKAGE_ARCH: amd64
      PYTHON_RUNTIMES: '3'

dogstatsd_deb-arm64

  dogstatsd_deb-arm64:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_arm64$DATADOG_AGENT_ARMBUILDIMAGES_SUFFIX:$DATADOG_AGENT_ARMBUILDIMAGES
    needs:
    - dogstatsd-arm64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - echo "About to package for $RELEASE_VERSION"
    - set +x
    - printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_GPG_KEY)" | gpg --import
      --batch
    - EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
      needs gitlab runner restart"; exit $EXIT; fi
    - DEB_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_SIGNING_PASSPHRASE)
      || exit $?; export DEB_SIGNING_PASSPHRASE
-   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
?                                                                              -----------------------
+   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
?                                                                 ++++++++++++
-     --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project ${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
+     "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project
+     ${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
    - curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
      | tar -xz -C /usr/local/bin dd-pkg
    - find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
    - "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n  find $OMNIBUS_PACKAGE_DIR\
      \ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
      \ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/datadog-${DD_PROJECT}_*_${PACKAGE_ARCH}.deb $S3_ARTIFACTS_URI/$DESTINATION_DEB
    stage: packaging
    tags:
    - arch:arm64
    variables:
      AGENT_MAJOR_VERSION: 7
      DD_PKG_ARCH: arm64
      DD_PROJECT: dogstatsd
      DESTINATION_DEB: datadog-dogstatsd_arm64.deb
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
      PACKAGE_ARCH: arm64
      PACKAGE_REQUIRED_FILES_LIST: test/required_files/dogstatsd-deb.txt
      RELEASE_VERSION: $RELEASE_VERSION_7

dogstatsd_deb-x64

  dogstatsd_deb-x64:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - dogstatsd-x64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - echo "About to package for $RELEASE_VERSION"
    - set +x
    - printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_GPG_KEY)" | gpg --import
      --batch
    - EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
      needs gitlab runner restart"; exit $EXIT; fi
    - DEB_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_SIGNING_PASSPHRASE)
      || exit $?; export DEB_SIGNING_PASSPHRASE
-   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
?                                                                              -----------------------
+   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
?                                                                 ++++++++++++
-     --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project ${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
+     "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project
+     ${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
    - curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
      | tar -xz -C /usr/local/bin dd-pkg
    - find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
    - "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n  find $OMNIBUS_PACKAGE_DIR\
      \ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
      \ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/datadog-${DD_PROJECT}_*_${PACKAGE_ARCH}.deb $S3_ARTIFACTS_URI/$DESTINATION_DEB
    stage: packaging
    tags:
    - arch:amd64
    variables:
      AGENT_MAJOR_VERSION: 7
      DD_PKG_ARCH: x86_64
      DD_PROJECT: dogstatsd
      DESTINATION_DEB: datadog-dogstatsd_amd64.deb
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
      PACKAGE_ARCH: amd64
      PACKAGE_REQUIRED_FILES_LIST: test/required_files/dogstatsd-deb.txt
      RELEASE_VERSION: $RELEASE_VERSION_7

dogstatsd_rpm-x64

  dogstatsd_rpm-x64:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    before_script: null
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/rpm_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - dogstatsd-x64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - echo "About to build for $RELEASE_VERSION"
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_GPG_KEY)" | gpg --import
      --batch
    - EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
      needs gitlab runner restart"; exit $EXIT; fi
    - RPM_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_SIGNING_PASSPHRASE)
      || exit $?; export RPM_SIGNING_PASSPHRASE
-   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
?                                                                              -----------------------
+   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
?                                                                 ++++++++++++
-     --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
?                                                                            ----------------------
+     "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT}
?    +++++++++++++++++++++++
+     ${OMNIBUS_EXTRA_ARGS}
    - ls -la $OMNIBUS_PACKAGE_DIR/
    - curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
      | tar -xz -C /usr/local/bin dd-pkg
    - find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
    - "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n  find $OMNIBUS_PACKAGE_DIR\
      \ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
      \ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
    stage: packaging
    tags:
    - arch:amd64
    variables:
      AGENT_MAJOR_VERSION: 7
      DD_PKG_ARCH: x86_64
      DD_PROJECT: dogstatsd
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
      PACKAGE_ARCH: amd64
      PACKAGE_REQUIRED_FILES_LIST: test/required_files/dogstatsd-rpm.txt
      RELEASE_VERSION: $RELEASE_VERSION_7

dogstatsd_suse-x64

  dogstatsd_suse-x64:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR_SUSE
    before_script: null
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/rpm_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - dogstatsd-x64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - echo "About to build for $RELEASE_VERSION"
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_GPG_KEY)" | gpg --import
      --batch
    - EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
      needs gitlab runner restart"; exit $EXIT; fi
    - RPM_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_SIGNING_PASSPHRASE)
      || exit $?; export RPM_SIGNING_PASSPHRASE
-   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
?                                                                              -----------------------
+   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
?                                                                 ++++++++++++
-     --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
?                                                                            ----------------------
+     "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT}
?    +++++++++++++++++++++++
+     ${OMNIBUS_EXTRA_ARGS}
    - ls -la $OMNIBUS_PACKAGE_DIR/
    - curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
      | tar -xz -C /usr/local/bin dd-pkg
    - find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
    - "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n  find $OMNIBUS_PACKAGE_DIR\
      \ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
      \ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
    - mkdir -p $OMNIBUS_PACKAGE_DIR_SUSE && mv $OMNIBUS_PACKAGE_DIR/*.rpm $OMNIBUS_PACKAGE_DIR_SUSE/
    stage: packaging
    tags:
    - arch:amd64
    variables:
      AGENT_MAJOR_VERSION: 7
      DD_PKG_ARCH: x86_64
      DD_PROJECT: dogstatsd
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      OMNIBUS_EXTRA_ARGS: --host-distribution=suse
      OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
      PACKAGE_ARCH: amd64
      PACKAGE_REQUIRED_FILES_LIST: test/required_files/dogstatsd-rpm.txt
      RELEASE_VERSION: $RELEASE_VERSION_7

installer_deb-amd64

  installer_deb-amd64:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - installer-amd64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - echo "About to package for $RELEASE_VERSION"
    - set +x
    - printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_GPG_KEY)" | gpg --import
      --batch
    - EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
      needs gitlab runner restart"; exit $EXIT; fi
    - DEB_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_SIGNING_PASSPHRASE)
      || exit $?; export DEB_SIGNING_PASSPHRASE
-   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
?                                                                              -----------------------
+   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
?                                                                 ++++++++++++
-     --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project ${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
+     "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project
+     ${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
    - curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
      | tar -xz -C /usr/local/bin dd-pkg
    - find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
    - "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n  find $OMNIBUS_PACKAGE_DIR\
      \ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
      \ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/datadog-${DD_PROJECT}_*_${PACKAGE_ARCH}.deb $S3_ARTIFACTS_URI/$DESTINATION_DEB
    stage: packaging
    tags:
    - arch:amd64
    variables:
      AGENT_MAJOR_VERSION: 7
      DD_PKG_ARCH: x86_64
      DD_PROJECT: installer
      DESTINATION_DEB: datadog-installer_7_amd64.deb
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
      PACKAGE_ARCH: amd64
      PACKAGE_REQUIRED_FILES_LIST: ''
      RELEASE_VERSION: $RELEASE_VERSION_7

installer_deb-arm64

  installer_deb-arm64:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_arm64$DATADOG_AGENT_ARMBUILDIMAGES_SUFFIX:$DATADOG_AGENT_ARMBUILDIMAGES
    needs:
    - installer-arm64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - echo "About to package for $RELEASE_VERSION"
    - set +x
    - printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_GPG_KEY)" | gpg --import
      --batch
    - EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
      needs gitlab runner restart"; exit $EXIT; fi
    - DEB_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_SIGNING_PASSPHRASE)
      || exit $?; export DEB_SIGNING_PASSPHRASE
-   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
?                                                                              -----------------------
+   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
?                                                                 ++++++++++++
-     --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project ${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
+     "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project
+     ${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
    - curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
      | tar -xz -C /usr/local/bin dd-pkg
    - find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
    - "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n  find $OMNIBUS_PACKAGE_DIR\
      \ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
      \ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/datadog-${DD_PROJECT}_*_${PACKAGE_ARCH}.deb $S3_ARTIFACTS_URI/$DESTINATION_DEB
    stage: packaging
    tags:
    - arch:arm64
    variables:
      AGENT_MAJOR_VERSION: 7
      DD_PKG_ARCH: arm64
      DD_PROJECT: installer
      DESTINATION_DEB: datadog-installer_7_arm64.deb
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
      PACKAGE_ARCH: arm64
      PACKAGE_REQUIRED_FILES_LIST: ''
      RELEASE_VERSION: $RELEASE_VERSION_7

installer_rpm-amd64

  installer_rpm-amd64:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    before_script: null
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/rpm_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - installer-amd64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - echo "About to build for $RELEASE_VERSION"
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_GPG_KEY)" | gpg --import
      --batch
    - EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
      needs gitlab runner restart"; exit $EXIT; fi
    - RPM_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_SIGNING_PASSPHRASE)
      || exit $?; export RPM_SIGNING_PASSPHRASE
-   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
?                                                                              -----------------------
+   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
?                                                                 ++++++++++++
-     --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
?                                                                            ----------------------
+     "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT}
?    +++++++++++++++++++++++
+     ${OMNIBUS_EXTRA_ARGS}
    - ls -la $OMNIBUS_PACKAGE_DIR/
    - curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
      | tar -xz -C /usr/local/bin dd-pkg
    - find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
    - "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n  find $OMNIBUS_PACKAGE_DIR\
      \ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
      \ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
    stage: packaging
    tags:
    - arch:amd64
    variables:
      AGENT_MAJOR_VERSION: 7
      DD_PKG_ARCH: x86_64
      DD_PROJECT: installer
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
      PACKAGE_ARCH: amd64
      PACKAGE_REQUIRED_FILES_LIST: ''
      RELEASE_VERSION: $RELEASE_VERSION_7

installer_rpm-arm64

  installer_rpm-arm64:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    before_script: null
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/rpm_arm64$DATADOG_AGENT_ARMBUILDIMAGES_SUFFIX:$DATADOG_AGENT_ARMBUILDIMAGES
    needs:
    - installer-arm64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - echo "About to build for $RELEASE_VERSION"
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_GPG_KEY)" | gpg --import
      --batch
    - EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
      needs gitlab runner restart"; exit $EXIT; fi
    - RPM_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_SIGNING_PASSPHRASE)
      || exit $?; export RPM_SIGNING_PASSPHRASE
-   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
?                                                                              -----------------------
+   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
?                                                                 ++++++++++++
-     --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
?                                                                            ----------------------
+     "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT}
?    +++++++++++++++++++++++
+     ${OMNIBUS_EXTRA_ARGS}
    - ls -la $OMNIBUS_PACKAGE_DIR/
    - curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
      | tar -xz -C /usr/local/bin dd-pkg
    - find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
    - "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n  find $OMNIBUS_PACKAGE_DIR\
      \ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
      \ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
    stage: packaging
    tags:
    - arch:arm64
    variables:
      AGENT_MAJOR_VERSION: 7
      DD_PKG_ARCH: arm64
      DD_PROJECT: installer
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
      PACKAGE_ARCH: arm64
      PACKAGE_REQUIRED_FILES_LIST: ''
      RELEASE_VERSION: $RELEASE_VERSION_7

installer_suse_rpm-amd64

  installer_suse_rpm-amd64:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR_SUSE
    before_script: null
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/rpm_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - installer-amd64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - echo "About to build for $RELEASE_VERSION"
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_GPG_KEY)" | gpg --import
      --batch
    - EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
      needs gitlab runner restart"; exit $EXIT; fi
    - RPM_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_SIGNING_PASSPHRASE)
      || exit $?; export RPM_SIGNING_PASSPHRASE
-   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
?                                                                              -----------------------
+   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
?                                                                 ++++++++++++
-     --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
?                                                                            ----------------------
+     "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT}
?    +++++++++++++++++++++++
+     ${OMNIBUS_EXTRA_ARGS}
    - ls -la $OMNIBUS_PACKAGE_DIR/
    - curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
      | tar -xz -C /usr/local/bin dd-pkg
    - find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
    - "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n  find $OMNIBUS_PACKAGE_DIR\
      \ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
      \ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
    - mkdir -p $OMNIBUS_PACKAGE_DIR_SUSE && mv $OMNIBUS_PACKAGE_DIR/*.rpm $OMNIBUS_PACKAGE_DIR_SUSE/
    stage: packaging
    tags:
    - arch:amd64
    variables:
      AGENT_MAJOR_VERSION: 7
      DD_PKG_ARCH: x86_64
      DD_PROJECT: installer
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      OMNIBUS_EXTRA_ARGS: --host-distribution=suse
      OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
      PACKAGE_ARCH: amd64
      PACKAGE_REQUIRED_FILES_LIST: ''
      RELEASE_VERSION: $RELEASE_VERSION_7

installer_suse_rpm-arm64

  installer_suse_rpm-arm64:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR_SUSE
    before_script: null
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/rpm_arm64$DATADOG_AGENT_ARMBUILDIMAGES_SUFFIX:$DATADOG_AGENT_ARMBUILDIMAGES
    needs:
    - installer-arm64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - echo "About to build for $RELEASE_VERSION"
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_GPG_KEY)" | gpg --import
      --batch
    - EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
      needs gitlab runner restart"; exit $EXIT; fi
    - RPM_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_SIGNING_PASSPHRASE)
      || exit $?; export RPM_SIGNING_PASSPHRASE
-   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
?                                                                              -----------------------
+   - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
?                                                                 ++++++++++++
-     --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
?                                                                            ----------------------
+     "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT}
?    +++++++++++++++++++++++
+     ${OMNIBUS_EXTRA_ARGS}
    - ls -la $OMNIBUS_PACKAGE_DIR/
    - curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
      | tar -xz -C /usr/local/bin dd-pkg
    - find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
    - "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n  find $OMNIBUS_PACKAGE_DIR\
      \ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
      \ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
    - mkdir -p $OMNIBUS_PACKAGE_DIR_SUSE && mv $OMNIBUS_PACKAGE_DIR/*.rpm $OMNIBUS_PACKAGE_DIR_SUSE/
    stage: packaging
    tags:
    - arch:arm64
    variables:
      AGENT_MAJOR_VERSION: 7
      DD_PKG_ARCH: arm64
      DD_PROJECT: installer
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      OMNIBUS_EXTRA_ARGS: --host-distribution=suse
      OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
      PACKAGE_ARCH: arm64
      PACKAGE_REQUIRED_FILES_LIST: ''
      RELEASE_VERSION: $RELEASE_VERSION_7

kmt_setup_env_secagent_arm64

  kmt_setup_env_secagent_arm64:
    after_script:
    - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $API_KEY_ORG2) || exit $?;
      export DD_API_KEY
    - export AWS_PROFILE=agent-qa-ci
    - FILTER_TEAM="Name=tag:team,Values=ebpf-platform"
    - FILTER_MANAGED="Name=tag:managed-by,Values=pulumi"
    - FILTER_STATE="Name=instance-state-name,Values=running"
    - FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}"
    - FILTER_ARCH="Name=tag:arch,Values=${ARCH}"
    - FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}"
    - FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}"
    - QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId'
    - QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress'
    - mkdir -p $CI_PROJECT_DIR/libvirt/log/$ARCH $CI_PROJECT_DIR/libvirt/xml $CI_PROJECT_DIR/libvirt/qemu
      $CI_PROJECT_DIR/libvirt/dnsmasq
    - INSTANCE_IP=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED
      $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE --output
      text --query $QUERY_PRIVATE_IPS)
    - echo "$ARCH-instance-ip" $INSTANCE_IP
    - ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP"
      "sudo virsh list --name | grep -v -E '^$' | xargs -I '{}' sh -c \"sudo virsh dumpxml
      '{}' > /tmp/ddvm-xml-'{}'.txt\""
    - ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP"
      "sudo virsh list --name | xargs -I '{}' sh -c \"sudo cp /var/log/libvirt/qemu/'{}'.log
      /tmp/qemu-ddvm-'{}'.log && sudo chown 1000:1000 /tmp/qemu-ddvm*\""
    - ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP"
      "mkdir /tmp/dnsmasq && sudo cp /var/lib/libvirt/dnsmasq/* /tmp/dnsmasq/ && sudo
      chown 1000:1000 /tmp/dnsmasq/*"
    - scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/ddvm-*.log"
      $CI_PROJECT_DIR/libvirt/log
    - scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/ddvm-xml-*"
      $CI_PROJECT_DIR/libvirt/xml
    - scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/qemu-ddvm-*.log"
      $CI_PROJECT_DIR/libvirt/qemu
    - scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/dnsmasq/*"
      $CI_PROJECT_DIR/libvirt/dnsmasq
    - "GO_ARCH=$ARCH\nif [ \"${ARCH}\" == \"x86_64\" ]; then\n  GO_ARCH=amd64\nfi\n"
    - cd test/new-e2e && GOOS=linux GOARCH="${GO_ARCH}" go build system-probe/vm-metrics/vm-metrics.go
    - scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE $CI_PROJECT_DIR/test/new-e2e/vm-metrics
      "ubuntu@$INSTANCE_IP:/home/ubuntu/vm-metrics"
    - ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP"
      "/home/ubuntu/vm-metrics -statsd-host=127.0.0.1 -statsd-port=8125 -libvirt-uri=/var/run/libvirt/libvirt-sock-ro
      --tag \"arch:${ARCH}\" --tag \"test-component:${TEST_COMPONENT}\" --tag \"ci-pipeline-id:${CI_PIPELINE_ID}\"
      --daemon -log-file /home/ubuntu/daemon.log"
    - inv -e kmt.tag-ci-job
    artifacts:
      paths:
      - $CI_PROJECT_DIR/stack.output
      - $CI_PROJECT_DIR/libvirt
      - $VMCONFIG_FILE
      when: always
    before_script:
+   - source /root/.bashrc
    - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $API_KEY_ORG2) || exit $?;
      export DD_API_KEY
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - mkdir -p ~/.aws
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_PROFILE >> ~/.aws/config ||
      exit $?
    - export AWS_PROFILE=agent-qa-ci
    - touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SSH_KEY > $AWS_EC2_SSH_KEY_FILE || exit
      $?
    - echo "" >> $AWS_EC2_SSH_KEY_FILE
    - chmod 600 $AWS_EC2_SSH_KEY_FILE
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/test-infra-definitions/runner$TEST_INFRA_DEFINITIONS_BUILDIMAGES_SUFFIX:$TEST_INFRA_DEFINITIONS_BUILDIMAGES
    needs:
    - go_deps
    - go_tools_deps
    rules:
    - allow_failure: true
      if: $CI_COMMIT_BRANCH == "main"
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - if: $RUN_KMT_TESTS == 'on'
    - changes:
        compare_to: main
        paths:
        - pkg/ebpf/**/*
        - pkg/security/**/*
        - pkg/eventmonitor/**/*
        - test/kitchen/site-cookbooks/dd-security-agent-check/**/*
        - test/kitchen/test/integration/security-agent-test/**/*
        - test/kitchen/test/integration/security-agent-stress/**/*
        - .gitlab/functional_test/security_agent.yml
        - .gitlab/kernel_matrix_testing/security_agent.yml
        - .gitlab/kernel_matrix_testing/common.yml
        - .gitlab/source_test/ebpf.yml
        - test/new-e2e/system-probe/**/*
        - test/new-e2e/scenarios/system-probe/**/*
        - test/new-e2e/pkg/runner/**/*
        - test/new-e2e/pkg/utils/**/*
        - test/new-e2e/go.mod
        - tasks/security_agent.py
        - tasks/kmt.py
        - tasks/kernel_matrix_testing/*
    - allow_failure: true
      when: manual
    script:
    - echo "s3://dd-pulumi-state?region=us-east-1&awssdk=v2&profile=$AWS_PROFILE" >
      $STACK_DIR
    - pulumi login $(cat $STACK_DIR | tr -d '\n')
    - inv -e kmt.gen-config --ci --arch=$ARCH --output-file=$VMCONFIG_FILE --sets=$TEST_SETS
      --vmconfig-template=$TEST_COMPONENT --memory=12288
    - inv -e system-probe.start-microvms --provision-instance --provision-microvms --vmconfig=$VMCONFIG_FILE
      $INSTANCE_TYPE_ARG $AMI_ID_ARG --ssh-key-name=$AWS_EC2_SSH_KEY_NAME --ssh-key-path=$AWS_EC2_SSH_KEY_FILE
      --infra-env=$INFRA_ENV --stack-name=kernel-matrix-testing-${TEST_COMPONENT}-${ARCH}-${CI_PIPELINE_ID}
      --run-agent
    - jq "." $CI_PROJECT_DIR/stack.output
    - pulumi logout
    stage: kernel_matrix_testing_prepare
    tags:
    - arch:amd64
    variables:
      AMI_ID_ARG: --arm-ami-id=$KERNEL_MATRIX_TESTING_ARM_AMI_ID
      ARCH: arm64
      AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key
      AWS_EC2_SSH_KEY_NAME: datadog-agent-ci
      AWS_REGION: us-east-1
      INFRA_ENV: aws/agent-qa
      INSTANCE_TYPE: m6gd.metal
      INSTANCE_TYPE_ARG: --instance-type-arm=$INSTANCE_TYPE
      KITCHEN_EC2_REGION: us-east-1
      KITCHEN_EC2_SG_IDS: sg-019917348cb0eb7e7
      KITCHEN_EC2_SUBNET: subnet-05d7c6b1b5cfea811
      KUBERNETES_MEMORY_LIMIT: 16Gi
      KUBERNETES_MEMORY_REQUEST: 12Gi
      LibvirtSSHKey: $CI_PROJECT_DIR/libvirt_rsa-arm
      PIPELINE_ID: $CI_PIPELINE_ID
      RESOURCE_TAGS: instance-type:${INSTANCE_TYPE},arch:${ARCH},test-component:${TEST_COMPONENT},git-branch:${CI_COMMIT_REF_NAME}
      STACK_DIR: $CI_PROJECT_DIR/stack.dir
      TEAM: ebpf-platform
      TEST_COMPONENT: security-agent
      TEST_SETS: all_tests
      VMCONFIG_FILE: ${CI_PROJECT_DIR}/vmconfig-${CI_PIPELINE_ID}-${ARCH}.json

kmt_setup_env_secagent_x64

  kmt_setup_env_secagent_x64:
    after_script:
    - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $API_KEY_ORG2) || exit $?;
      export DD_API_KEY
    - export AWS_PROFILE=agent-qa-ci
    - FILTER_TEAM="Name=tag:team,Values=ebpf-platform"
    - FILTER_MANAGED="Name=tag:managed-by,Values=pulumi"
    - FILTER_STATE="Name=instance-state-name,Values=running"
    - FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}"
    - FILTER_ARCH="Name=tag:arch,Values=${ARCH}"
    - FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}"
    - FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}"
    - QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId'
    - QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress'
    - mkdir -p $CI_PROJECT_DIR/libvirt/log/$ARCH $CI_PROJECT_DIR/libvirt/xml $CI_PROJECT_DIR/libvirt/qemu
      $CI_PROJECT_DIR/libvirt/dnsmasq
    - INSTANCE_IP=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED
      $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE --output
      text --query $QUERY_PRIVATE_IPS)
    - echo "$ARCH-instance-ip" $INSTANCE_IP
    - ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP"
      "sudo virsh list --name | grep -v -E '^$' | xargs -I '{}' sh -c \"sudo virsh dumpxml
      '{}' > /tmp/ddvm-xml-'{}'.txt\""
    - ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP"
      "sudo virsh list --name | xargs -I '{}' sh -c \"sudo cp /var/log/libvirt/qemu/'{}'.log
      /tmp/qemu-ddvm-'{}'.log && sudo chown 1000:1000 /tmp/qemu-ddvm*\""
    - ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP"
      "mkdir /tmp/dnsmasq && sudo cp /var/lib/libvirt/dnsmasq/* /tmp/dnsmasq/ && sudo
      chown 1000:1000 /tmp/dnsmasq/*"
    - scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/ddvm-*.log"
      $CI_PROJECT_DIR/libvirt/log
    - scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/ddvm-xml-*"
      $CI_PROJECT_DIR/libvirt/xml
    - scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/qemu-ddvm-*.log"
      $CI_PROJECT_DIR/libvirt/qemu
    - scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/dnsmasq/*"
      $CI_PROJECT_DIR/libvirt/dnsmasq
    - "GO_ARCH=$ARCH\nif [ \"${ARCH}\" == \"x86_64\" ]; then\n  GO_ARCH=amd64\nfi\n"
    - cd test/new-e2e && GOOS=linux GOARCH="${GO_ARCH}" go build system-probe/vm-metrics/vm-metrics.go
    - scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE $CI_PROJECT_DIR/test/new-e2e/vm-metrics
      "ubuntu@$INSTANCE_IP:/home/ubuntu/vm-metrics"
    - ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP"
      "/home/ubuntu/vm-metrics -statsd-host=127.0.0.1 -statsd-port=8125 -libvirt-uri=/var/run/libvirt/libvirt-sock-ro
      --tag \"arch:${ARCH}\" --tag \"test-component:${TEST_COMPONENT}\" --tag \"ci-pipeline-id:${CI_PIPELINE_ID}\"
      --daemon -log-file /home/ubuntu/daemon.log"
    - inv -e kmt.tag-ci-job
    artifacts:
      paths:
      - $CI_PROJECT_DIR/stack.output
      - $CI_PROJECT_DIR/libvirt
      - $VMCONFIG_FILE
      when: always
    before_script:
+   - source /root/.bashrc
    - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $API_KEY_ORG2) || exit $?;
      export DD_API_KEY
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - mkdir -p ~/.aws
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_PROFILE >> ~/.aws/config ||
      exit $?
    - export AWS_PROFILE=agent-qa-ci
    - touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SSH_KEY > $AWS_EC2_SSH_KEY_FILE || exit
      $?
    - echo "" >> $AWS_EC2_SSH_KEY_FILE
    - chmod 600 $AWS_EC2_SSH_KEY_FILE
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/test-infra-definitions/runner$TEST_INFRA_DEFINITIONS_BUILDIMAGES_SUFFIX:$TEST_INFRA_DEFINITIONS_BUILDIMAGES
    needs:
    - go_deps
    - go_tools_deps
    rules:
    - allow_failure: true
      if: $CI_COMMIT_BRANCH == "main"
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - if: $RUN_KMT_TESTS == 'on'
    - changes:
        compare_to: main
        paths:
        - pkg/ebpf/**/*
        - pkg/security/**/*
        - pkg/eventmonitor/**/*
        - test/kitchen/site-cookbooks/dd-security-agent-check/**/*
        - test/kitchen/test/integration/security-agent-test/**/*
        - test/kitchen/test/integration/security-agent-stress/**/*
        - .gitlab/functional_test/security_agent.yml
        - .gitlab/kernel_matrix_testing/security_agent.yml
        - .gitlab/kernel_matrix_testing/common.yml
        - .gitlab/source_test/ebpf.yml
        - test/new-e2e/system-probe/**/*
        - test/new-e2e/scenarios/system-probe/**/*
        - test/new-e2e/pkg/runner/**/*
        - test/new-e2e/pkg/utils/**/*
        - test/new-e2e/go.mod
        - tasks/security_agent.py
        - tasks/kmt.py
        - tasks/kernel_matrix_testing/*
    - allow_failure: true
      when: manual
    script:
    - echo "s3://dd-pulumi-state?region=us-east-1&awssdk=v2&profile=$AWS_PROFILE" >
      $STACK_DIR
    - pulumi login $(cat $STACK_DIR | tr -d '\n')
    - inv -e kmt.gen-config --ci --arch=$ARCH --output-file=$VMCONFIG_FILE --sets=$TEST_SETS
      --vmconfig-template=$TEST_COMPONENT --memory=12288
    - inv -e system-probe.start-microvms --provision-instance --provision-microvms --vmconfig=$VMCONFIG_FILE
      $INSTANCE_TYPE_ARG $AMI_ID_ARG --ssh-key-name=$AWS_EC2_SSH_KEY_NAME --ssh-key-path=$AWS_EC2_SSH_KEY_FILE
      --infra-env=$INFRA_ENV --stack-name=kernel-matrix-testing-${TEST_COMPONENT}-${ARCH}-${CI_PIPELINE_ID}
      --run-agent
    - jq "." $CI_PROJECT_DIR/stack.output
    - pulumi logout
    stage: kernel_matrix_testing_prepare
    tags:
    - arch:amd64
    variables:
      AMI_ID_ARG: --x86-ami-id=$KERNEL_MATRIX_TESTING_X86_AMI_ID
      ARCH: x86_64
      AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key
      AWS_EC2_SSH_KEY_NAME: datadog-agent-ci
      AWS_REGION: us-east-1
      INFRA_ENV: aws/agent-qa
      INSTANCE_TYPE: m5d.metal
      INSTANCE_TYPE_ARG: --instance-type-x86=$INSTANCE_TYPE
      KITCHEN_EC2_REGION: us-east-1
      KITCHEN_EC2_SG_IDS: sg-019917348cb0eb7e7
      KITCHEN_EC2_SUBNET: subnet-05d7c6b1b5cfea811
      KUBERNETES_MEMORY_LIMIT: 16Gi
      KUBERNETES_MEMORY_REQUEST: 12Gi
      LibvirtSSHKey: $CI_PROJECT_DIR/libvirt_rsa-x86
      PIPELINE_ID: $CI_PIPELINE_ID
      RESOURCE_TAGS: instance-type:${INSTANCE_TYPE},arch:${ARCH},test-component:${TEST_COMPONENT},git-branch:${CI_COMMIT_REF_NAME}
      STACK_DIR: $CI_PROJECT_DIR/stack.dir
      TEAM: ebpf-platform
      TEST_COMPONENT: security-agent
      TEST_SETS: all_tests
      VMCONFIG_FILE: ${CI_PROJECT_DIR}/vmconfig-${CI_PIPELINE_ID}-${ARCH}.json

kmt_setup_env_sysprobe_arm64

  kmt_setup_env_sysprobe_arm64:
    after_script:
    - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $API_KEY_ORG2) || exit $?;
      export DD_API_KEY
    - export AWS_PROFILE=agent-qa-ci
    - FILTER_TEAM="Name=tag:team,Values=ebpf-platform"
    - FILTER_MANAGED="Name=tag:managed-by,Values=pulumi"
    - FILTER_STATE="Name=instance-state-name,Values=running"
    - FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}"
    - FILTER_ARCH="Name=tag:arch,Values=${ARCH}"
    - FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}"
    - FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}"
    - QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId'
    - QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress'
    - mkdir -p $CI_PROJECT_DIR/libvirt/log/$ARCH $CI_PROJECT_DIR/libvirt/xml $CI_PROJECT_DIR/libvirt/qemu
      $CI_PROJECT_DIR/libvirt/dnsmasq
    - INSTANCE_IP=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED
      $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE --output
      text --query $QUERY_PRIVATE_IPS)
    - echo "$ARCH-instance-ip" $INSTANCE_IP
    - ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP"
      "sudo virsh list --name | grep -v -E '^$' | xargs -I '{}' sh -c \"sudo virsh dumpxml
      '{}' > /tmp/ddvm-xml-'{}'.txt\""
    - ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP"
      "sudo virsh list --name | xargs -I '{}' sh -c \"sudo cp /var/log/libvirt/qemu/'{}'.log
      /tmp/qemu-ddvm-'{}'.log && sudo chown 1000:1000 /tmp/qemu-ddvm*\""
    - ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP"
      "mkdir /tmp/dnsmasq && sudo cp /var/lib/libvirt/dnsmasq/* /tmp/dnsmasq/ && sudo
      chown 1000:1000 /tmp/dnsmasq/*"
    - scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/ddvm-*.log"
      $CI_PROJECT_DIR/libvirt/log
    - scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/ddvm-xml-*"
      $CI_PROJECT_DIR/libvirt/xml
    - scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/qemu-ddvm-*.log"
      $CI_PROJECT_DIR/libvirt/qemu
    - scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/dnsmasq/*"
      $CI_PROJECT_DIR/libvirt/dnsmasq
    - "GO_ARCH=$ARCH\nif [ \"${ARCH}\" == \"x86_64\" ]; then\n  GO_ARCH=amd64\nfi\n"
    - cd test/new-e2e && GOOS=linux GOARCH="${GO_ARCH}" go build system-probe/vm-metrics/vm-metrics.go
    - scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE $CI_PROJECT_DIR/test/new-e2e/vm-metrics
      "ubuntu@$INSTANCE_IP:/home/ubuntu/vm-metrics"
    - ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP"
      "/home/ubuntu/vm-metrics -statsd-host=127.0.0.1 -statsd-port=8125 -libvirt-uri=/var/run/libvirt/libvirt-sock-ro
      --tag \"arch:${ARCH}\" --tag \"test-component:${TEST_COMPONENT}\" --tag \"ci-pipeline-id:${CI_PIPELINE_ID}\"
      --daemon -log-file /home/ubuntu/daemon.log"
    - inv -e kmt.tag-ci-job
    artifacts:
      paths:
      - $CI_PROJECT_DIR/stack.output
      - $CI_PROJECT_DIR/libvirt
      - $VMCONFIG_FILE
      when: always
    before_script:
+   - source /root/.bashrc
    - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $API_KEY_ORG2) || exit $?;
      export DD_API_KEY
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - mkdir -p ~/.aws
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_PROFILE >> ~/.aws/config ||
      exit $?
    - export AWS_PROFILE=agent-qa-ci
    - touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SSH_KEY > $AWS_EC2_SSH_KEY_FILE || exit
      $?
    - echo "" >> $AWS_EC2_SSH_KEY_FILE
    - chmod 600 $AWS_EC2_SSH_KEY_FILE
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/test-infra-definitions/runner$TEST_INFRA_DEFINITIONS_BUILDIMAGES_SUFFIX:$TEST_INFRA_DEFINITIONS_BUILDIMAGES
    needs:
    - go_deps
    - go_tools_deps
    rules:
    - if: $CI_COMMIT_BRANCH == "main"
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - if: $RUN_KMT_TESTS == 'on'
    - changes:
        compare_to: main
        paths:
        - pkg/collector/corechecks/ebpf/**/*
        - pkg/collector/corechecks/servicediscovery/module/*
        - pkg/ebpf/**/*
        - pkg/network/**/*
        - pkg/process/monitor/*
        - pkg/util/kernel/**/*
        - .gitlab/kernel_matrix_testing/system_probe.yml
        - .gitlab/kernel_matrix_testing/common.yml
        - .gitlab/source_test/ebpf.yml
        - test/new-e2e/system-probe/**/*
        - test/new-e2e/scenarios/system-probe/**/*
        - test/new-e2e/pkg/runner/**/*
        - test/new-e2e/pkg/utils/**/*
        - test/new-e2e/go.mod
        - tasks/system_probe.py
        - tasks/kmt.py
        - tasks/kernel_matrix_testing/*
    script:
    - echo "s3://dd-pulumi-state?region=us-east-1&awssdk=v2&profile=$AWS_PROFILE" >
      $STACK_DIR
    - pulumi login $(cat $STACK_DIR | tr -d '\n')
    - inv -e kmt.gen-config --ci --arch=$ARCH --output-file=$VMCONFIG_FILE --sets=$TEST_SETS
      --vmconfig-template=$TEST_COMPONENT --memory=12288
    - inv -e system-probe.start-microvms --provision-instance --provision-microvms --vmconfig=$VMCONFIG_FILE
      $INSTANCE_TYPE_ARG $AMI_ID_ARG --ssh-key-name=$AWS_EC2_SSH_KEY_NAME --ssh-key-path=$AWS_EC2_SSH_KEY_FILE
      --infra-env=$INFRA_ENV --stack-name=kernel-matrix-testing-${TEST_COMPONENT}-${ARCH}-${CI_PIPELINE_ID}
      --run-agent
    - jq "." $CI_PROJECT_DIR/stack.output
    - pulumi logout
    stage: kernel_matrix_testing_prepare
    tags:
    - arch:amd64
    variables:
      AMI_ID_ARG: --arm-ami-id=$KERNEL_MATRIX_TESTING_ARM_AMI_ID
      ARCH: arm64
      AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key
      AWS_EC2_SSH_KEY_NAME: datadog-agent-ci
      AWS_REGION: us-east-1
      INFRA_ENV: aws/agent-qa
      INSTANCE_TYPE: m6gd.metal
      INSTANCE_TYPE_ARG: --instance-type-arm=$INSTANCE_TYPE
      KITCHEN_EC2_REGION: us-east-1
      KITCHEN_EC2_SG_IDS: sg-019917348cb0eb7e7
      KITCHEN_EC2_SUBNET: subnet-05d7c6b1b5cfea811
      KUBERNETES_MEMORY_LIMIT: 16Gi
      KUBERNETES_MEMORY_REQUEST: 12Gi
      LibvirtSSHKey: $CI_PROJECT_DIR/libvirt_rsa-arm
      PIPELINE_ID: $CI_PIPELINE_ID
      RESOURCE_TAGS: instance-type:${INSTANCE_TYPE},arch:${ARCH},test-component:${TEST_COMPONENT},git-branch:${CI_COMMIT_REF_NAME}
      STACK_DIR: $CI_PROJECT_DIR/stack.dir
      TEAM: ebpf-platform
      TEST_COMPONENT: system-probe
      TEST_SETS: only_usm,no_usm
      VMCONFIG_FILE: ${CI_PROJECT_DIR}/vmconfig-${CI_PIPELINE_ID}-${ARCH}.json

kmt_setup_env_sysprobe_x64

  kmt_setup_env_sysprobe_x64:
    after_script:
    - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $API_KEY_ORG2) || exit $?;
      export DD_API_KEY
    - export AWS_PROFILE=agent-qa-ci
    - FILTER_TEAM="Name=tag:team,Values=ebpf-platform"
    - FILTER_MANAGED="Name=tag:managed-by,Values=pulumi"
    - FILTER_STATE="Name=instance-state-name,Values=running"
    - FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}"
    - FILTER_ARCH="Name=tag:arch,Values=${ARCH}"
    - FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}"
    - FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}"
    - QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId'
    - QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress'
    - mkdir -p $CI_PROJECT_DIR/libvirt/log/$ARCH $CI_PROJECT_DIR/libvirt/xml $CI_PROJECT_DIR/libvirt/qemu
      $CI_PROJECT_DIR/libvirt/dnsmasq
    - INSTANCE_IP=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED
      $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE --output
      text --query $QUERY_PRIVATE_IPS)
    - echo "$ARCH-instance-ip" $INSTANCE_IP
    - ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP"
      "sudo virsh list --name | grep -v -E '^$' | xargs -I '{}' sh -c \"sudo virsh dumpxml
      '{}' > /tmp/ddvm-xml-'{}'.txt\""
    - ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP"
      "sudo virsh list --name | xargs -I '{}' sh -c \"sudo cp /var/log/libvirt/qemu/'{}'.log
      /tmp/qemu-ddvm-'{}'.log && sudo chown 1000:1000 /tmp/qemu-ddvm*\""
    - ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP"
      "mkdir /tmp/dnsmasq && sudo cp /var/lib/libvirt/dnsmasq/* /tmp/dnsmasq/ && sudo
      chown 1000:1000 /tmp/dnsmasq/*"
    - scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/ddvm-*.log"
      $CI_PROJECT_DIR/libvirt/log
    - scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/ddvm-xml-*"
      $CI_PROJECT_DIR/libvirt/xml
    - scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/qemu-ddvm-*.log"
      $CI_PROJECT_DIR/libvirt/qemu
    - scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/dnsmasq/*"
      $CI_PROJECT_DIR/libvirt/dnsmasq
    - "GO_ARCH=$ARCH\nif [ \"${ARCH}\" == \"x86_64\" ]; then\n  GO_ARCH=amd64\nfi\n"
    - cd test/new-e2e && GOOS=linux GOARCH="${GO_ARCH}" go build system-probe/vm-metrics/vm-metrics.go
    - scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE $CI_PROJECT_DIR/test/new-e2e/vm-metrics
      "ubuntu@$INSTANCE_IP:/home/ubuntu/vm-metrics"
    - ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP"
      "/home/ubuntu/vm-metrics -statsd-host=127.0.0.1 -statsd-port=8125 -libvirt-uri=/var/run/libvirt/libvirt-sock-ro
      --tag \"arch:${ARCH}\" --tag \"test-component:${TEST_COMPONENT}\" --tag \"ci-pipeline-id:${CI_PIPELINE_ID}\"
      --daemon -log-file /home/ubuntu/daemon.log"
    - inv -e kmt.tag-ci-job
    artifacts:
      paths:
      - $CI_PROJECT_DIR/stack.output
      - $CI_PROJECT_DIR/libvirt
      - $VMCONFIG_FILE
      when: always
    before_script:
+   - source /root/.bashrc
    - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $API_KEY_ORG2) || exit $?;
      export DD_API_KEY
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - mkdir -p ~/.aws
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_PROFILE >> ~/.aws/config ||
      exit $?
    - export AWS_PROFILE=agent-qa-ci
    - touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SSH_KEY > $AWS_EC2_SSH_KEY_FILE || exit
      $?
    - echo "" >> $AWS_EC2_SSH_KEY_FILE
    - chmod 600 $AWS_EC2_SSH_KEY_FILE
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/test-infra-definitions/runner$TEST_INFRA_DEFINITIONS_BUILDIMAGES_SUFFIX:$TEST_INFRA_DEFINITIONS_BUILDIMAGES
    needs:
    - go_deps
    - go_tools_deps
    rules:
    - if: $CI_COMMIT_BRANCH == "main"
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - if: $RUN_KMT_TESTS == 'on'
    - changes:
        compare_to: main
        paths:
        - pkg/collector/corechecks/ebpf/**/*
        - pkg/collector/corechecks/servicediscovery/module/*
        - pkg/ebpf/**/*
        - pkg/network/**/*
        - pkg/process/monitor/*
        - pkg/util/kernel/**/*
        - .gitlab/kernel_matrix_testing/system_probe.yml
        - .gitlab/kernel_matrix_testing/common.yml
        - .gitlab/source_test/ebpf.yml
        - test/new-e2e/system-probe/**/*
        - test/new-e2e/scenarios/system-probe/**/*
        - test/new-e2e/pkg/runner/**/*
        - test/new-e2e/pkg/utils/**/*
        - test/new-e2e/go.mod
        - tasks/system_probe.py
        - tasks/kmt.py
        - tasks/kernel_matrix_testing/*
    script:
    - echo "s3://dd-pulumi-state?region=us-east-1&awssdk=v2&profile=$AWS_PROFILE" >
      $STACK_DIR
    - pulumi login $(cat $STACK_DIR | tr -d '\n')
    - inv -e kmt.gen-config --ci --arch=$ARCH --output-file=$VMCONFIG_FILE --sets=$TEST_SETS
      --vmconfig-template=$TEST_COMPONENT --memory=12288
    - inv -e system-probe.start-microvms --provision-instance --provision-microvms --vmconfig=$VMCONFIG_FILE
      $INSTANCE_TYPE_ARG $AMI_ID_ARG --ssh-key-name=$AWS_EC2_SSH_KEY_NAME --ssh-key-path=$AWS_EC2_SSH_KEY_FILE
      --infra-env=$INFRA_ENV --stack-name=kernel-matrix-testing-${TEST_COMPONENT}-${ARCH}-${CI_PIPELINE_ID}
      --run-agent
    - jq "." $CI_PROJECT_DIR/stack.output
    - pulumi logout
    stage: kernel_matrix_testing_prepare
    tags:
    - arch:amd64
    variables:
      AMI_ID_ARG: --x86-ami-id=$KERNEL_MATRIX_TESTING_X86_AMI_ID
      ARCH: x86_64
      AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key
      AWS_EC2_SSH_KEY_NAME: datadog-agent-ci
      AWS_REGION: us-east-1
      INFRA_ENV: aws/agent-qa
      INSTANCE_TYPE: m5d.metal
      INSTANCE_TYPE_ARG: --instance-type-x86=$INSTANCE_TYPE
      KITCHEN_EC2_REGION: us-east-1
      KITCHEN_EC2_SG_IDS: sg-019917348cb0eb7e7
      KITCHEN_EC2_SUBNET: subnet-05d7c6b1b5cfea811
      KUBERNETES_MEMORY_LIMIT: 16Gi
      KUBERNETES_MEMORY_REQUEST: 12Gi
      LibvirtSSHKey: $CI_PROJECT_DIR/libvirt_rsa-x86
      PIPELINE_ID: $CI_PIPELINE_ID
      RESOURCE_TAGS: instance-type:${INSTANCE_TYPE},arch:${ARCH},test-component:${TEST_COMPONENT},git-branch:${CI_COMMIT_REF_NAME}
      STACK_DIR: $CI_PROJECT_DIR/stack.dir
      TEAM: ebpf-platform
      TEST_COMPONENT: system-probe
      TEST_SETS: only_usm,no_usm
      VMCONFIG_FILE: ${CI_PROJECT_DIR}/vmconfig-${CI_PIPELINE_ID}-${ARCH}.json

prepare_secagent_ebpf_functional_tests_arm64

  prepare_secagent_ebpf_functional_tests_arm64:
    artifacts:
      paths:
      - $CI_PROJECT_DIR/kmt-deps
      - $DD_AGENT_TESTING_DIR/site-cookbooks/dd-security-agent-check/files
      when: always
    before_script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache_tools.tar.xz
+   - source /root/.bashrc
    - inv -e install-tools
    - mkdir -p $DATADOG_AGENT_EMBEDDED_PATH/bin
    - mkdir -p $DATADOG_AGENT_EMBEDDED_PATH/include
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.$ARCH /tmp/clang-bpf
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.$ARCH /tmp/llc-bpf
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/system-probe_arm64$DATADOG_AGENT_SYSPROBE_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_SYSPROBE_BUILDIMAGES
    needs:
    - go_deps
    - go_tools_deps
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - inv -e kmt.prepare --ci --component="security-agent"
    - mkdir -p /opt/datadog-agent/embedded/bin
    - cp /tmp/clang-bpf /opt/datadog-agent/embedded/bin/clang-bpf
    - cp /tmp/llc-bpf /opt/datadog-agent/embedded/bin/llc-bpf
    - invoke -e security-agent.kitchen-prepare --skip-linters
    stage: source_test
    tags:
    - arch:arm64
    variables:
      ARCH: arm64

prepare_secagent_ebpf_functional_tests_x64

  prepare_secagent_ebpf_functional_tests_x64:
    artifacts:
      paths:
      - $CI_PROJECT_DIR/kmt-deps
      - $DD_AGENT_TESTING_DIR/site-cookbooks/dd-security-agent-check/files
      when: always
    before_script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache_tools.tar.xz
+   - source /root/.bashrc
    - inv -e install-tools
    - mkdir -p $DATADOG_AGENT_EMBEDDED_PATH/bin
    - mkdir -p $DATADOG_AGENT_EMBEDDED_PATH/include
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.$ARCH /tmp/clang-bpf
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.$ARCH /tmp/llc-bpf
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/system-probe_x64$DATADOG_AGENT_SYSPROBE_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_SYSPROBE_BUILDIMAGES
    needs:
    - go_deps
    - go_tools_deps
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - inv -e kmt.prepare --ci --component="security-agent"
    - mkdir -p /opt/datadog-agent/embedded/bin
    - cp /tmp/clang-bpf /opt/datadog-agent/embedded/bin/clang-bpf
    - cp /tmp/llc-bpf /opt/datadog-agent/embedded/bin/llc-bpf
    - invoke -e security-agent.kitchen-prepare --skip-linters
    stage: source_test
    tags:
    - arch:amd64
    variables:
      ARCH: amd64

prepare_sysprobe_ebpf_functional_tests_arm64

  prepare_sysprobe_ebpf_functional_tests_arm64:
    artifacts:
      paths:
      - $CI_PROJECT_DIR/kmt-deps
      when: always
    before_script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache_tools.tar.xz
+   - source /root/.bashrc
    - inv -e install-tools
    - mkdir -p $DATADOG_AGENT_EMBEDDED_PATH/bin
    - mkdir -p $DATADOG_AGENT_EMBEDDED_PATH/include
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.$ARCH /tmp/clang-bpf
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.$ARCH /tmp/llc-bpf
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/system-probe_arm64$DATADOG_AGENT_SYSPROBE_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_SYSPROBE_BUILDIMAGES
    needs:
    - go_deps
    - go_tools_deps
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - inv -e kmt.prepare --ci --component="system-probe"
    stage: source_test
    tags:
    - arch:arm64
    variables:
      ARCH: arm64
      KUBERNETES_CPU_REQUEST: 4

prepare_sysprobe_ebpf_functional_tests_x64

  prepare_sysprobe_ebpf_functional_tests_x64:
    artifacts:
      paths:
      - $CI_PROJECT_DIR/kmt-deps
      when: always
    before_script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache_tools.tar.xz
+   - source /root/.bashrc
    - inv -e install-tools
    - mkdir -p $DATADOG_AGENT_EMBEDDED_PATH/bin
    - mkdir -p $DATADOG_AGENT_EMBEDDED_PATH/include
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.$ARCH /tmp/clang-bpf
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.$ARCH /tmp/llc-bpf
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/system-probe_x64$DATADOG_AGENT_SYSPROBE_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_SYSPROBE_BUILDIMAGES
    needs:
    - go_deps
    - go_tools_deps
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - inv -e kmt.prepare --ci --component="system-probe"
    stage: source_test
    tags:
    - arch:amd64
    variables:
      ARCH: amd64
      KUBERNETES_CPU_REQUEST: 4

security_go_generate_check

  security_go_generate_check:
    before_script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache_tools.tar.xz
+   - source /root/.bashrc
    - pip3 install wheel
    - pip3 install -r docs/cloud-workload-security/scripts/requirements-docs.txt
    - inv -e install-tools
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/system-probe_x64$DATADOG_AGENT_SYSPROBE_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_SYSPROBE_BUILDIMAGES
    needs:
    - go_deps
    - go_tools_deps
    script:
    - inv -e security-agent.go-generate-check
    stage: source_test
    tags:
    - arch:amd64
    variables:
      KUBERNETES_CPU_REQUEST: 4

tests_ebpf_arm64

  tests_ebpf_arm64:
    before_script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache_tools.tar.xz
+   - source /root/.bashrc
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/system-probe_arm64$DATADOG_AGENT_SYSPROBE_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_SYSPROBE_BUILDIMAGES
    needs:
    - go_deps
    - go_tools_deps
    script:
    - inv -e install-tools
    - inv -e system-probe.object-files
    - invoke -e linter.go --build system-probe-unit-tests --cpus 4 --targets ./pkg
    - invoke -e security-agent.run-ebpf-unit-tests --verbose
    - invoke -e linter.go --targets=./pkg/security/tests --cpus 4 --build-tags="functionaltests
      stresstests trivy containerd linux_bpf ebpf_bindata"
    stage: source_test
    tags:
    - arch:arm64
    variables:
      ARCH: arm64
      KUBERNETES_CPU_REQUEST: 6
      KUBERNETES_MEMORY_LIMIT: 16Gi
      KUBERNETES_MEMORY_REQUEST: 16Gi
      TASK_ARCH: arm64

tests_ebpf_x64

  tests_ebpf_x64:
    before_script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache_tools.tar.xz
+   - source /root/.bashrc
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/system-probe_x64$DATADOG_AGENT_SYSPROBE_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_SYSPROBE_BUILDIMAGES
    needs:
    - go_deps
    - go_tools_deps
    script:
    - inv -e install-tools
    - inv -e system-probe.object-files
    - invoke -e linter.go --build system-probe-unit-tests --cpus 4 --targets ./pkg
    - invoke -e security-agent.run-ebpf-unit-tests --verbose
    - invoke -e linter.go --targets=./pkg/security/tests --cpus 4 --build-tags="functionaltests
      stresstests trivy containerd linux_bpf ebpf_bindata"
    stage: source_test
    tags:
    - arch:amd64
    variables:
      ARCH: amd64
      KUBERNETES_CPU_REQUEST: 6
      KUBERNETES_MEMORY_LIMIT: 16Gi
      KUBERNETES_MEMORY_REQUEST: 16Gi
      TASK_ARCH: x64

upload_secagent_tests_arm64

  upload_secagent_tests_arm64:
    after_script:
    - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $API_KEY_ORG2) || exit $?;
      export DD_API_KEY
    - inv -e kmt.tag-ci-job
    allow_failure: true
    artifacts:
      paths:
      - $CI_PROJECT_DIR/connector-${ARCH}
      when: always
    before_script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - mkdir -p ~/.aws
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_PROFILE >> ~/.aws/config ||
      exit $?
    - export AWS_PROFILE=agent-qa-ci
    - touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SSH_KEY > $AWS_EC2_SSH_KEY_FILE || exit
      $?
    - echo "" >> $AWS_EC2_SSH_KEY_FILE
    - chmod 600 $AWS_EC2_SSH_KEY_FILE
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/system-probe_arm64$DATADOG_AGENT_SYSPROBE_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_SYSPROBE_BUILDIMAGES
    needs:
    - go_deps
    - prepare_secagent_ebpf_functional_tests_arm64
    rules:
    - allow_failure: true
      if: $CI_COMMIT_BRANCH == "main"
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - if: $RUN_KMT_TESTS == 'on'
    - changes:
        compare_to: main
        paths:
        - pkg/ebpf/**/*
        - pkg/security/**/*
        - pkg/eventmonitor/**/*
        - test/kitchen/site-cookbooks/dd-security-agent-check/**/*
        - test/kitchen/test/integration/security-agent-test/**/*
        - test/kitchen/test/integration/security-agent-stress/**/*
        - .gitlab/functional_test/security_agent.yml
        - .gitlab/kernel_matrix_testing/security_agent.yml
        - .gitlab/kernel_matrix_testing/common.yml
        - .gitlab/source_test/ebpf.yml
        - test/new-e2e/system-probe/**/*
        - test/new-e2e/scenarios/system-probe/**/*
        - test/new-e2e/pkg/runner/**/*
        - test/new-e2e/pkg/utils/**/*
        - test/new-e2e/go.mod
        - tasks/security_agent.py
        - tasks/kmt.py
        - tasks/kernel_matrix_testing/*
    - allow_failure: true
      when: manual
    script:
+   - source /root/.bashrc
    - pushd $CI_PROJECT_DIR/kmt-deps/ci/$ARCH
    - tar czvf $TEST_ARCHIVE_NAME opt
    - popd
    - FILTER_TEAM="Name=tag:team,Values=ebpf-platform"
    - FILTER_MANAGED="Name=tag:managed-by,Values=pulumi"
    - FILTER_STATE="Name=instance-state-name,Values=running"
    - FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}"
    - FILTER_ARCH="Name=tag:arch,Values=${ARCH}"
    - FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}"
    - FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}"
    - QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId'
    - QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress'
    - "COUNTER=0\nwhile [[ $(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED\
      \ $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE\
      \ --output text --query $QUERY_INSTANCE_IDS  | wc -l ) != \"1\" && $COUNTER -le\
      \ 80 ]]; do COUNTER=$[$COUNTER +1]; echo \"[${COUNTER}] Waiting for instance\"\
      ; sleep 30; done\n# check that instance is ready, or fail\nif [ $(aws ec2 describe-instances\
      \ --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT\
      \ $FILTER_INSTANCE_TYPE --output text --query $QUERY_INSTANCE_IDS | wc -l) -ne\
      \ \"1\" ]; then\n    echo \"Instance NOT found\"\n    touch ${CI_PROJECT_DIR}/instance_not_found\n\
      \    \"false\"\nfi\necho \"Instance found\"\nINSTANCE_ID=$(aws ec2 describe-instances\
      \ --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT\
      \ $FILTER_INSTANCE_TYPE --output text --query $QUERY_INSTANCE_IDS)\naws ec2 wait\
      \ instance-status-ok --instance-ids $INSTANCE_ID\nsleep 10\n"
    - INSTANCE_IP=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED
      $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE --output
      text --query $QUERY_PRIVATE_IPS)
    - echo "$ARCH-instance-ip" $INSTANCE_IP
    - mkdir -p ~/.ssh && chmod 700 ~/.ssh
    - echo -e "Host metal_instance\nHostname $INSTANCE_IP\nUser ubuntu\nStrictHostKeyChecking
      no\nIdentityFile $AWS_EC2_SSH_KEY_FILE\n" | tee -a ~/.ssh/config
    - chmod 600 ~/.ssh/config
    - scp $CI_PROJECT_DIR/kmt-deps/ci/$ARCH/$TEST_ARCHIVE_NAME metal_instance:/opt/kernel-version-testing/
    - pushd $CI_PROJECT_DIR/test/new-e2e
    - go build -o $CI_PROJECT_DIR/connector-${ARCH} $CI_PROJECT_DIR/test/new-e2e/system-probe/connector/main.go
    - popd
    - scp $CI_PROJECT_DIR/connector-${ARCH} metal_instance:/home/ubuntu/connector
    stage: kernel_matrix_testing_prepare
    tags:
    - arch:arm64
    variables:
      ARCH: arm64
      AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key
      INSTANCE_TYPE: m6gd.metal
      TEST_ARCHIVE_NAME: tests-$ARCH.tar.gz
      TEST_COMPONENT: security-agent

upload_secagent_tests_x64

  upload_secagent_tests_x64:
    after_script:
    - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $API_KEY_ORG2) || exit $?;
      export DD_API_KEY
    - inv -e kmt.tag-ci-job
    allow_failure: true
    artifacts:
      paths:
      - $CI_PROJECT_DIR/connector-${ARCH}
      when: always
    before_script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - mkdir -p ~/.aws
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_PROFILE >> ~/.aws/config ||
      exit $?
    - export AWS_PROFILE=agent-qa-ci
    - touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SSH_KEY > $AWS_EC2_SSH_KEY_FILE || exit
      $?
    - echo "" >> $AWS_EC2_SSH_KEY_FILE
    - chmod 600 $AWS_EC2_SSH_KEY_FILE
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/system-probe_x64$DATADOG_AGENT_SYSPROBE_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_SYSPROBE_BUILDIMAGES
    needs:
    - go_deps
    - prepare_secagent_ebpf_functional_tests_x64
    rules:
    - allow_failure: true
      if: $CI_COMMIT_BRANCH == "main"
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - if: $RUN_KMT_TESTS == 'on'
    - changes:
        compare_to: main
        paths:
        - pkg/ebpf/**/*
        - pkg/security/**/*
        - pkg/eventmonitor/**/*
        - test/kitchen/site-cookbooks/dd-security-agent-check/**/*
        - test/kitchen/test/integration/security-agent-test/**/*
        - test/kitchen/test/integration/security-agent-stress/**/*
        - .gitlab/functional_test/security_agent.yml
        - .gitlab/kernel_matrix_testing/security_agent.yml
        - .gitlab/kernel_matrix_testing/common.yml
        - .gitlab/source_test/ebpf.yml
        - test/new-e2e/system-probe/**/*
        - test/new-e2e/scenarios/system-probe/**/*
        - test/new-e2e/pkg/runner/**/*
        - test/new-e2e/pkg/utils/**/*
        - test/new-e2e/go.mod
        - tasks/security_agent.py
        - tasks/kmt.py
        - tasks/kernel_matrix_testing/*
    - allow_failure: true
      when: manual
    script:
+   - source /root/.bashrc
    - pushd $CI_PROJECT_DIR/kmt-deps/ci/$ARCH
    - tar czvf $TEST_ARCHIVE_NAME opt
    - popd
    - FILTER_TEAM="Name=tag:team,Values=ebpf-platform"
    - FILTER_MANAGED="Name=tag:managed-by,Values=pulumi"
    - FILTER_STATE="Name=instance-state-name,Values=running"
    - FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}"
    - FILTER_ARCH="Name=tag:arch,Values=${ARCH}"
    - FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}"
    - FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}"
    - QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId'
    - QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress'
    - "COUNTER=0\nwhile [[ $(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED\
      \ $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE\
      \ --output text --query $QUERY_INSTANCE_IDS  | wc -l ) != \"1\" && $COUNTER -le\
      \ 80 ]]; do COUNTER=$[$COUNTER +1]; echo \"[${COUNTER}] Waiting for instance\"\
      ; sleep 30; done\n# check that instance is ready, or fail\nif [ $(aws ec2 describe-instances\
      \ --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT\
      \ $FILTER_INSTANCE_TYPE --output text --query $QUERY_INSTANCE_IDS | wc -l) -ne\
      \ \"1\" ]; then\n    echo \"Instance NOT found\"\n    touch ${CI_PROJECT_DIR}/instance_not_found\n\
      \    \"false\"\nfi\necho \"Instance found\"\nINSTANCE_ID=$(aws ec2 describe-instances\
      \ --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT\
      \ $FILTER_INSTANCE_TYPE --output text --query $QUERY_INSTANCE_IDS)\naws ec2 wait\
      \ instance-status-ok --instance-ids $INSTANCE_ID\nsleep 10\n"
    - INSTANCE_IP=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED
      $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE --output
      text --query $QUERY_PRIVATE_IPS)
    - echo "$ARCH-instance-ip" $INSTANCE_IP
    - mkdir -p ~/.ssh && chmod 700 ~/.ssh
    - echo -e "Host metal_instance\nHostname $INSTANCE_IP\nUser ubuntu\nStrictHostKeyChecking
      no\nIdentityFile $AWS_EC2_SSH_KEY_FILE\n" | tee -a ~/.ssh/config
    - chmod 600 ~/.ssh/config
    - scp $CI_PROJECT_DIR/kmt-deps/ci/$ARCH/$TEST_ARCHIVE_NAME metal_instance:/opt/kernel-version-testing/
    - pushd $CI_PROJECT_DIR/test/new-e2e
    - go build -o $CI_PROJECT_DIR/connector-${ARCH} $CI_PROJECT_DIR/test/new-e2e/system-probe/connector/main.go
    - popd
    - scp $CI_PROJECT_DIR/connector-${ARCH} metal_instance:/home/ubuntu/connector
    stage: kernel_matrix_testing_prepare
    tags:
    - arch:amd64
    variables:
      ARCH: x86_64
      AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key
      INSTANCE_TYPE: m5d.metal
      TEST_ARCHIVE_NAME: tests-$ARCH.tar.gz
      TEST_COMPONENT: security-agent

upload_sysprobe_tests_arm64

  upload_sysprobe_tests_arm64:
    after_script:
    - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $API_KEY_ORG2) || exit $?;
      export DD_API_KEY
    - inv -e kmt.tag-ci-job
    artifacts:
      paths:
      - $CI_PROJECT_DIR/connector-${ARCH}
      when: always
    before_script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - mkdir -p ~/.aws
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_PROFILE >> ~/.aws/config ||
      exit $?
    - export AWS_PROFILE=agent-qa-ci
    - touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SSH_KEY > $AWS_EC2_SSH_KEY_FILE || exit
      $?
    - echo "" >> $AWS_EC2_SSH_KEY_FILE
    - chmod 600 $AWS_EC2_SSH_KEY_FILE
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/system-probe_arm64$DATADOG_AGENT_SYSPROBE_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_SYSPROBE_BUILDIMAGES
    needs:
    - go_deps
    - prepare_sysprobe_ebpf_functional_tests_arm64
    - tests_ebpf_arm64
    rules:
    - if: $CI_COMMIT_BRANCH == "main"
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - if: $RUN_KMT_TESTS == 'on'
    - changes:
        compare_to: main
        paths:
        - pkg/collector/corechecks/ebpf/**/*
        - pkg/collector/corechecks/servicediscovery/module/*
        - pkg/ebpf/**/*
        - pkg/network/**/*
        - pkg/process/monitor/*
        - pkg/util/kernel/**/*
        - .gitlab/kernel_matrix_testing/system_probe.yml
        - .gitlab/kernel_matrix_testing/common.yml
        - .gitlab/source_test/ebpf.yml
        - test/new-e2e/system-probe/**/*
        - test/new-e2e/scenarios/system-probe/**/*
        - test/new-e2e/pkg/runner/**/*
        - test/new-e2e/pkg/utils/**/*
        - test/new-e2e/go.mod
        - tasks/system_probe.py
        - tasks/kmt.py
        - tasks/kernel_matrix_testing/*
    script:
+   - source /root/.bashrc
    - pushd $CI_PROJECT_DIR/kmt-deps/ci/$ARCH
    - tar czvf $TEST_ARCHIVE_NAME opt
    - popd
    - FILTER_TEAM="Name=tag:team,Values=ebpf-platform"
    - FILTER_MANAGED="Name=tag:managed-by,Values=pulumi"
    - FILTER_STATE="Name=instance-state-name,Values=running"
    - FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}"
    - FILTER_ARCH="Name=tag:arch,Values=${ARCH}"
    - FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}"
    - FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}"
    - QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId'
    - QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress'
    - "COUNTER=0\nwhile [[ $(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED\
      \ $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE\
      \ --output text --query $QUERY_INSTANCE_IDS  | wc -l ) != \"1\" && $COUNTER -le\
      \ 80 ]]; do COUNTER=$[$COUNTER +1]; echo \"[${COUNTER}] Waiting for instance\"\
      ; sleep 30; done\n# check that instance is ready, or fail\nif [ $(aws ec2 describe-instances\
      \ --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT\
      \ $FILTER_INSTANCE_TYPE --output text --query $QUERY_INSTANCE_IDS | wc -l) -ne\
      \ \"1\" ]; then\n    echo \"Instance NOT found\"\n    touch ${CI_PROJECT_DIR}/instance_not_found\n\
      \    \"false\"\nfi\necho \"Instance found\"\nINSTANCE_ID=$(aws ec2 describe-instances\
      \ --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT\
      \ $FILTER_INSTANCE_TYPE --output text --query $QUERY_INSTANCE_IDS)\naws ec2 wait\
      \ instance-status-ok --instance-ids $INSTANCE_ID\nsleep 10\n"
    - INSTANCE_IP=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED
      $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE --output
      text --query $QUERY_PRIVATE_IPS)
    - echo "$ARCH-instance-ip" $INSTANCE_IP
    - mkdir -p ~/.ssh && chmod 700 ~/.ssh
    - echo -e "Host metal_instance\nHostname $INSTANCE_IP\nUser ubuntu\nStrictHostKeyChecking
      no\nIdentityFile $AWS_EC2_SSH_KEY_FILE\n" | tee -a ~/.ssh/config
    - chmod 600 ~/.ssh/config
    - scp $CI_PROJECT_DIR/kmt-deps/ci/$ARCH/$TEST_ARCHIVE_NAME metal_instance:/opt/kernel-version-testing/
    - pushd $CI_PROJECT_DIR/test/new-e2e
    - go build -o $CI_PROJECT_DIR/connector-${ARCH} $CI_PROJECT_DIR/test/new-e2e/system-probe/connector/main.go
    - popd
    - scp $CI_PROJECT_DIR/connector-${ARCH} metal_instance:/home/ubuntu/connector
    stage: kernel_matrix_testing_prepare
    tags:
    - arch:arm64
    variables:
      ARCH: arm64
      AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key
      INSTANCE_TYPE: m6gd.metal
      TEST_ARCHIVE_NAME: tests-$ARCH.tar.gz
      TEST_COMPONENT: system-probe

upload_sysprobe_tests_x64

  upload_sysprobe_tests_x64:
    after_script:
    - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $API_KEY_ORG2) || exit $?;
      export DD_API_KEY
    - inv -e kmt.tag-ci-job
    artifacts:
      paths:
      - $CI_PROJECT_DIR/connector-${ARCH}
      when: always
    before_script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - mkdir -p ~/.aws
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_PROFILE >> ~/.aws/config ||
      exit $?
    - export AWS_PROFILE=agent-qa-ci
    - touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SSH_KEY > $AWS_EC2_SSH_KEY_FILE || exit
      $?
    - echo "" >> $AWS_EC2_SSH_KEY_FILE
    - chmod 600 $AWS_EC2_SSH_KEY_FILE
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/system-probe_x64$DATADOG_AGENT_SYSPROBE_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_SYSPROBE_BUILDIMAGES
    needs:
    - go_deps
    - prepare_sysprobe_ebpf_functional_tests_x64
    - tests_ebpf_x64
    rules:
    - if: $CI_COMMIT_BRANCH == "main"
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - if: $RUN_KMT_TESTS == 'on'
    - changes:
        compare_to: main
        paths:
        - pkg/collector/corechecks/ebpf/**/*
        - pkg/collector/corechecks/servicediscovery/module/*
        - pkg/ebpf/**/*
        - pkg/network/**/*
        - pkg/process/monitor/*
        - pkg/util/kernel/**/*
        - .gitlab/kernel_matrix_testing/system_probe.yml
        - .gitlab/kernel_matrix_testing/common.yml
        - .gitlab/source_test/ebpf.yml
        - test/new-e2e/system-probe/**/*
        - test/new-e2e/scenarios/system-probe/**/*
        - test/new-e2e/pkg/runner/**/*
        - test/new-e2e/pkg/utils/**/*
        - test/new-e2e/go.mod
        - tasks/system_probe.py
        - tasks/kmt.py
        - tasks/kernel_matrix_testing/*
    script:
+   - source /root/.bashrc
    - pushd $CI_PROJECT_DIR/kmt-deps/ci/$ARCH
    - tar czvf $TEST_ARCHIVE_NAME opt
    - popd
    - FILTER_TEAM="Name=tag:team,Values=ebpf-platform"
    - FILTER_MANAGED="Name=tag:managed-by,Values=pulumi"
    - FILTER_STATE="Name=instance-state-name,Values=running"
    - FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}"
    - FILTER_ARCH="Name=tag:arch,Values=${ARCH}"
    - FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}"
    - FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}"
    - QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId'
    - QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress'
    - "COUNTER=0\nwhile [[ $(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED\
      \ $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE\
      \ --output text --query $QUERY_INSTANCE_IDS  | wc -l ) != \"1\" && $COUNTER -le\
      \ 80 ]]; do COUNTER=$[$COUNTER +1]; echo \"[${COUNTER}] Waiting for instance\"\
      ; sleep 30; done\n# check that instance is ready, or fail\nif [ $(aws ec2 describe-instances\
      \ --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT\
      \ $FILTER_INSTANCE_TYPE --output text --query $QUERY_INSTANCE_IDS | wc -l) -ne\
      \ \"1\" ]; then\n    echo \"Instance NOT found\"\n    touch ${CI_PROJECT_DIR}/instance_not_found\n\
      \    \"false\"\nfi\necho \"Instance found\"\nINSTANCE_ID=$(aws ec2 describe-instances\
      \ --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT\
      \ $FILTER_INSTANCE_TYPE --output text --query $QUERY_INSTANCE_IDS)\naws ec2 wait\
      \ instance-status-ok --instance-ids $INSTANCE_ID\nsleep 10\n"
    - INSTANCE_IP=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED
      $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE --output
      text --query $QUERY_PRIVATE_IPS)
    - echo "$ARCH-instance-ip" $INSTANCE_IP
    - mkdir -p ~/.ssh && chmod 700 ~/.ssh
    - echo -e "Host metal_instance\nHostname $INSTANCE_IP\nUser ubuntu\nStrictHostKeyChecking
      no\nIdentityFile $AWS_EC2_SSH_KEY_FILE\n" | tee -a ~/.ssh/config
    - chmod 600 ~/.ssh/config
    - scp $CI_PROJECT_DIR/kmt-deps/ci/$ARCH/$TEST_ARCHIVE_NAME metal_instance:/opt/kernel-version-testing/
    - pushd $CI_PROJECT_DIR/test/new-e2e
    - go build -o $CI_PROJECT_DIR/connector-${ARCH} $CI_PROJECT_DIR/test/new-e2e/system-probe/connector/main.go
    - popd
    - scp $CI_PROJECT_DIR/connector-${ARCH} metal_instance:/home/ubuntu/connector
    stage: kernel_matrix_testing_prepare
    tags:
    - arch:amd64
    variables:
      ARCH: x86_64
      AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key
      INSTANCE_TYPE: m5d.metal
      TEST_ARCHIVE_NAME: tests-$ARCH.tar.gz
      TEST_COMPONENT: system-probe

Added Jobs

.fips-compliance-e2e

.fips-compliance-e2e:
  after_script:
  - $CI_PROJECT_DIR/tools/ci/junit_upload.sh
  artifacts:
    expire_in: 2 weeks
    paths:
    - $E2E_OUTPUT_DIR
    - junit-*.tgz
    reports:
      annotations:
      - $EXTERNAL_LINKS_PATH
    when: always
  before_script:
  - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_e2e.tar.xz -C $GOPATH/pkg/mod/cache
  - rm -f modcache_e2e.tar.xz
  - mkdir -p ~/.aws
  - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_PROFILE >> ~/.aws/config ||
    exit $?
  - export AWS_PROFILE=agent-qa-ci
  - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SSH_PUBLIC_KEY_RSA > $E2E_PUBLIC_KEY_PATH
    || exit $?
  - touch $E2E_PRIVATE_KEY_PATH && chmod 600 $E2E_PRIVATE_KEY_PATH && $CI_PROJECT_DIR/tools/ci/fetch_secret.sh
    $SSH_KEY_RSA > $E2E_PRIVATE_KEY_PATH || exit $?
  - pulumi login "s3://dd-pulumi-state?region=us-east-1&awssdk=v2&profile=$AWS_PROFILE"
  - ARM_CLIENT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_TESTS_AZURE_CLIENT_ID)
    || exit $?; export ARM_CLIENT_ID
  - ARM_CLIENT_SECRET=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_TESTS_AZURE_CLIENT_SECRET)
    || exit $?; export ARM_CLIENT_SECRET
  - ARM_TENANT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_TESTS_AZURE_TENANT_ID)
    || exit $?; export ARM_TENANT_ID
  - ARM_SUBSCRIPTION_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_TESTS_AZURE_SUBSCRIPTION_ID)
    || exit $?; export ARM_SUBSCRIPTION_ID
  - GOOGLE_CREDENTIALS=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_TESTS_GCP_CREDENTIALS)
    || exit $?; export GOOGLE_CREDENTIALS
  - inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
  image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/test-infra-definitions/runner$TEST_INFRA_DEFINITIONS_BUILDIMAGES_SUFFIX:$TEST_INFRA_DEFINITIONS_BUILDIMAGES
  needs:
  - go_e2e_deps
  rules:
  - if: $RUN_E2E_TESTS == "off"
    when: never
  - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
    when: never
  - if: $RUN_E2E_TESTS == "on"
    when: on_success
  - if: $CI_COMMIT_BRANCH == "main"
    when: on_success
  - if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
    when: on_success
  - if: $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+-rc\.[0-9]+$/
    when: on_success
  - changes:
      compare_to: main
      paths:
      - test/new-e2e/pkg/**/*
      - test/new-e2e/go.mod
  - changes:
      compare_to: main
      paths:
      - cmd/**/*
      - pkg/**/*
      - comp/**/*
      - test/new-e2e/tests/agent-subcommands/**/*
  - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
    when: never
  - allow_failure: true
    when: manual
  script:
  - inv -e new-e2e-tests.run --targets $TARGETS -c ddagent:imagePullRegistry=669783387624.dkr.ecr.us-east-1.amazonaws.com
    -c ddagent:imagePullUsername=AWS -c ddagent:imagePullPassword=$(aws ecr get-login-password)
    --junit-tar junit-${CI_JOB_ID}.tgz ${EXTRA_PARAMS} --test-washer
  stage: check_fips_compliance
  tags:
  - arch:amd64
  variables:
    E2E_COMMIT_SHA: $CI_COMMIT_SHORT_SHA
    E2E_KEY_PAIR_NAME: datadog-agent-ci-rsa
    E2E_OUTPUT_DIR: $CI_PROJECT_DIR/e2e-output
    E2E_PIPELINE_ID: $CI_PIPELINE_ID
    E2E_PRIVATE_KEY_PATH: /tmp/agent-qa-ssh-key
    E2E_PUBLIC_KEY_PATH: /tmp/agent-qa-ssh-key.pub
    EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
    KUBERNETES_CPU_REQUEST: 6
    KUBERNETES_MEMORY_LIMIT: 16Gi
    KUBERNETES_MEMORY_REQUEST: 12Gi
    SHOULD_RUN_IN_FLAKES_FINDER: 'true'
    TARGETS: ./tests/fips-compliance
    TEAM: agent-shared-components

fips-compliance-e2e-linux

fips-compliance-e2e-linux:
  after_script:
  - $CI_PROJECT_DIR/tools/ci/junit_upload.sh
  artifacts:
    expire_in: 2 weeks
    paths:
    - $E2E_OUTPUT_DIR
    - junit-*.tgz
    reports:
      annotations:
      - $EXTERNAL_LINKS_PATH
    when: always
  before_script:
  - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_e2e.tar.xz -C $GOPATH/pkg/mod/cache
  - rm -f modcache_e2e.tar.xz
  - mkdir -p ~/.aws
  - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_PROFILE >> ~/.aws/config ||
    exit $?
  - export AWS_PROFILE=agent-qa-ci
  - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SSH_PUBLIC_KEY_RSA > $E2E_PUBLIC_KEY_PATH
    || exit $?
  - touch $E2E_PRIVATE_KEY_PATH && chmod 600 $E2E_PRIVATE_KEY_PATH && $CI_PROJECT_DIR/tools/ci/fetch_secret.sh
    $SSH_KEY_RSA > $E2E_PRIVATE_KEY_PATH || exit $?
  - pulumi login "s3://dd-pulumi-state?region=us-east-1&awssdk=v2&profile=$AWS_PROFILE"
  - ARM_CLIENT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_TESTS_AZURE_CLIENT_ID)
    || exit $?; export ARM_CLIENT_ID
  - ARM_CLIENT_SECRET=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_TESTS_AZURE_CLIENT_SECRET)
    || exit $?; export ARM_CLIENT_SECRET
  - ARM_TENANT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_TESTS_AZURE_TENANT_ID)
    || exit $?; export ARM_TENANT_ID
  - ARM_SUBSCRIPTION_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_TESTS_AZURE_SUBSCRIPTION_ID)
    || exit $?; export ARM_SUBSCRIPTION_ID
  - GOOGLE_CREDENTIALS=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_TESTS_GCP_CREDENTIALS)
    || exit $?; export GOOGLE_CREDENTIALS
  - inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
  image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/test-infra-definitions/runner$TEST_INFRA_DEFINITIONS_BUILDIMAGES_SUFFIX:$TEST_INFRA_DEFINITIONS_BUILDIMAGES
  needs:
  - go_e2e_deps
  - qa_agent
  parallel:
    matrix:
    - EXTRA_PARAMS: --run "TestLinuxFIPSComplianceSuite"
    - EXTRA_PARAMS: --run "TestFIPSCiphersSuite"
  rules:
  - if: $RUN_E2E_TESTS == "off"
    when: never
  - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
    when: never
  - if: $RUN_E2E_TESTS == "on"
    when: on_success
  - if: $CI_COMMIT_BRANCH == "main"
    when: on_success
  - if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
    when: on_success
  - if: $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+-rc\.[0-9]+$/
    when: on_success
  - changes:
      compare_to: main
      paths:
      - test/new-e2e/pkg/**/*
      - test/new-e2e/go.mod
  - changes:
      compare_to: main
      paths:
      - cmd/**/*
      - pkg/**/*
      - comp/**/*
      - test/new-e2e/tests/agent-subcommands/**/*
  - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
    when: never
  - allow_failure: true
    when: manual
  script:
  - inv -e new-e2e-tests.run --targets $TARGETS -c ddagent:imagePullRegistry=669783387624.dkr.ecr.us-east-1.amazonaws.com
    -c ddagent:imagePullUsername=AWS -c ddagent:imagePullPassword=$(aws ecr get-login-password)
    --junit-tar junit-${CI_JOB_ID}.tgz ${EXTRA_PARAMS} --test-washer
  stage: check_fips_compliance
  tags:
  - arch:amd64
  variables:
    E2E_COMMIT_SHA: $CI_COMMIT_SHORT_SHA
    E2E_KEY_PAIR_NAME: datadog-agent-ci-rsa
    E2E_OUTPUT_DIR: $CI_PROJECT_DIR/e2e-output
    E2E_PIPELINE_ID: $CI_PIPELINE_ID
    E2E_PRIVATE_KEY_PATH: /tmp/agent-qa-ssh-key
    E2E_PUBLIC_KEY_PATH: /tmp/agent-qa-ssh-key.pub
    EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
    KUBERNETES_CPU_REQUEST: 6
    KUBERNETES_MEMORY_LIMIT: 16Gi
    KUBERNETES_MEMORY_REQUEST: 12Gi
    SHOULD_RUN_IN_FLAKES_FINDER: 'true'
    TARGETS: ./tests/fips-compliance
    TEAM: agent-shared-components

fips-compliance-e2e-windows

fips-compliance-e2e-windows:
  after_script:
  - $CI_PROJECT_DIR/tools/ci/junit_upload.sh
  artifacts:
    expire_in: 2 weeks
    paths:
    - $E2E_OUTPUT_DIR
    - junit-*.tgz
    reports:
      annotations:
      - $EXTERNAL_LINKS_PATH
    when: always
  before_script:
  - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_e2e.tar.xz -C $GOPATH/pkg/mod/cache
  - rm -f modcache_e2e.tar.xz
  - mkdir -p ~/.aws
  - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_PROFILE >> ~/.aws/config ||
    exit $?
  - export AWS_PROFILE=agent-qa-ci
  - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SSH_PUBLIC_KEY_RSA > $E2E_PUBLIC_KEY_PATH
    || exit $?
  - touch $E2E_PRIVATE_KEY_PATH && chmod 600 $E2E_PRIVATE_KEY_PATH && $CI_PROJECT_DIR/tools/ci/fetch_secret.sh
    $SSH_KEY_RSA > $E2E_PRIVATE_KEY_PATH || exit $?
  - pulumi login "s3://dd-pulumi-state?region=us-east-1&awssdk=v2&profile=$AWS_PROFILE"
  - ARM_CLIENT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_TESTS_AZURE_CLIENT_ID)
    || exit $?; export ARM_CLIENT_ID
  - ARM_CLIENT_SECRET=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_TESTS_AZURE_CLIENT_SECRET)
    || exit $?; export ARM_CLIENT_SECRET
  - ARM_TENANT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_TESTS_AZURE_TENANT_ID)
    || exit $?; export ARM_TENANT_ID
  - ARM_SUBSCRIPTION_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_TESTS_AZURE_SUBSCRIPTION_ID)
    || exit $?; export ARM_SUBSCRIPTION_ID
  - GOOGLE_CREDENTIALS=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_TESTS_GCP_CREDENTIALS)
    || exit $?; export GOOGLE_CREDENTIALS
  - inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
  image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/test-infra-definitions/runner$TEST_INFRA_DEFINITIONS_BUILDIMAGES_SUFFIX:$TEST_INFRA_DEFINITIONS_BUILDIMAGES
  needs:
  - go_e2e_deps
  - deploy_windows_testing-a7
  parallel:
    matrix:
    - EXTRA_PARAMS: --run "TestWindowsFIPSComplianceSuite"
  rules:
  - if: $RUN_E2E_TESTS == "off"
    when: never
  - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
    when: never
  - if: $RUN_E2E_TESTS == "on"
    when: on_success
  - if: $CI_COMMIT_BRANCH == "main"
    when: on_success
  - if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
    when: on_success
  - if: $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+-rc\.[0-9]+$/
    when: on_success
  - changes:
      compare_to: main
      paths:
      - test/new-e2e/pkg/**/*
      - test/new-e2e/go.mod
  - changes:
      compare_to: main
      paths:
      - cmd/**/*
      - pkg/**/*
      - comp/**/*
      - test/new-e2e/tests/agent-subcommands/**/*
  - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
    when: never
  - allow_failure: true
    when: manual
  script:
  - inv -e new-e2e-tests.run --targets $TARGETS -c ddagent:imagePullRegistry=669783387624.dkr.ecr.us-east-1.amazonaws.com
    -c ddagent:imagePullUsername=AWS -c ddagent:imagePullPassword=$(aws ecr get-login-password)
    --junit-tar junit-${CI_JOB_ID}.tgz ${EXTRA_PARAMS} --test-washer
  stage: check_fips_compliance
  tags:
  - arch:amd64
  variables:
    E2E_COMMIT_SHA: $CI_COMMIT_SHORT_SHA
    E2E_KEY_PAIR_NAME: datadog-agent-ci-rsa
    E2E_OUTPUT_DIR: $CI_PROJECT_DIR/e2e-output
    E2E_PIPELINE_ID: $CI_PIPELINE_ID
    E2E_PRIVATE_KEY_PATH: /tmp/agent-qa-ssh-key
    E2E_PUBLIC_KEY_PATH: /tmp/agent-qa-ssh-key.pub
    EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
    KUBERNETES_CPU_REQUEST: 6
    KUBERNETES_MEMORY_LIMIT: 16Gi
    KUBERNETES_MEMORY_REQUEST: 12Gi
    SHOULD_RUN_IN_FLAKES_FINDER: 'true'
    TARGETS: ./tests/fips-compliance
    TEAM: agent-shared-components

Changes Summary

Removed	Modified	Added	Renamed
0	57	3	0

[pr-commenter]

Test changes on VM

Use this command from test-infra-definitions to manually test this PR changes on a VM:

inv create-vm --pipeline-id=45072228 --os-family=ubuntu

Note: This applies to commit 98faafe

[github-actions]

Serverless Benchmark Results

BenchmarkStartEndInvocation comparison between 3ca96b6 and 98c1d48.

tl;dr

Use these benchmarks as an insight tool during development.

1. Skim down the vs base column in each chart. If there is a ~, then there was no statistically significant change to the benchmark. Otherwise, ensure the estimated percent change is either negative or very small.

2. The last row of each chart is the geomean. Ensure this percentage is either negative or very small.

What is this benchmarking?

The BenchmarkStartEndInvocation compares the amount of time it takes to call the start-invocation and end-invocation endpoints. For universal instrumentation languages (Dotnet, Golang, Java, Ruby), this represents the majority of the duration overhead added by our tracing layer.

The benchmark is run using a large variety of lambda request payloads. In the charts below, there is one row for each event payload type.

How do I interpret these charts?

The charts below comes from benchstat. They represent the statistical change in duration (sec/op), memory overhead (B/op), and allocations (allocs/op).

The benchstat docs explain how to interpret these charts.

Before the comparison table, we see common file-level configuration. If there are benchmarks with different configuration (for example, from different packages), benchstat will print separate tables for each configuration.

The table then compares the two input files for each benchmark. It shows the median and 95% confidence interval summaries for each benchmark before and after the change, and an A/B comparison under "vs base". ... The p-value measures how likely it is that any differences were due to random chance (i.e., noise). The "~" means benchstat did not detect a statistically significant difference between the two inputs. ...

Note that "statistically significant" is not the same as "large": with enough low-noise data, even very small changes can be distinguished from noise and considered statistically significant. It is, of course, generally easier to distinguish large changes from noise.

Finally, the last row of the table shows the geometric mean of each column, giving an overall picture of how the benchmarks changed. Proportional changes in the geomean reflect proportional changes in the benchmarks. For example, given n benchmarks, if sec/op for one of them increases by a factor of 2, then the sec/op geomean will increase by a factor of ⁿ√2.

I need more help

First off, do not worry if the benchmarks are failing. They are not tests. The intention is for them to be a tool for you to use during development.

If you would like a hand interpreting the results come chat with us in #serverless-agent in the internal DataDog slack or in #serverless in the public DataDog slack. We're happy to help!

Benchmark stats

goos: linux
goarch: amd64
pkg: github.com/DataDog/datadog-agent/pkg/serverless/daemon
cpu: AMD EPYC 7763 64-Core Processor                
                                      │ baseline/benchmark.log │       current/benchmark.log        │
                                      │         sec/op         │   sec/op     vs base               │
api-gateway-appsec.json                            90.20µ ± 4%   86.30µ ± 5%  -4.32% (p=0.019 n=10)
api-gateway-kong-appsec.json                       71.27µ ± 3%   69.00µ ± 3%  -3.18% (p=0.002 n=10)
api-gateway-kong.json                              70.41µ ± 2%   66.40µ ± 2%  -5.70% (p=0.000 n=10)
api-gateway-non-proxy-async.json                   110.8µ ± 1%   107.6µ ± 2%  -2.88% (p=0.000 n=10)
api-gateway-non-proxy.json                         113.7µ ± 2%   108.8µ ± 3%  -4.28% (p=0.004 n=10)
api-gateway-websocket-connect.json                 74.96µ ± 1%   72.96µ ± 2%  -2.67% (p=0.000 n=10)
api-gateway-websocket-default.json                 67.39µ ± 1%   65.49µ ± 3%  -2.81% (p=0.009 n=10)
api-gateway-websocket-disconnect.json              67.38µ ± 1%   66.06µ ± 2%  -1.96% (p=0.004 n=10)
api-gateway.json                                   121.9µ ± 1%   120.0µ ± 1%  -1.58% (p=0.000 n=10)
application-load-balancer.json                     67.79µ ± 1%   65.85µ ± 2%  -2.86% (p=0.000 n=10)
cloudfront.json                                    50.70µ ± 2%   48.33µ ± 1%  -4.66% (p=0.000 n=10)
cloudwatch-events.json                             40.97µ ± 2%   39.04µ ± 2%  -4.70% (p=0.000 n=10)
cloudwatch-logs.json                               67.51µ ± 1%   66.34µ ± 1%  -1.73% (p=0.007 n=10)
custom.json                                        33.09µ ± 1%   32.22µ ± 2%  -2.63% (p=0.000 n=10)
dynamodb.json                                      96.86µ ± 2%   96.66µ ± 2%       ~ (p=0.579 n=10)
empty.json                                         31.46µ ± 1%   31.03µ ± 2%       ~ (p=0.075 n=10)
eventbridge-custom.json                            50.60µ ± 2%   49.00µ ± 3%  -3.17% (p=0.009 n=10)
eventbridge-no-bus.json                            49.82µ ± 2%   47.98µ ± 3%  -3.70% (p=0.000 n=10)
eventbridge-no-timestamp.json                      49.58µ ± 3%   47.84µ ± 2%  -3.51% (p=0.005 n=10)
http-api.json                                      77.79µ ± 2%   75.10µ ± 3%  -3.47% (p=0.000 n=10)
kinesis-batch.json                                 74.92µ ± 1%   72.92µ ± 3%  -2.67% (p=0.019 n=10)
kinesis.json                                       57.48µ ± 1%   56.88µ ± 2%       ~ (p=0.315 n=10)
s3.json                                            62.48µ ± 1%   61.64µ ± 2%  -1.35% (p=0.035 n=10)
sns-batch.json                                     95.09µ ± 2%   92.97µ ± 2%  -2.22% (p=0.023 n=10)
sns.json                                           67.75µ ± 1%   66.69µ ± 4%       ~ (p=0.218 n=10)
snssqs.json                                        114.5µ ± 1%   112.2µ ± 1%  -1.97% (p=0.002 n=10)
snssqs_no_dd_context.json                          103.4µ ± 1%   100.8µ ± 1%  -2.52% (p=0.000 n=10)
sqs-aws-header.json                                59.05µ ± 1%   58.81µ ± 2%       ~ (p=0.739 n=10)
sqs-batch.json                                    100.43µ ± 2%   96.99µ ± 2%  -3.42% (p=0.001 n=10)
sqs.json                                           72.75µ ± 3%   71.05µ ± 1%  -2.34% (p=0.011 n=10)
sqs_no_dd_context.json                             65.56µ ± 2%   66.05µ ± 1%       ~ (p=0.436 n=10)
stepfunction.json                                  48.96µ ± 5%   47.15µ ± 3%  -3.71% (p=0.007 n=10)
geomean                                            68.77µ        66.96µ       -2.63%

                                      │ baseline/benchmark.log │        current/benchmark.log        │
                                      │          B/op          │     B/op      vs base               │
api-gateway-appsec.json                           37.34Ki ± 0%   37.33Ki ± 0%       ~ (p=0.197 n=10)
api-gateway-kong-appsec.json                      26.94Ki ± 0%   26.94Ki ± 0%       ~ (p=0.897 n=10)
api-gateway-kong.json                             24.44Ki ± 0%   24.43Ki ± 0%       ~ (p=0.566 n=10)
api-gateway-non-proxy-async.json                  48.15Ki ± 0%   48.14Ki ± 0%       ~ (p=0.325 n=10)
api-gateway-non-proxy.json                        47.38Ki ± 0%   47.37Ki ± 0%       ~ (p=0.382 n=10)
api-gateway-websocket-connect.json                25.55Ki ± 0%   25.55Ki ± 0%       ~ (p=0.810 n=10)
api-gateway-websocket-default.json                21.45Ki ± 0%   21.45Ki ± 0%       ~ (p=0.616 n=10)
api-gateway-websocket-disconnect.json             21.24Ki ± 0%   21.23Ki ± 0%       ~ (p=0.385 n=10)
api-gateway.json                                  49.63Ki ± 0%   49.62Ki ± 0%       ~ (p=0.725 n=10)
application-load-balancer.json                    23.33Ki ± 0%   23.32Ki ± 0%       ~ (p=0.085 n=10)
cloudfront.json                                   17.70Ki ± 0%   17.69Ki ± 0%       ~ (p=0.072 n=10)
cloudwatch-events.json                            11.75Ki ± 0%   11.74Ki ± 0%  -0.10% (p=0.008 n=10)
cloudwatch-logs.json                              53.39Ki ± 0%   53.39Ki ± 0%       ~ (p=0.400 n=10)
custom.json                                       9.776Ki ± 0%   9.768Ki ± 0%       ~ (p=0.362 n=10)
dynamodb.json                                     40.83Ki ± 0%   40.83Ki ± 0%       ~ (p=0.809 n=10)
empty.json                                        9.339Ki ± 0%   9.362Ki ± 0%  +0.25% (p=0.004 n=10)
eventbridge-custom.json                           15.06Ki ± 0%   15.05Ki ± 0%       ~ (p=0.469 n=10)
eventbridge-no-bus.json                           14.04Ki ± 0%   14.02Ki ± 0%       ~ (p=0.172 n=10)
eventbridge-no-timestamp.json                     14.05Ki ± 0%   14.04Ki ± 0%       ~ (p=0.796 n=10)
http-api.json                                     23.92Ki ± 0%   23.91Ki ± 0%       ~ (p=0.927 n=10)
kinesis-batch.json                                27.15Ki ± 0%   27.10Ki ± 0%  -0.16% (p=0.027 n=10)
kinesis.json                                      17.92Ki ± 0%   17.95Ki ± 0%       ~ (p=0.118 n=10)
s3.json                                           20.45Ki ± 0%   20.44Ki ± 1%       ~ (p=0.529 n=10)
sns-batch.json                                    38.82Ki ± 0%   38.81Ki ± 0%       ~ (p=0.926 n=10)
sns.json                                          24.09Ki ± 0%   24.09Ki ± 0%       ~ (p=0.739 n=10)
snssqs.json                                       50.77Ki ± 0%   50.74Ki ± 0%       ~ (p=0.542 n=10)
snssqs_no_dd_context.json                         44.99Ki ± 0%   44.96Ki ± 0%       ~ (p=0.123 n=10)
sqs-aws-header.json                               18.85Ki ± 1%   18.97Ki ± 0%       ~ (p=0.105 n=10)
sqs-batch.json                                    41.83Ki ± 0%   41.82Ki ± 0%       ~ (p=0.811 n=10)
sqs.json                                          25.67Ki ± 0%   25.71Ki ± 1%       ~ (p=0.971 n=10)
sqs_no_dd_context.json                            20.78Ki ± 1%   20.84Ki ± 0%       ~ (p=0.165 n=10)
stepfunction.json                                 14.36Ki ± 2%   14.30Ki ± 1%       ~ (p=0.739 n=10)
geomean                                           24.50Ki        24.50Ki       +0.00%

                                      │ baseline/benchmark.log │        current/benchmark.log        │
                                      │       allocs/op        │ allocs/op   vs base                 │
api-gateway-appsec.json                             630.0 ± 0%   629.0 ± 0%       ~ (p=0.656 n=10)
api-gateway-kong-appsec.json                        488.0 ± 0%   488.0 ± 0%       ~ (p=1.000 n=10) ¹
api-gateway-kong.json                               466.0 ± 0%   466.0 ± 0%       ~ (p=1.000 n=10)
api-gateway-non-proxy-async.json                    726.0 ± 0%   726.0 ± 0%       ~ (p=1.000 n=10)
api-gateway-non-proxy.json                          716.5 ± 0%   716.0 ± 0%       ~ (p=0.650 n=10)
api-gateway-websocket-connect.json                  453.5 ± 0%   453.0 ± 0%       ~ (p=0.650 n=10)
api-gateway-websocket-default.json                  380.0 ± 0%   379.0 ± 0%       ~ (p=0.656 n=10)
api-gateway-websocket-disconnect.json               370.0 ± 0%   370.0 ± 0%       ~ (p=1.000 n=10) ¹
api-gateway.json                                    791.0 ± 0%   791.0 ± 0%       ~ (p=1.000 n=10)
application-load-balancer.json                      353.0 ± 0%   353.0 ± 0%       ~ (p=1.000 n=10)
cloudfront.json                                     284.0 ± 0%   284.0 ± 0%       ~ (p=1.000 n=10) ¹
cloudwatch-events.json                              221.0 ± 0%   220.0 ± 0%  -0.45% (p=0.020 n=10)
cloudwatch-logs.json                                216.0 ± 0%   216.0 ± 0%       ~ (p=1.000 n=10)
custom.json                                         169.0 ± 1%   169.0 ± 1%       ~ (p=1.000 n=10)
dynamodb.json                                       589.0 ± 0%   589.0 ± 0%       ~ (p=1.000 n=10)
empty.json                                          160.0 ± 0%   160.0 ± 1%       ~ (p=0.474 n=10)
eventbridge-custom.json                             267.0 ± 0%   267.0 ± 0%       ~ (p=0.582 n=10)
eventbridge-no-bus.json                             258.0 ± 0%   258.0 ± 0%       ~ (p=0.365 n=10)
eventbridge-no-timestamp.json                       258.0 ± 0%   258.0 ± 0%       ~ (p=1.000 n=10)
http-api.json                                       434.0 ± 0%   434.0 ± 0%       ~ (p=0.569 n=10)
kinesis-batch.json                                  392.0 ± 0%   391.0 ± 1%  -0.26% (p=0.016 n=10)
kinesis.json                                        286.0 ± 0%   287.0 ± 0%       ~ (p=0.057 n=10)
s3.json                                             359.0 ± 1%   358.0 ± 1%       ~ (p=0.374 n=10)
sns-batch.json                                      456.5 ± 0%   456.0 ± 0%       ~ (p=0.812 n=10)
sns.json                                            324.0 ± 0%   324.0 ± 0%       ~ (p=0.352 n=10)
snssqs.json                                         440.0 ± 0%   439.5 ± 0%       ~ (p=0.681 n=10)
snssqs_no_dd_context.json                           401.0 ± 0%   401.0 ± 0%       ~ (p=0.254 n=10)
sqs-aws-header.json                                 274.0 ± 1%   276.0 ± 0%       ~ (p=0.160 n=10)
sqs-batch.json                                      506.0 ± 0%   506.0 ± 0%       ~ (p=0.963 n=10)
sqs.json                                            352.0 ± 1%   353.0 ± 1%       ~ (p=0.812 n=10)
sqs_no_dd_context.json                              325.5 ± 1%   326.5 ± 0%       ~ (p=0.144 n=10)
stepfunction.json                                   239.0 ± 2%   237.5 ± 1%       ~ (p=0.662 n=10)
geomean                                             363.7        363.6       -0.02%
¹ all samples are equal