deploy datadog to kops eks cluster and upgrade argocd

[upodroid]

I upgraded argocd and configured it to assume an AWS role that grants access to the EKS clusters and I connected the kops EKS cluster to ArgoCD

Part of #5170

Manual changes I need to reconcile:

• EKS access policy changes to the kops cluster

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: upodroid

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [upodroid]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-infra-ci-robot]

Ran Plan for dir: infra/gcp/terraform/k8s-infra-prow-build workspace: default

Plan Error

Show Output

running 'sh -c' '/atlantis/bin/terraform1.12.2 init -input=false -upgrade' in '/atlantis/repos/kubernetes/k8s.io/8180/default/infra/gcp/terraform/k8s-infra-prow-build': exit status 1
Initializing the backend...

Successfully configured the backend "gcs"! Terraform will automatically
use this backend unless the backend configuration changes.
Upgrading modules...
Downloading registry.terraform.io/terraform-google-modules/iam/google 8.1.0 for iam...
- iam in .terraform/modules/iam/modules/projects_iam
- iam.helper in .terraform/modules/iam/modules/helper
- project in ../modules/gke-project
- prow_build_cluster in ../modules/gke-cluster
- prow_build_nodepool_c4_highmem_8_localssd in ../modules/gke-nodepool
- prow_build_nodepool_c4a_highmem_8_localssd in ../modules/gke-nodepool
- prow_build_nodepool_c4d_highmem_8_localssd in ../modules/gke-nodepool
- prow_build_nodepool_n1_highmem_8_localssd in ../modules/gke-nodepool
Downloading git::https://github.com/GoogleCloudPlatform/cloud-foundation-fabric.git?ref=v39.0.0&depth=1 for sig_node_node_pool_1_n4_highmem_8...
- sig_node_node_pool_1_n4_highmem_8 in .terraform/modules/sig_node_node_pool_1_n4_highmem_8/modules/gke-nodepool
- workload_identity_service_accounts in ../modules/workload-identity-service-account
╷
│ Error: Duplicate resource "google_iam_workload_identity_pool_provider" configuration
│ 
│   on iam.tf line 65:
│   65: resource "google_iam_workload_identity_pool_provider" "eks_cluster" {
│ 
│ A google_iam_workload_identity_pool_provider resource named "eks_cluster"
│ was already declared at iam.tf:48,1-68. Resource names must be unique per
│ type in each module.
╵

[k8s-infra-ci-robot]

Ran Plan for dir: infra/gcp/terraform/k8s-infra-prow-build workspace: default

Show Output

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
+ create
~ update in-place

Terraform will perform the following actions:

  # google_iam_workload_identity_pool_provider.eks_kops will be created
+ resource "google_iam_workload_identity_pool_provider" "eks_kops" {
      + attribute_mapping                  = {
          + "google.subject" = "assertion.sub"
        }
      + display_name                       = "kops"
      + id                                 = (known after apply)
      + name                               = (known after apply)
      + project                            = "k8s-infra-prow-build"
      + state                              = (known after apply)
      + workload_identity_pool_id          = "prow-eks"
      + workload_identity_pool_provider_id = "kops"

      + oidc {
          + allowed_audiences = [
              + "sts.googleapis.com",
            ]
          + issuer_uri        = "https://oidc.eks.us-east-2.amazonaws.com/id/7283E85C59E9C4129CFD07BAC9378D44"
        }
    }

  # google_vmwareengine_network_peering.gvce_peering will be updated in-place
~ resource "google_vmwareengine_network_peering" "gvce_peering" {
      ~ export_custom_routes_with_public_ip = false -> true
        id                                  = "projects/k8s-infra-prow-build/locations/global/networkPeerings/peer-with-gcve-project"
      ~ import_custom_routes_with_public_ip = false -> true
        name                                = "peer-with-gcve-project"
        # (13 unchanged attributes hidden)
    }

  # module.iam.google_project_iam_binding.project_iam_authoritative["default--roles/secretmanager.secretAccessor"] will be updated in-place
~ resource "google_project_iam_binding" "project_iam_authoritative" {
        id      = "k8s-infra-prow-build/roles/secretmanager.secretAccessor"
      ~ members = [
          - "principal://iam.googleapis.com/projects/180382678033/locations/global/workloadIdentityPools/k8s-infra-prow-build-trusted.svc.id.goog/subject/ns/external-secrets/sa/external-secrets",
            # (2 unchanged elements hidden)
        ]
        # (3 unchanged attributes hidden)
    }

  # module.prow_build_nodepool_c4_highmem_8_localssd.google_container_node_pool.node_pool will be updated in-place
~ resource "google_container_node_pool" "node_pool" {
        id                          = "projects/k8s-infra-prow-build/locations/us-central1/clusters/prow-build/nodePools/pool6-20250327232037500200000001"
        name                        = "pool6-20250327232037500200000001"
        # (10 unchanged attributes hidden)

      ~ autoscaling {
          ~ max_node_count       = 25 -> 80
            # (4 unchanged attributes hidden)
        }

        # (3 unchanged blocks hidden)
    }

Plan: 1 to add, 3 to change, 0 to destroy.

• ▶️ To apply this plan, comment:

  atlantis apply -d infra/gcp/terraform/k8s-infra-prow-build

• 🚮 To delete this plan and lock, click here
• 🔁 To plan this project again, comment:

  atlantis plan -d infra/gcp/terraform/k8s-infra-prow-build

Plan: 1 to add, 3 to change, 0 to destroy.

---

• ⏩ To apply all unapplied plans from this Pull Request, comment:

  atlantis apply

• 🚮 To delete all plans and locks from this Pull Request, comment:

  atlantis unlock

[ameukam]

I don't advise to run anything in the kOps cluster. This cluster is supposed to be gone by EoY. All prowjobs should run the eks-prow-build-cluster.

[upodroid]

I'm trying the dd deployment here before running it on the main cluster. We'll clean it up before decommissioning the cluster at the end of the year.

[k8s-infra-ci-robot]

Ran Plan for dir: infra/gcp/terraform/k8s-infra-prow-build workspace: default

Show Output

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
+ create
~ update in-place

Terraform will perform the following actions:

  # google_iam_workload_identity_pool_provider.eks_kops will be created
+ resource "google_iam_workload_identity_pool_provider" "eks_kops" {
      + attribute_mapping                  = {
          + "google.subject" = "assertion.sub"
        }
      + display_name                       = "kops"
      + id                                 = (known after apply)
      + name                               = (known after apply)
      + project                            = "k8s-infra-prow-build"
      + state                              = (known after apply)
      + workload_identity_pool_id          = "prow-eks"
      + workload_identity_pool_provider_id = "kops"

      + oidc {
          + allowed_audiences = [
              + "sts.googleapis.com",
            ]
          + issuer_uri        = "https://oidc.eks.us-east-2.amazonaws.com/id/7283E85C59E9C4129CFD07BAC9378D44"
        }
    }

  # google_vmwareengine_network_peering.gvce_peering will be updated in-place
~ resource "google_vmwareengine_network_peering" "gvce_peering" {
      ~ export_custom_routes_with_public_ip = false -> true
        id                                  = "projects/k8s-infra-prow-build/locations/global/networkPeerings/peer-with-gcve-project"
      ~ import_custom_routes_with_public_ip = false -> true
        name                                = "peer-with-gcve-project"
        # (13 unchanged attributes hidden)
    }

  # module.iam.google_project_iam_binding.project_iam_authoritative["default--roles/secretmanager.secretAccessor"] will be updated in-place
~ resource "google_project_iam_binding" "project_iam_authoritative" {
        id      = "k8s-infra-prow-build/roles/secretmanager.secretAccessor"
      ~ members = [
          - "principal://iam.googleapis.com/projects/180382678033/locations/global/workloadIdentityPools/k8s-infra-prow-build-trusted.svc.id.goog/subject/ns/external-secrets/sa/external-secrets",
            # (2 unchanged elements hidden)
        ]
        # (3 unchanged attributes hidden)
    }

  # module.prow_build_nodepool_c4_highmem_8_localssd.google_container_node_pool.node_pool will be updated in-place
~ resource "google_container_node_pool" "node_pool" {
        id                          = "projects/k8s-infra-prow-build/locations/us-central1/clusters/prow-build/nodePools/pool6-20250327232037500200000001"
        name                        = "pool6-20250327232037500200000001"
        # (10 unchanged attributes hidden)

      ~ autoscaling {
          ~ max_node_count       = 25 -> 80
            # (4 unchanged attributes hidden)
        }

        # (3 unchanged blocks hidden)
    }

Plan: 1 to add, 3 to change, 0 to destroy.

• ▶️ To apply this plan, comment:

  atlantis apply -d infra/gcp/terraform/k8s-infra-prow-build

• 🚮 To delete this plan and lock, click here
• 🔁 To plan this project again, comment:

  atlantis plan -d infra/gcp/terraform/k8s-infra-prow-build

Plan: 1 to add, 3 to change, 0 to destroy.

---

• ⏩ To apply all unapplied plans from this Pull Request, comment:

  atlantis apply

• 🚮 To delete all plans and locks from this Pull Request, comment:

  atlantis unlock

[ameukam]

/lgtm

[k8s-infra-ci-robot]

Locks and plans deleted for the projects and workspaces modified in this pull request:

• dir: infra/gcp/terraform/k8s-infra-prow-build workspace: default