Move off rust-lang-ci

[Kobzol]

We would like to get rid of the rust-lang-ci organization, which should no longer be needed:

• We can now configure secrets on GH per branch, so we can configure secrets only for the auto and try branches, not for all CI.
• We get 500 concurrent jobs across all repos in the rust-lang organization, so that should be enough to run all CI in a single org.
• We no longer use self-hosted runners.

The following (probably non-exhaustive) list below tracks what needs to be done to get rid of rust-lang-ci:

• Configure secrets for the auto and try branches on rust-lang/rust.
  • This could be done either using GitHub environments or using AWS OIDC.
  • Once we get rid of homu, we could get rid of all secrets and use AWS OIDCS for everything.
  • Set GitHub environments secrets in rust-lang-ci. Edit r-l/r to use GitHub environments secrets (maybe the secrets can be called differently from the repository secrets so that they don't collide) and test a dry run job. The goal is to make sure that the CI works well with environments before the migration. Consider doing the migration to environment secrets before moving off rust-lang-ci. I.e. making rust-lang-ci work with environment secrets instead of repository secrets.
  • Add bors environment to CI rust#141323
• Make sure that the homu bors GitHub account has access to the auto and try branches on rust-lang/rust.
  • Make sure that homu can actually force-push these branches. (Move off rust-lang-ci #188 (comment))
• Create try-perf and perf-tmp branches/branch protections in rust-lang/rust.
  • Add support for rust-timer merge bot and try-perf/perf-tmp branch protections team#1828
  • Make sure that the rustbot rust-timer GitHub account can force-push these branches. (Move off rust-lang-ci #188 (comment))
  • (Maybe move the unrolling logic from rustc-perf into rustbot or bots? But that's unrelated.)
• Switch CI to use Docker ghcr.io caches from the rust-lang organization
  • Use Docker cache from the current repository rust#141280
• Test if CI works on the try and auto branches and if it can access environment secrets.
• Modify codebuild to account for the repository switch.
• Reconfigure homu to avoid using the rust-lang-ci fork (configured here) and redeploy homu - probably the scariest part.
  • Move bors CI jobs from rust-lang-ci to rust-lang homu#233 - this is the PR that will do the actual move off rust-lang-ci.
• Update downstream projects
  • https://github.com/rust-lang/rustc-perf
    • [do not merge] Use rust-lang instead of rust-lang-ci for all commits rustc-perf#2023
    • I think that we can proactively switch to rust-lang here. Old try build links will not work (because these will be only in rust-lang-ci), but old merge links should work, as the merge commits should be both in rust-lang and rust-lang-ci.
  • https://github.com/rust-lang/team
    • https://github.com/rust-lang/team/blob/f97d54dfe99a73bc3fe13cda348a77fac3feaa9b/teams/infra.toml#L33
  • https://github.com/rust-lang/cargo-bisect-rustc
    • Use rust-lang/rust instead of rust-lang-ci/rust for unrolled rollup build commit URLs cargo-bisect-rustc#381
  • https://github.com/rust-lang/simpleinfra
    • https://github.com/rust-lang/simpleinfra/blob/23bf177ffd5200f995369b1ce40c59d2cd4a0c97/terraform/monitorbot/app.tf#L17
    • https://github.com/rust-lang/simpleinfra/blob/23bf177ffd5200f995369b1ce40c59d2cd4a0c97/terraform/rustc-ci/environments.tf#L9
• revert Move dist-x86_64-linux CI job to GitHub temporarily rust#141388 AND change the try build to run on codebuild as well https://github.com/rust-lang/rust/blob/1d679446b01e65f9bc9ae609d0ae1e4a9c0ccaa3/src/ci/github-actions/jobs.yml#L131

This idea was discussed on Zulip.

[pietroalbini]

A missing thing is that rustup also refers to the docker images built by rust-lang/rust's CI.

[Kobzol]

That's the same thing as Switch CI to use Docker ghcr.io caches from the rust-lang organization; rust's CI writes the image tag into a file stored at S3, and rustup downloads it. Whatever is written by Rust's CI into that file will determine from where rustup downloads the Docker images (be it rust-lang or rust-lang-ci's ghcr.io hub).

[Mark-Simulacrum]

Just to check, even if we did use self-hosted runners I think those are also fine in a single org now (GitHub upstream made changes for that), right?

[pietroalbini]

Just to check, even if we did use self-hosted runners I think those are also fine in a single org now (GitHub upstream made changes for that), right?

Yes. If the runners are placed in a runner group, it's possible to restrict which workflows can access them. We could then restrict them to rust-lang/rust/.github/workflows/ci.yml@auto to only allow them to execute ci.yml on the auto branch.

[marcoieni]

Configure secrets for the auto and try branches on rust-lang/rust.

It looks like I need to create a new GitHub Environment to configure the secret. Should I call this environment bors?

Make sure that the homu bors GitHub account has access to the auto and try branches on rust-lang/rust.

Added these two branch protection rules to rust-lang/rust:

both have the following settings (which are the same of master, too):

I checked this box in the first comment of this issue 👍

[Kobzol]

Couldn't it be a repo secret, as opposed to environment secret? But an env. secret is also fine, bors sounds like a reasonable name.

[marcoieni]

I thought we wanted to configure these secrets only for the try and auto branches.

A repository secret can be read from any branch, right?

[Kobzol]

Right. Makes sense, environment it is, then :) I suppose that the lack of environment/per-branch secrets was one of the reasons why rust-lang-ci was needed in the first place, originally.

[marcoieni]

I created the environment and added the first secret

[marcoieni]

Secrets

• DATADOG_API_KEY: I copied it from the datadog dashboard. There were many API keys but I copied the one that was created when the rust-lang-ci secret was set.
• TOOLSTATE_REPO_ACCESS_TOKEN: How do I create this token? There are some instructions here but they are probably outdated and I wonder if "public_repo" permission is still enough. The token is used to create issues here and write PR comments here. I found in 1password a "rust-toolstate 2021-02-22" token in the rust-highfive user. I can try to copy it. Apparently the token is used to push in https://github.com/rust-lang-nursery/rust-toolstate
• there are also various aws related secrets which I'm investigating if I can read them somehow from terraform. If not, I will investigate what permissions they should have and create new ones. Of course let me know if you remember more details about the aws secrets 👍

[marcoieni]

I found the iam users for the aws keys 👍

I will create a new access key for these two users.
I'll delete old unused keys because we can have at maximum 2 keys per user.
For example, one key in the artifacts account was never used:

[marcoieni]

I didn't realize that the keys I deleted where created from terraform in the simpleinfra/terraform/rustc-ci module. The keys deletion resulted in a terraform plan diff:

Terraform detected the following changes made outside of Terraform since the last "terraform apply" which may have affected this plan:

  # module.public.module.artifacts_user.aws_iam_access_key.ci has been deleted
  - resource "aws_iam_access_key" "ci" {
      - id                   = "AKIA46X5W6CZDRNFRBJK" -> null
      - secret               = (sensitive value) -> null
        # (4 unchanged attributes hidden)
    }

  # module.public.module.caches_user.aws_iam_access_key.ci has been deleted
  - resource "aws_iam_access_key" "ci" {
      - id                   = "AKIA46X5W6CZGPEWL7UN" -> null
      - secret               = (sensitive value) -> null
        # (4 unchanged attributes hidden)
    }


Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan may include actions to undo or
respond to these changes.

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create
  ~ update in-place
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # module.public.module.artifacts_user.aws_iam_access_key.ci will be created
  + resource "aws_iam_access_key" "ci" {
      + create_date                    = (known after apply)
      + encrypted_secret               = (known after apply)
      + encrypted_ses_smtp_password_v4 = (known after apply)
      + id                             = (known after apply)
      + key_fingerprint                = (known after apply)
      + secret                         = (sensitive value)
      + ses_smtp_password_v4           = (sensitive value)
      + status                         = "Active"
      + user                           = "ci--rust-lang--rust--artifacts"
    }

  # module.public.module.artifacts_user.aws_iam_user.ci will be updated in-place
  ~ resource "aws_iam_user" "ci" {
        id            = "ci--rust-lang--rust--artifacts"
        name          = "ci--rust-lang--rust--artifacts"
      ~ tags          = {
          - "AKIA46X5W6CZAWZRBC7B" = "Created by Marco when migrating from rust-lang-ci org. Stored in GitHub Actions environments secrets." -> null
        }
      ~ tags_all      = {
          - "AKIA46X5W6CZAWZRBC7B" = "Created by Marco when migrating from rust-lang-ci org. Stored in GitHub Actions environments secrets." -> null
        }
        # (4 unchanged attributes hidden)
    }

  # module.public.module.artifacts_user.github_actions_secret.aws_access_key_id must be replaced
-/+ resource "github_actions_secret" "aws_access_key_id" {
      ~ created_at      = "2021-10-17 14:40:34 +0000 UTC" -> (known after apply)
      ~ id              = "rust:ARTIFACTS_AWS_ACCESS_KEY_ID" -> (known after apply)
      ~ plaintext_value = (sensitive value) # forces replacement
      ~ updated_at      = "2021-10-17 14:40:34 +0000 UTC" -> (known after apply)
        # (2 unchanged attributes hidden)
    }

  # module.public.module.artifacts_user.github_actions_secret.aws_secret_access_key must be replaced
-/+ resource "github_actions_secret" "aws_secret_access_key" {
      ~ created_at      = "2021-10-17 14:40:32 +0000 UTC" -> (known after apply)
      ~ id              = "rust:ARTIFACTS_AWS_SECRET_ACCESS_KEY" -> (known after apply)
      ~ plaintext_value = (sensitive value) # forces replacement
      ~ updated_at      = "2021-10-17 14:40:32 +0000 UTC" -> (known after apply)
        # (2 unchanged attributes hidden)
    }

  # module.public.module.caches_user.aws_iam_access_key.ci will be created
  + resource "aws_iam_access_key" "ci" {
      + create_date                    = (known after apply)
      + encrypted_secret               = (known after apply)
      + encrypted_ses_smtp_password_v4 = (known after apply)
      + id                             = (known after apply)
      + key_fingerprint                = (known after apply)
      + secret                         = (sensitive value)
      + ses_smtp_password_v4           = (sensitive value)
      + status                         = "Active"
      + user                           = "ci--rust-lang--rust--caches"
    }

  # module.public.module.caches_user.github_actions_secret.aws_access_key_id must be replaced
-/+ resource "github_actions_secret" "aws_access_key_id" {
      ~ created_at      = "2021-10-17 14:40:34 +0000 UTC" -> (known after apply)
      ~ id              = "rust:CACHES_AWS_ACCESS_KEY_ID" -> (known after apply)
      ~ plaintext_value = (sensitive value) # forces replacement
      ~ updated_at      = "2021-10-17 14:40:34 +0000 UTC" -> (known after apply)
        # (2 unchanged attributes hidden)
    }

  # module.public.module.caches_user.github_actions_secret.aws_secret_access_key must be replaced
-/+ resource "github_actions_secret" "aws_secret_access_key" {
      ~ created_at      = "2021-10-17 14:40:32 +0000 UTC" -> (known after apply)
      ~ id              = "rust:CACHES_AWS_SECRET_ACCESS_KEY" -> (known after apply)
      ~ plaintext_value = (sensitive value) # forces replacement
      ~ updated_at      = "2021-10-17 14:40:32 +0000 UTC" -> (known after apply)
        # (2 unchanged attributes hidden)
    }

Plan: 6 to add, 1 to change, 4 to destroy.

I discussed this with JD.

Learnings

• The GitHub CI was using the legacy keys. The once I deleted are the ones created from the gha-iam-user module

Questions

• why the CI didn't switch to the new keys?
• In the rust-lang-ci/rust GitHub Actions secrets we have four AWS_SECRET_ACCESS_KEY_{ID} secrets but it seems we only use two of them. Am I missing something and maybe we need multiple keys because we have different permissions associated to these keys? Why are we using the {ID} technique instead of using the secret ARTIFACTS_AWS_SECRET_ACCESS_KEY (which seems unused): imo it's a simpler approach.

Plan

Here's how we want to move forward:

1. Leave the terraform module simpleinfra/terraform/rustc-ci dirty. I.e. don't touch it for now.
2. Create a new terragrunt module where I copy paste the parts I need for rust-lang/rust (e.g. I don't need the difference between repo and source anymore). I will use rustc-ci--rust-lang--rust as a iam_prefix to avoid conflicts with existing resources.
3. import the buckets into the terragrunt state (with terraform import). Instructions here.
4. Apply the module to the bors-kindergarden repo to test if the access keys are applied correctly. Jakub, does it make sense to treat bors-kindergarden as the "staging" environment of the rust CI?
   EDIT: I'm not sure this makes sense as bors-kindergarden should test bors, not the entire rust ci (with the caches and artifacts).
5. Apply the same module to rust-lang/rust
6. Migrate rust-lang-ci
7. Delete everything from the simpleinfra/terraform/rustc-ci module, except the s3 bucket from this module.

Wdyt?