Skip to content

Document the new release process for Rustup #3844

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 4 commits into
base: master
Choose a base branch
from
+209 −58
Draft

Document the new release process for Rustup #3844

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
0663c6c
Document the new release process for Rustup
jdno May 28, 2024
75c7fd9
Improve description of stable releases
jdno May 28, 2024
39e643c
Use a dedicated S3 bucket for Rustup
jdno Jun 21, 2024
8926940
Rewrite intro and fix spelling mistake
jdno Jun 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
267 changes: 209 additions & 58 deletions doc/dev-guide/src/release-process.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,66 +1,217 @@
# Making a release
# Release Process

Before making a release, ensure that `rustup-init.sh` is behaving correctly,
This document describes the release process of Rustup and is split into two
sections: the first section explains the release process, while the second
section documents the steps that need to be followed to cut a release.

- [About the Release Process](#about-the-release-process)
- [Historic Context](#historic-context)
- [Beta and Stable Releases](#beta-and-stable-releases)
- [Release Artifacts](#release-artifacts)
- [Automation](#automation)
- [How to Cut a Release](#how-to-cut-a-release)
- [1. Test `rustup-init.sh`](#1-test-rustup-initsh)
- [2. Update Version](#2-update-version)
- [3. Update Checksum](#3-update-checksum)
- [4. Sync `master` to `stable`](#4-sync-master-to-stable)
- [5. Test the `beta` Release](#5-test-the-beta-release)
- [6. Prepare Blog Post for `stable` Release](#6-prepare-blog-post-for-stable-release)
- [7. Request the `stable` Release](#7-request-the-stable-release)
- [8. Tag the `stable` Release](#8-tag-the-stable-release)

## About the Release Process

This section explains the release process for Rustup.

### Historic Context

The Rustup release process has evolved over time. In the past, the process
involved more manual steps and large parts of it were executed locally. This
required careful execution and coordination to avoid accidentally overwriting
existing releases, since the tooling did not implement sanity checks that would
prevent destructive actions.

A full step-by-step guide for the old process can be found in
the [old developer guide](https://github.com/rust-lang/rustup/blob/1cf1b5a6d80c978e0dcaabbce5f10b3861612425/doc/dev-guide/src/release-process.md).
The following summarizes how release artifacts were created and copied around.

In the past, both beta and stable releases were started by merging a new commit
into the `stable` branch in [rust-lang/rustup]. This started a new build on
GitHub Actions that produced release
artifacts ([source](https://github.com/rust-lang/rustup/blob/1cf1b5a6d80c978e0dcaabbce5f10b3861612425/.github/workflows/ci.yaml#L144-L151)),
which were uploaded to `s3://dev-static-rust-lang-org/rustup/dist`. As new
commits were merged into `stable`, they would overwrite the artifacts from prior
builds.

The release artifacts were then copied to the final location by running a script
on a local machine. This script would download the artifacts from the
`dev-static` bucket on S3 and upload them to the `static` bucket. The script
also generated a new manifest with the given version, and uploaded a copy of the
files to an archive directory.

Given that the process was manual, involved copying files to the local machine,
and waiting between creating a `beta` and a `stable` release, there was a risk
of human error. When we had to update the script after four years without a
change, we [decided](https://github.com/rust-lang/rustup/pull/3819) to redesign
and automate the release process.

### Beta and Stable Releases

Rustup can be released to two different environments: `beta` and `stable`. The
main difference between the two is that they use different values for the
`RUSTUP_UPDATE_ROOT` environment variable:

- A beta release is deployed on `https://dev-static.rust-lang.org/rustup`.
- A stable release is deployed on `https://static.rust-lang.org/rustup`.

By switching between those two values, Rustup effectively provides two
"self update channels", making beta testing possible with `rustup self update`.

### Release Artifacts

The release artifacts are produced by CI when a new commit is merged into the
`master` branch, and they are uploaded to the `rustup-builds` bucket on S3.
There, they are put into a folder named after their commit hash, for example the
artifacts for commit `1cf1b5a` would be uploaded to
`s3://rustup-builds/1cf1b5a`.

When a new `beta` release is cut, the artifacts are copied to two new locations
in the `dev-static` bucket on S3:

- One copy is put into an archive named after the version, e.g.
`/rustup/archive/1.0.0`.
- Another copy is put into the `/rustup/dist` folder, which is where clients
look for new versions.

When a new `stable` release is cut, the artifacts are copied to the `static`
bucket on S3, following the same process as the `beta` release:

- One copy is archived into `/rustup/archive/1.0.0`.
- Another copy is put into `/rustup/dist`.

This ensures backwards compatibility with the old release process, while also
ensuring that release artifacts are not overwritten by new builds.

### Automation

The interaction with the release artifacts is fully automated.

First, artifacts are produced automatically by
the [`CI`](https://github.com/rust-lang/rustup/blob/master/.github/workflows/ci.yaml)
job on GitHub Actions when a new commit is merged into the `stable` branch, and
are then automatically uploaded to their respective S3 bucket by the action as
well.

Second, when making a release, the artifacts are copied to their final locations
by the [promote-release] tool. This reduces the risk of human error and ensures
that the release process is consistent and reliable. The tool is also run in a
secure environment on AWS CodeBuild, reducing the risk of leaking sensitive
credentials that would give write access (past) releases.

For a `beta` release, `promote-release` performs the following actions:

1. Query GitHub's API to get the latest commit on the `stable` branch
2. Confirm that `s3://rustup-builds/${commit}` exists
3. Get the new version number from the `stable` branch
1. Download `Cargo.toml` from `stable`
2. Parse the file and read the `version` field
4. Confirm that `s3://dev-static/rustup/archive/${version}` does not exist yet
5. Copy `s3://rustup-builds/${commit}` to `s3://dev-static/rustup/archive/${version}`
6. Copy `s3://rustup-builds/${commit}` to `s3://dev-static/rustup/dist`
7. Generate a new manifest and upload it to `s3://dev-static/rustup/dist`

For a new `stable` release, the process is the same. The only difference is that
the steps 4-6 copy the artifacts to the `static` bucket.

## How to Cut a Release

This section documents the steps that a maintainer should follow to cut a new
release of Rustup.

### 1. Test `rustup-init.sh`

Before cutting a release, ensure that `rustup-init.sh` is behaving correctly,
and that you're satisfied that nothing in the ecosystem is breaking because
of the update. A useful set of things to check includes verifying that
real-world toolchains install okay, and that `rust-analyzer` isn't broken by
the release. While it's not our responsibility if they depend on non-stable
APIs, we should behave well if we can.

As a maintainer, you have two options to choose from when cutting a new
release: a beta release or an official release.
The main difference between the two is that they use different values for
the `RUSTUP_UPDATE_ROOT` environment variable:
- A beta release is deployed on `https://dev-static.rust-lang.org/rustup`.
- An official release is deployed on `https://static.rust-lang.org/rustup`.

By switching between those two values, Rustup effectively provides two "self
update channels", making beta testing possible with `rustup self update`.

Producing the final release artifacts is a bit involved because of the way
Rustup is distributed.
Below is a list of things to be done in order to cut a new [b]eta release
or an official [r]elease:

1. [b/r] In a separate PR:
1. If the version strings in `Cargo.toml`s haven't been updated:
- Decide what the new version number `$VER_NUM` should be.
> **Note:** We always increment the *minor* number unless:
> - A major incompatibility has been introduced in this release:
> increment the *major* number instead.
> - This release is a hotfix because the last one had a defect:
> increment the *patch* number instead.
- Update `Cargo.toml` and `download/Cargo.toml` to have that same new
version number, then run `cargo build` and review `Cargo.lock` changes.
### 2. Update Version

The release process gets metadata from the `Cargo.toml` file, so ensure that
the version number in `Cargo.toml` is correct.

1. If the version strings in `Cargo.toml`s haven't been updated:
- Decide what the new version number `$VER_NUM` should be.
> **Note:** We always increment the *minor* number unless:
> - A major incompatibility has been introduced in this release:
> increment the *major* number instead.
> - This release is a hotfix because the last one had a defect:
> increment the *patch* number instead.
- Update `Cargo.toml` and `download/Cargo.toml` to have that same new
version number, then run `cargo build` and review `Cargo.lock` changes.
If all looks well, make a commit.
2. Update `CHANGELOG.md` accordingly if necessary.
2. [b/r] After merging the PR made in step 1, in a separate PR:
1. Update the commit shasum in `rustup-init.sh` to match the latest commit
on `master`.
3. [b/r] After merging the PR made in step 2, sync `master` to `stable` using
`--ff-only`:
- `git fetch origin master:master`
- `git checkout stable && git merge --ff-only master`
- `git push origin HEAD:stable`
4. [b/r] While you wait for green CI on `stable`, double-check the
functionality of `rustup-init.sh` and `rustup-init` just in case.
5. [b/r] Ensure all of CI is green on the `stable` branch.
Once it is, check through a representative proportion of the builds looking
for the reported version statements to ensure that
we definitely built something cleanly which reports as the right version
number when run `--version`.
6. [r] Make a new PR to the [Rust Blog] adding a new release announcement post.
7. [b/r] Ping someone in the release team to perform the actual release.
They can find instructions in `ci/sync-dist.py`.
> **Note:** Some manual testing occurs here, so hopefully they'll catch
anything egregious in which case abort the change and roll back.
8. [b] Once the beta release has happened, post a new topic named "Seeking beta
testers for Rustup $VER_NUM" on the [Internals Forum].
9. [r] Once the official release has happened, prepare and push a tag on the
latest `stable` commit.
- `git tag -as $VER_NUM -m $VER_NUM` (optionally without `-s` if not GPG
signing the tag)
- `git push origin $VER_NUM`

[Rust Blog]: https://github.com/rust-lang/blog.rust-lang.org
[Internals Forum]: https://internals.rust-lang.org
2. Update `CHANGELOG.md` accordingly if necessary.

Submit the changes in a PR and merge it.

### 3. Update Checksum

After merging the PR in the previous step, update the commit SHA checksum in
`rustup-init.sh` to match the latest commit on `master`. Submit the change in a
PR and merge it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This step sounds surprising to me. If someone updates this to some other commit, how will we notice? Is there an opportunity to adjust the script as part of CI publishing it?

Copy link
Member

@rami3l rami3l May 28, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Mark-Simulacrum This is the most painful point for me personally during the past 2 releases (v1.27.0 and v1.27.1, for which I actually handled the release process), and I have no idea why this have to be the case.

Well, when we migrated to GitHub Merge Queue in v1.27.0, we have almost broken the whole workflow (which relied on merges instead of rebases and thus assumed shasum stability; to keep the coherence with the old workflow, a 2nd PR is required with GHMQ enabled for script traceability, see #3653 (comment) for more context). The current description is the closest I could get from the "spirit" of the original.

I'd love to see this step being automated. As long as the resulting rustup-init.sh script is traceable via the shasum when it's published onto rustup.sh etc., I personally don't see a particular reason we cannot leave a placeholder on master for example.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can look into automating this step, but I would do that after refactoring the release process as-is. This is already quite a big change and I don't want to slow down the implementation or add additional complexity to it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, for sure. Mostly just flagging it since it sounded off.

### 4. Sync `master` to `stable`

After merging the PR in the previous step, sync `master` to `stable` using
`--ff-only`:

```shell
git fetch origin master:master
git checkout stable && git merge --ff-only master
git push origin HEAD:stable
```

This will kick off a new build on GitHub Actions, which will produce the release
artifacts, upload them to `dev-static`, and make the new `beta` release
available at `RUSTUP_UPDATE_ROOT=https://dev-static.rust-lang.org/rustup`.

### 5. Test the `beta` Release

While you wait for green CI on `stable`, double-check the functionality of
`rustup-init.sh` and `rustup-init` just in case.

Ensure all of CI is green on the `stable` branch. Once it is, check through a
representative proportion of the builds looking for the reported version
statements to ensure that we definitely built something cleanly which reports as
the right version number when run `--version`.

Once the beta release has happened, post a new topic named "Seeking beta testers
for Rustup $VER_NUM" on the [Internals Forum].

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We've automated this for rustc, maybe can reuse that for rustup...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy to create a tracking issue for this so that we can look into it once we've refactored the tooling.

### 6. Prepare Blog Post for `stable` Release

Make a new PR to the [Rust Blog] adding a new release announcement post.

### 7. Request the `stable` Release

Ping someone in the release team to perform the `stable` release.

They will have to start a new CodeBuild job on AWS to run the `promote-release`
tool for Rustup.

### 8. Tag the `stable` Release

Once the official release has happened, prepare and push a tag on the latest
`stable` commit.

- `git tag -as $VER_NUM -m $VER_NUM` (optionally without `-s` if not GPG
signing the tag)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we do this in promote release? We do for rust and cargo (also always signing the commits with the official key).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Mark-Simulacrum Will be cool. So far I have to ping t-release before bedtime, and manually do the tagging the next morning 😓

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand correctly what this mean, I agree it would be cool.

Currently I have an automated process to make sure the script that I download from rustup.rs is the same as a specific tagged release from this repository. Until a few hours ago it was v1.27.1. Now it should be a commit between e2d9e7e and master that would be tagged v1.28.0. But there is no such tag yet.

If I understand correctly, this tagging step is supposed to happen after the file is distributed. And this comment suggests to do it at the same time. Is that right?

- `git push origin $VER_NUM`

[internals forum]: https://internals.rust-lang.org
[promote-release]: https://github.com/rust-lang/promote-release
[rust-lang/rustup]:https://github.com/rust-lang/rustup
[rust blog]: https://github.com/rust-lang/blog.rust-lang.org

Loading