Skip to content

Update book for backport, release and sync #12762

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from
Draft

Update book for backport, release and sync #12762

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 9 additions & 7 deletions book/src/development/infrastructure/release.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,15 +51,14 @@ git commit -m "Bump Clippy version -> 0.1.XY" **/*Cargo.toml
For both updating the `beta` and the `stable` branch, the first step is to find
the Clippy commit of the last Clippy sync done in the respective Rust branch.

Running the following commands _in the Rust repo_ will get the commit for the
specified `<branch>`:
Running the following command will get the commit for the specified branch:

```bash
git switch <branch>
SHA=$(git log --oneline -- src/tools/clippy/ | grep -o "Merge commit '[a-f0-9]*' into .*" | head -1 | sed -e "s/Merge commit '\([a-f0-9]*\)' into .*/\1/g")
$ SHA=$(cargo dev release commit /path/to/rust <branch>)
```

Where `<branch>` is one of `stable`, `beta`, or `master`.
Where the `/path/to/rust` is a relative path to a Rust clone and the `<branch>`
is one of `stable`, `beta`, or `master`.

## Update the `beta` branch

Expand Down Expand Up @@ -88,9 +87,12 @@ git push upstream stable
After updating the `stable` branch, tag the HEAD commit and push it to the
Clippy repo.

> Note: Only push the tag, once the Deploy GitHub action of the `beta` branch is
> finished. Otherwise the deploy for the tag might fail.

```bash
git tag rust-1.XX.0 # XX should be exchanged with the corresponding version
git push upstream rust-1.XX.0 # `upstream` is the `rust-lang/rust-clippy` remote
git tag rust-1.XY.0 # XY should be exchanged with the corresponding version
git push upstream rust-1.XY.0 # `upstream` is the `rust-lang/rust-clippy` remote
```

After this, the release should be available on the Clippy [tags page].
Expand Down
133 changes: 60 additions & 73 deletions book/src/development/infrastructure/sync.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
Clippy currently gets built with a pinned nightly version.

In the `rust-lang/rust` repository, where rustc resides, there's a copy of
Clippy that compiler hackers modify from time to time to adapt to changes in the
Clippy that compiler devs modify from time to time to adapt to changes in the
unstable API of the compiler.

We need to sync these changes back to this repository periodically, and the
Expand All @@ -15,94 +15,81 @@ done in a bi-weekly basis if there's no urgent changes. This is done starting on
the day of the Rust stable release and then every other week. That way we
guarantee that we keep this repo up to date with the latest compiler API, and
every feature in Clippy is available for 2 weeks in nightly, before it can get
to beta. For reference, the first sync following this cadence was performed the
to beta. For reference, the first sync following this cadence was performed on
2020-08-27.

This process is described in detail in the following sections. For general
information about `subtree`s in the Rust repository see [the rustc-dev-guide][subtree].
This process is described in detail in the following sections.

[subtree]: https://rustc-dev-guide.rust-lang.org/external-repos.html#external-dependencies-subtree
## Installing `josh-proxy`

## Patching git-subtree to work with big repos
The sync is done with [JOSH] and fully scripted with `cargo dev sync`. The only
requirement is to install the `josh-proxy` binary from GitHub

Currently, there's a bug in `git-subtree` that prevents it from working properly
with the [`rust-lang/rust`] repo. There's an open PR to fix that, but it's
stale. Before continuing with the following steps, we need to manually apply
that fix to our local copy of `git-subtree`.
<!-- FIXME: Change to a release version once >r23.12.04 is released -->

You can get the patched version of `git-subtree` from [here][gitgitgadget-pr].
Put this file under `/usr/lib/git-core` (making a backup of the previous file)
and make sure it has the proper permissions:

```bash
sudo cp --backup /path/to/patched/git-subtree.sh /usr/lib/git-core/git-subtree
sudo chmod --reference=/usr/lib/git-core/git-subtree~ /usr/lib/git-core/git-subtree
sudo chown --reference=/usr/lib/git-core/git-subtree~ /usr/lib/git-core/git-subtree
```sh
$ RUSTFLAGS="--cap-lints warn" cargo +stable install josh-proxy --git https://github.com/josh-project/josh
```

> _Note:_ The first time running `git subtree push` a cache has to be built.
> This involves going through the complete Clippy history once. For this you
> have to increase the stack limit though, which you can do with `ulimit -s
> 60000`. Make sure to run the `ulimit` command from the same session you call
> git subtree.
[JOSH]: https://josh-project.github.io/josh/

## Performing the sync from [`rust-lang/rust`] to Clippy

> _Note:_ If you are a Debian user, `dash` is the shell used by default for
> scripts instead of `sh`. This shell has a hardcoded recursion limit set to
> 1,000. In order to make this process work, you need to force the script to run
> `bash` instead. You can do this by editing the first line of the `git-subtree`
> script and changing `sh` to `bash`.
Doing the sync now is just running

> Note: The following sections assume that you have set up remotes following the
> instructions in [defining remotes].
```
$ cargo dev sync pull
```

[gitgitgadget-pr]: https://github.com/gitgitgadget/git/pull/493
[defining remotes]: release.md#defining-remotes
This command will update the nightly toolchain in the `rust-toolchain` file and
will pull the changes from the Rust repository.

## Performing the sync from [`rust-lang/rust`] to Clippy
If there should be merge conflicts, resolve them now and commit with the message
`Merge from rustc`.[^1]

Here is a TL;DR version of the sync process (all the following commands have
to be run inside the `rust` directory):

1. Clone the [`rust-lang/rust`] repository or make sure it is up-to-date.
2. Checkout the commit from the latest available nightly. You can get it using
`rustup check`.
3. Sync the changes to the rust-copy of Clippy to your Clippy fork:
```bash
# Be sure to either use a net-new branch, e.g. `rustup`, or delete the branch beforehand
# because changes cannot be fast forwarded and you have to run this command again.
git subtree push -P src/tools/clippy clippy-local rustup
```

> _Note:_ Most of the time you have to create a merge commit in the
> `rust-clippy` repo (this has to be done in the Clippy repo, not in the
> rust-copy of Clippy):
```bash
git fetch upstream # assuming upstream is the rust-lang/rust remote
git switch rustup
git merge upstream/master --no-ff
```
> Note: This is one of the few instances where a merge commit is allowed in
> a PR.
4. Bump the nightly version in the Clippy repository by running these commands:
```bash
cargo dev sync update_nightly
git commit -m "Bump nightly version -> YYYY-MM-DD" rust-toolchain.toml clippy_utils/README.md
```
5. Open a PR to `rust-lang/rust-clippy` and wait for it to get merged (to
accelerate the process ping the `@rust-lang/clippy` team in your PR and/or
ask them in the [Zulip] stream.)
> Note: If the version tests fail, refer to [bump version] in the release
> documentation.

Open a PR to `rust-lang/rust-clippy` and if you are a Clippy maintainer, you can
`r+` the PR yourself. If not, change `r? @ghost` to `r? clippy` and a Clippy
maintainer will get assigned. To accelerate the process ping the Clippy team on
[Zulip].


[bump version]: release.md#bump-version
[Zulip]: https://rust-lang.zulipchat.com/#narrow/stream/clippy
[`rust-lang/rust`]: https://github.com/rust-lang/rust

[^1]: The message is not really important, but consistency is nice.

## Performing the sync from Clippy to [`rust-lang/rust`]

All the following commands have to be run inside the `rust` directory.
The other direction is done by running

```
$ cargo dev sync push /path/to/rust --user <GitHub-name>
```

Where the `/path/to/rust` is a relative path to a Rust clone and the
`<GitHub-name>` is your GitHub user name. This is required for pushing the sync
to GitHub and opening a PR.

If everything went right, there will be a GitHub link that has to be used to
open the sync PR in the Rust repository. The PR description must look like this:

```
Clippy subtree update

r? @ghost

1. Make sure you have checked out the latest `master` of `rust-lang/rust`.
2. Sync the `rust-lang/rust-clippy` master to the rust-copy of Clippy:
```bash
git switch -c clippy-subtree-update
git subtree pull -P src/tools/clippy clippy-upstream master
```
3. Open a PR to [`rust-lang/rust`]
Sync from Clippy commit: rust-lang/rust-clippy@<sha1>
```

The title must be kept as is, to [tell triagebot] that this is a sync PR.

The second line must be kept as is, to [find the Clippy commit] during a
release.

[find the Clippy commit]: release.md#find-the-clippy-commit
[tell triagebot]: https://github.com/rust-lang/rust/pull/114157

[`rust-lang/rust`]: https://github.com/rust-lang/rust