Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: automate creation of GitHub releases and tags #2027

Closed
wants to merge 8 commits into from
Closed
Show file tree
Hide file tree
Changes from 7 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 52 additions & 0 deletions .github/workflows/release-check.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
name: Release Checker

on:
pull_request_target:
paths:
- "**/Cargo.toml"
types: [ opened, synchronize, reopened, labeled, unlabeled ]
workflow_dispatch:

permissions:
contents: write
pull-requests: write

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:
release-check:
uses: ipdxco/unified-github-workflows/.github/workflows/release-check.yml@v1.0
with:
sources: |
[
"fvm/Cargo.toml",
"testing/integration/Cargo.toml",
"ipld/amt/Cargo.toml",
"ipld/bitfield/Cargo.toml",
"ipld/blockstore/Cargo.toml",
"ipld/car/Cargo.toml",
"ipld/encoding/Cargo.toml",
"ipld/hamt/Cargo.toml",
"ipld/kamt/Cargo.toml",
"sdk/Cargo.toml",
"shared/Cargo.toml"
]
separator: "@"
cargo-publish:
needs: [release-check]
if: needs.release-check.outputs.json != '[]'
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
source: ${{ fromJSON(needs.release-check.outputs.json).*.source }}
steps:
- uses: actions/checkout@v4
- env:
source: ${{ matrix.source }}
run: |
pushd "$(dirname $source)"
cargo publish --dry-run
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this will fail because it won't publish in the right order.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I.e., if we cut a release of the sdk + shared at the same time (which we always do), we'd need to push a release of shared to some fake registry for the release of the sdk to pass a cargo publish dry run (I think).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I see what you mean now! Yes, that would definitely show up as a failure in the proposed process. Would it make sense to split the release in that case, as in, you first release shared and only follow up with sdk release after shared is fully out?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's going to make releases take even longer, which defeats the point of having automation.

I think the answer here is manual dry-run and it's not going to be pretty.

  1. Figure out which crates need to be published.
  2. Either determine the correct order, or hard-code it in a config.
  3. Run cargo vendor --versioned-dirs, creating a .cargo/config.toml file to respect the vendor directory.
  4. In publish order (reverse dependency order):
  5. Run cargo package.
  6. Copy the package (the extracted one) out of target/packages into vendor.
  7. Generate the .cargo-checksum.json file for that manually vendored package. This is the most annoying step, but shouldn't be hard with find + jq...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is very helfpul, thank you. I included this in the improvements section of the release automation doc in a3fb5a9. I also described what happens inside the current state of the automation in more detail.

I think the big decision here is whether the partial automation would be useful here, or whether we should wait until we can fully handle cargo publishes with respect to the dependency graph.

cc @BigLep

popd
36 changes: 36 additions & 0 deletions .github/workflows/releaser.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
name: Releaser

on:
push:
paths:
- "**/Cargo.toml"
workflow_dispatch:

permissions:
contents: write

concurrency:
group: ${{ github.workflow }}-${{ github.sha }}
cancel-in-progress: true

jobs:
releaser:
uses: ipdxco/unified-github-workflows/.github/workflows/releaser.yml@v1.0
with:
sources: |
[
"fvm/Cargo.toml",
"testing/integration/Cargo.toml",
"ipld/amt/Cargo.toml",
"ipld/bitfield/Cargo.toml",
"ipld/blockstore/Cargo.toml",
"ipld/car/Cargo.toml",
"ipld/encoding/Cargo.toml",
"ipld/hamt/Cargo.toml",
"ipld/kamt/Cargo.toml",
"sdk/Cargo.toml",
"shared/Cargo.toml"
]
separator: "@"
secrets:
UCI_GITHUB_TOKEN: ${{ secrets.UCI_GITHUB_TOKEN }}
48 changes: 43 additions & 5 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,11 +76,7 @@ Once the release is prepared, it'll go through a review:
2. Make sure that we're correctly following semver.
3. Make sure that we're not missing anything in the changelogs.

Finally, an [FVM "owner"](https://github.com/orgs/filecoin-project/teams/fvm-crate-owners/members) will:

1. Merge the release PR to master.
2. For each released crate, create a git tag: `crate_name@crate_version`.
3. Run `cargo publish` for each released crate (in dependency order).
Finally, an [FVM "owner"](https://github.com/orgs/filecoin-project/teams/fvm-crate-owners/members) will follow the [release process](#release-process).

### Crate Dependency Graph

Expand Down Expand Up @@ -112,3 +108,45 @@ If that works, proceed with releasing these crates.
[chat]: https://docs.filecoin.io/about-filecoin/chat-and-discussion-forums/#chat
[discuss]: https://github.com/filecoin-project/fvm-specs/discussions
[issue]: https://github.com/filecoin-project/ref-fvm/issues

### Release Process

This section describes the automated parts of the release process and the manual steps FVM owners need to take.

#### Current State

1. On a release pull request creation, a [Release Checker](.github/workflows/release-check.yml) workflow will run. It will perform the following actions:
galargh marked this conversation as resolved.
Show resolved Hide resolved
1. Extract the version from the modified `Cargo.toml` files. Process each crate in the workspace **independently**.
galargh marked this conversation as resolved.
Show resolved Hide resolved
2. Check if a git tag for the version, using the `crate_name@version` as the pattern, already exists. Continue only if it does not.
3. Create a draft GitHub release with the version as the tag.
4. Comment on the pull request with a link to the draft release.
5. Run `cargo publish --dry-run` for the crate for which the release is proposed.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was explicitly requested here - that's why it was added. But it's not clear to me whether we want it at all without being able to incorporate dependency graph knowledge into the dry run publish process.

2. On pull request merge, a [Releaser](.github/workflows/release.yml) workflow will run. It will perform the following actions:
1. Extract the version from the modified `Cargo.toml` files. Process each crate in the workspace **independently**.
galargh marked this conversation as resolved.
Show resolved Hide resolved
2. Check if a git tag for the version, using the `crate_name@version` as the pattern, already exists. Continue only if it does not.
3. Check if a draft GitHub release with the version as the tag exists.
4. If the draft release exists, publish it. Otherwise, create a new release with the version as the tag.
3. **[MANUAL]** Run `cargo publish` for each crate that has been released in the [reverse dependency order](#crate-dependency-graph).
galargh marked this conversation as resolved.
Show resolved Hide resolved

#### Known Limitations

1. `cargo publish --dry-run` will fail if a crate has a dependency on a crate that has not been published yet (i.e. that is being published in the same release).
2. `cargo publish` has to be run manually.

#### Possible Improvements

1. Run `cargo publish --dry-run` in the [reverse dependency order](#crate-dependency-graph). Use a local registry to simulate the dependencies that are not yet published.
2. Run `cargo publish` in the [**reverse dependency order**](#crate-dependency-graph) automatically after the merge.

<details>
<summary>How to?</summary>

1. Figure out which crates need to be published.
2. Either determine the correct order, or hard-code it in a config.
3. Run `cargo vendor --versioned-dirs`, creating a `.cargo/config.toml` file to respect the vendor directory.
4. In publish order (reverse dependency order):
1. Run `cargo package`.
2. Copy the package (the extracted one) out of `target/packages` into `vendor`.
3. Generate the `.cargo-checksum.json` file for that manually vendored package. This is the most annoying step, but shouldn't be hard with `find` + `jq`...

</details>
Loading