[INFRA] Streamline the way we notify other repositories when a new version is available #2864
Description
Current status and problems
Historically, when a new tag was created, we built the navigation demo and then notified the repository of examples that the demo was available.Then, when we introduced the R package, we notify it of a new release when the npm publish was done (it requires to download the IIFE bundle from the npm package).
At a first glance, we see that we notify the repositories at a different pace, whereas we should notify them at the same time. And this should be done when we are sure that the npm package is published. If this fails, we generally fix the problem and create a new release.
But there are more important issues with the current way of doing things.
In the repository of examples, the projects' package.json files are updated when a new version is available. This triggers a build of these projects, which requires the new package version to be available on npmjs.
By notifying the repository of examples after a tag (and building the navigation demo), we have no guarantee that the npm package is available for building the projects. The "npm publish" workflow can take time to run, or can be queued because all GitHub runners are already running other jobs.
In the past, errors occurred several times because of that. This requires running the GitHub workflow again, which is boring and slows down the release.
This has been even more critical since we have merged process-analytics/bpmn-visualization-examples#541: the first step of the GitHub workflow updating the version downloads the npm package (to compute SubResource Integrity). There are more chances than ever that it is not available at that time.It occurs during the first update done after this PR was merged.
See errors in https://github.com/process-analytics/bpmn-visualization-examples/actions/runs/6222658937/job/16887034563:
Mon, 18 Sep 2023 12:42:40 GMT Run curl https://registry.npmjs.org/bpmn-visualization/-/bpmn-visualization-0.40.0.tgz -o bpmn-visualization.tgz
Mon, 18 Sep 2023 12:42:40 GMT % Total % Received % Xferd Average Speed Time Time Time Current
Mon, 18 Sep 2023 12:42:40 GMT Dload Upload Total Spent Left Speed
Mon, 18 Sep 2023 12:42:40 GMT
Mon, 18 Sep 2023 12:42:41 GMT 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
Mon, 18 Sep 2023 12:42:41 GMT 100 21 100 21 0 0 49 0 --:--:-- --:--:-- --:--:-- 49
Mon, 18 Sep 2023 12:42:42 GMT npm ERR! code E404
Mon, 18 Sep 2023 12:42:42 GMT npm ERR! 404 No match found for version 0.40.0
Mon, 18 Sep 2023 12:42:42 GMT npm ERR! 404
Mon, 18 Sep 2023 12:42:42 GMT npm ERR! 404 'bpmn-visualization@0.40.0' is not in this registry.
Mon, 18 Sep 2023 12:42:42 GMT npm ERR! 404
Mon, 18 Sep 2023 12:42:42 GMT npm ERR! 404 Note that you can also install from a
Mon, 18 Sep 2023 12:42:42 GMT npm ERR! 404 tarball, folder, http url, or git url.
Here are the logs of the npm publish process in the bpmn-visualization repository : they show that the package is published more than one minute later, so the update fails for sure (see https://github.com/process-analytics/bpmn-visualization-js/actions/runs/6222648900/job/16887005367):
Mon, 18 Sep 2023 12:43:59 GMT npm notice
Mon, 18 Sep 2023 12:43:59 GMT npm notice bpmn-visualization@0.40.0
Mon, 18 Sep 2023 12:43:59 GMT npm notice === Tarball Contents ===
Mon, 18 Sep 2023 12:43:59 GMT npm notice 11.4kB LICENSE
Mon, 18 Sep 2023 12:43:59 GMT npm notice 552B NOTICE
Mon, 18 Sep 2023 12:43:59 GMT npm notice 10.6kB README.md
Mon, 18 Sep 2023 12:43:59 GMT npm notice 59.9kB dist/bpmn-visualization.d.ts
Mon, 18 Sep 2023 12:43:59 GMT npm notice 202.8kB dist/bpmn-visualization.esm.js
Mon, 18 Sep 2023 12:43:59 GMT npm notice 1.7MB dist/bpmn-visualization.js
Mon, 18 Sep 2023 12:43:59 GMT npm notice 987.0kB dist/bpmn-visualization.min.js
Mon, 18 Sep 2023 12:43:59 GMT npm notice 932B dist/not-supported-ts-versions.d.ts
Mon, 18 Sep 2023 12:43:59 GMT npm notice 6.8kB package.json
Mon, 18 Sep 2023 12:43:59 GMT npm notice === Tarball Details ===
Mon, 18 Sep 2023 12:43:59 GMT npm notice name: bpmn-visualization
Mon, 18 Sep 2023 12:43:59 GMT npm notice version: 0.40.0
Mon, 18 Sep 2023 12:43:59 GMT npm notice filename: bpmn-visualization-0.40.0.tgz
Mon, 18 Sep 2023 12:43:59 GMT npm notice package size: 570.3 kB
Mon, 18 Sep 2023 12:43:59 GMT npm notice unpacked size: 3.0 MB
Mon, 18 Sep 2023 12:43:59 GMT npm notice shasum: bfc14d6d01647c5c8982ca0d20f750d233ae77df
Mon, 18 Sep 2023 12:43:59 GMT npm notice integrity: sha512-74oZO1M2J3TL6[...]s4QqkWWUL8QcA==
Mon, 18 Sep 2023 12:43:59 GMT npm notice total files: 9
Mon, 18 Sep 2023 12:43:59 GMT npm notice
Mon, 18 Sep 2023 12:43:59 GMT npm notice Publishing to https://registry.npmjs.org/ with tag latest and public access
Mon, 18 Sep 2023 12:43:59 GMT npm notice publish Signed provenance statement with source and build information from GitHub Actions
Mon, 18 Sep 2023 12:43:59 GMT npm notice publish Provenance statement published to transparency log: https://search.sigstore.dev/?logIndex=37060827
Proposals
The examples repository must be notified of new version availability only when the npm package is published.
It also requires to retrieve the "navigation demo".
flowchart LR
newTag((new tag)) --> bng(build navigation demo) --> notifEx((notify repository\n of examples))
newTag --> npm(npm publish) --> notifR((notify\n R package))
Current implementation
Proposal 1
IMPORTANT: this will not be implemented, see "Proposal 2"
A first solution could be like the following:
- keep the existing "npm publish" workflow which is triggered when a new tag is created
- at the end of this workflow, it triggers a "new workflow" in charge of preparing assets and notifying the other repositories
- the triggering can be done by sending an
repository_dispatchat the end of the "npm publish" workflow including the tag/version. - A workflow_run could also trigger the "new workflow", but passing the version info is more tricky. See https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#using-data-from-the-triggering-workflow that show the work to do to retrieve a value from the initial workflow.
- the "new workflow" can also be triggered using a
workflow_dispatchfor manual replay - the "new workflow" includes the job that currently build the navigation demo and notify the repository of examples
- it also includes a job that send a
repository_dispatchto all repositories (using a matrix configuration - array) requiring to be notified that a new version is available on npm (for now, bpmn-visualization-examples, for others, see below)
Note: In the description above, the "new workflow" could be the existing workflow building the navigation demo that we would update with the proposal. It would probably have to be renamed to match what it achieves. Note that this involves modifying the event sent to the example repository, as it contains the name of the workflow that stores the navigation demo artifact.
flowchart LR
newTag((new tag)) --> npm(npm publish) --> sendEvent((notify\n wf2))
wf2((wf2:\n received\n event)) --> gateway{+} --> bng(build navigation demo) --> notifEx((notify repository\n of examples))
gateway --> notifR((notify\n R package))
gateway --> notifRepo1((notify\n repo1))
gateway --> notifRepo2((notify\n repo2))
Proposal 1
Proposal 2
IMPORTANT: this is the proposal that will be implemented
Overview
- The workflow that manages the "npm publish" only performs the publishing to better separate the responsibilities
- Once the publishing is done, an event is sent to notify that the operation is completed
- The existing workflow that build the "demo" is now triggered when it receives the event, not when the git tag is created
- A new workflow is created to notify other repositories that the new version of the npm package is available
flowchart LR
newTag((new tag)) --> npm(npm publish) --> sendEvent((send event\n new npm package\n version available))
wf1((wf1:\n received\n event)) --> bng(build navigation demo) --> notifEx((notify repository\n of examples))
wf2((wf2:\n received\n event)) --> gateway{+}
gateway --> notifR((notify\n R package))
gateway --> notifRepo1((notify\n repo1))
gateway --> notifRepo2((notify\n repo2))
Proposal 2
Repositories to notify when a new version is available
While we currently notify the repository of examples and bpmnVisualizationR, other repositories may also require to be notified when a new version is available.
Managed with dependabot: if there is an urgent need for an update, it can be run manually:
- bv-addons: chore(deps): bump bpmn-visualization from 0.39.0 to 0.40.0 bpmn-visualization-addons#125
- demo template: chore(deps): bump bpmn-visualization from 0.39.0 to 0.40.0 bpmn-visualization-demo-template#31
- bonita-day 2023 demo: chore(deps): bump the process-analytics group with 2 updates bonita-day-demo-2023#131
- icpm 2022 demo: chore(deps): bump bpmn-visualization from 0.37.0 to 0.39.0 icpm-demo-2022#99
Not possible to managed with dependabot
bpmnVisualizationR: require to download the IIFE bundle- repository of examples: url to update (with SubResource Integrity computation), projects update not manageable by dependabot as not a "workspace" aware project (may be supported if we used npm workspaces, pnpm, yarn, ... or alterative to dependabot like renovate?)
To check
- bpmn-visualization-tools: not possible today, but should be possible if we switch to a "workspaces" aware build
- pm4py?
Metadata
Assignees
Type
Projects
Status
Done
Activity