Harden Docker mise install against transient upstream failures#9000
Harden Docker mise install against transient upstream failures#9000claude[bot] wants to merge 3 commits into
mise install against transient upstream failures#9000Conversation
Two consecutive staging deploys failed because the tool-install layer hit an HTTP 504 from an upstream tool source during `mise install`. - Move the `mise install` layer above the turbo-pruned dependency-manifest COPYs in all five service Dockerfiles so the tool layer stays cached in the registry build cache across JS dependency churn and most builds never contact upstream tool sources at all - Bump the pinned mise from 2026.4.22 to 2026.7.0: since v2026.4.24 (jdx/mise#9414) mise retries transient download failures (5xx/408/429, network errors, mid-stream drops) with jittered backoff by default, whereas 2026.4.22 defaulted to zero retries so a single 504 was fatal - Pin `biome` to 2.5.2 so builds no longer resolve "latest" on every run Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EcBiQCfeizMQJubtqXkiMs
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #9000 +/- ##
=======================================
Coverage 59.74% 59.74%
=======================================
Files 1370 1370
Lines 135170 135170
Branches 6067 6067
=======================================
+ Hits 80760 80761 +1
+ Misses 53477 53476 -1
Partials 933 933 Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Merging this PR will degrade performance by 15.38%
|
| Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|
| ❌ | bit_matrix/dense/iter_row[64] |
140.8 ns | 170 ns | -17.16% |
| ❌ | bit_matrix/dense/iter_row[200] |
185.8 ns | 215 ns | -13.57% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing claude/docker-mise-install-hardening (e5a2a2c) with main (e765bd6)
…file, set evidence-based min_version - Remove `biome` from the integration-worker `mise install` list: the repo migrated formatting to oxfmt, and nothing in the image's turbo build graph (build -> codegen tasks of the app and its workspace dependency closure) invokes biome or oxfmt, so no replacement is needed - Enable `lockfile = true` so a committed `mise.lock` stays maintained by `mise install` once generated - Set `min_version = "2026.6.13"` (instead of matching the Docker pin): the floor is driven by lockfile-content stability for this repo's tools (node source-lock outcomes in v2026.6.13, java lock identity in v2026.6.4, rust-toolchain.toml idiomatic locking in v2026.6.3) plus `mise lock` failing loudly on unresolvable tools (v2026.6.13); Docker images stay on MISE_VERSION=2026.7.0 which is above the floor Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EcBiQCfeizMQJubtqXkiMs
…pts, mise pin The repo's migration to oxfmt left biome installed but never invoked: no script in any package.json runs biome, there is no biome.json, and the only build-path formatter (hash-api's codegen afterOneFileWrite oxfmt hook) is outside the ai-worker-ts / frontend / petrinaut-website dependency closures. Drop biome from the ai-worker-ts Dockerfile tool list, both vercel-install.sh scripts, and the [tools] pin added earlier in this branch, so nothing resolves or downloads it at all. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EcBiQCfeizMQJubtqXkiMs
Benchmark results
|
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 2002 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 1002 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: high, policies: 3314 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: medium, policies: 1527 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 2078 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 1033 | Flame Graph |
policy_resolution_medium
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 102 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 52 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: high, policies: 269 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: medium, policies: 108 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 133 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 63 | Flame Graph |
policy_resolution_none
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 2 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 2 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 8 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 3 | Flame Graph |
policy_resolution_small
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 52 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 26 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: high, policies: 94 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: medium, policies: 27 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 66 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 29 | Flame Graph |
read_scaling_complete
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_id;one_depth | 1 entities | Flame Graph | |
| entity_by_id;one_depth | 10 entities | Flame Graph | |
| entity_by_id;one_depth | 25 entities | Flame Graph | |
| entity_by_id;one_depth | 5 entities | Flame Graph | |
| entity_by_id;one_depth | 50 entities | Flame Graph | |
| entity_by_id;two_depth | 1 entities | Flame Graph | |
| entity_by_id;two_depth | 10 entities | Flame Graph | |
| entity_by_id;two_depth | 25 entities | Flame Graph | |
| entity_by_id;two_depth | 5 entities | Flame Graph | |
| entity_by_id;two_depth | 50 entities | Flame Graph | |
| entity_by_id;zero_depth | 1 entities | Flame Graph | |
| entity_by_id;zero_depth | 10 entities | Flame Graph | |
| entity_by_id;zero_depth | 25 entities | Flame Graph | |
| entity_by_id;zero_depth | 5 entities | Flame Graph | |
| entity_by_id;zero_depth | 50 entities | Flame Graph |
read_scaling_linkless
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_id | 1 entities | Flame Graph | |
| entity_by_id | 10 entities | Flame Graph | |
| entity_by_id | 100 entities | Flame Graph | |
| entity_by_id | 1000 entities | Flame Graph | |
| entity_by_id | 10000 entities | Flame Graph |
representative_read_entity
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/block/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/book/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/building/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/organization/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/page/v/2
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/person/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/playlist/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/song/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/uk-address/v/1
|
Flame Graph |
representative_read_entity_type
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| get_entity_type_by_id | Account ID: bf5a9ef5-dc3b-43cf-a291-6210c0321eba
|
Flame Graph |
representative_read_multiple_entities
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_property | traversal_paths=0 | 0 | |
| entity_by_property | traversal_paths=255 | 1,resolve_depths=inherit:1;values:255;properties:255;links:127;link_dests:126;type:true | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:0;link_dests:0;type:false | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:1;link_dests:0;type:true | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:2;links:1;link_dests:0;type:true | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:2;properties:2;links:1;link_dests:0;type:true | |
| link_by_source_by_property | traversal_paths=0 | 0 | |
| link_by_source_by_property | traversal_paths=255 | 1,resolve_depths=inherit:1;values:255;properties:255;links:127;link_dests:126;type:true | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:0;link_dests:0;type:false | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:1;link_dests:0;type:true | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:2;links:1;link_dests:0;type:true | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:2;properties:2;links:1;link_dests:0;type:true |
scenarios
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| full_test | query-limited | Flame Graph | |
| full_test | query-unlimited | Flame Graph | |
| linked_queries | query-limited | Flame Graph | |
| linked_queries | query-unlimited | Flame Graph |
Requested by Tim Diekmann, Bilal Mahmoud · Slack thread
🌟 What is the purpose of this PR?
Before: the service Dockerfiles run
mise install <tools>andyarn installin a single layer after copying the turbo-pruned dependency manifests, so any JS dependency change invalidates the tool layer and every such build re-downloads node/java/protoc/… from upstream sources. The pinned mise (2026.4.22) performs zero HTTP retries by default (http_retries = 0at that version), so a single transient 504 from an upstream tool source fails the whole image build. This failed two consecutive staging deploys in the "Build and push integration-worker (amd64)" job: run 29033102373 and run 29033986038. The images also installedbiome— unpinned (resolving "latest" every build) and, since the repo-wide migration to oxfmt, never actually invoked by anything.After: the tool-install layer is built before the dependency-manifest COPYs, so it stays cached in the GHCR buildx registry cache across JS dependency churn and most builds never contact upstream tool sources at all. When a build does hit upstream, mise ≥ 2026.4.24 retries transient failures — 5xx/408/429, network errors, mid-stream drops — with jittered backoff by default (jdx/mise#9414). A committed
mise.lock(withlockfile = true, groundwork in this PR — see status below) removes the remaining resolution/API traffic so the whole failure class goes away.biomeis removed everywhere — both Dockerfiles that installed it, both Vercel install scripts, and no mise pin — so it is no longer resolved or downloaded at all.Which mitigation answers what:
for-loop retry wrapper: mise's own retry landed in v2026.4.24, so bumping the pin is the non-bandaid version of the same protection.mise.lock+ strict locked installs → installs use pre-resolved URLs/checksums and skip GitHub/aqua API calls entirely (jdx/mise#8679, fixing #8677, in v2026.3.11; attestation-probe caching in v2026.5.6 via #9741).mise installRUN in all five Dockerfiles already receivesGITHUB_TOKENvia--mount=type=secret,id=GITHUB_TOKEN,env=GITHUB_TOKEN, passed by.github/actions/docker-build-pushfromgithub.token. A token only raises API rate limits; it would not have helped with a 504.🔗 Related links
🚫 Blocked by
mise.lockgeneration: the sandbox this PR was authored in cannot reach GitHub release/API endpoints for third-party repos, so the lockfile could not be generated here. One command on an unrestricted machine produces it:mise lock -p linux-x64,linux-arm64,macos-x64,macos-arm64(then commit.config/mise/mise.lock). Until it lands,MISE_LOCKED=1must NOT be added to the Dockerfiles (strict mode fails without lock URLs); everything else in this PR stands alone.🔍 What does this change?
apps/{hash-api,hash-ai-worker-ts,hash-frontend,hash-graph,hash-integration-worker}/docker/Dockerfile:builderstage installs system packages + mise tools before theCOPY --from=base /app/out/json …manifest copies;yarn install --immutablesplit into its own RUN after them. Tool versions come from/etc/mise/config.toml(copied in the first stage), so the install needs nothing from the pruned context.MISE_VERSIONbumped2026.4.22→2026.7.0.biomeremoved from every place that installed it —apps/hash-integration-worker/docker/Dockerfile,apps/hash-ai-worker-ts/docker/Dockerfile,apps/hash-frontend/vercel-install.sh,apps/petrinaut-website/vercel-install.sh— with no replacement. Evidence: no script in anypackage.jsonin the repo invokesbiome, there is nobiome.json, and no package depends on@biomejs/*— the oxfmt migration is complete. The only formatter invoked from the turbobuild→codegengraph at all ishash-api's codegenafterOneFileWrite: oxfmthook, andhash-apiis not in the dependency closure of the integration-worker, ai-worker-ts, frontend, or petrinaut-website builds (the api/frontend images already installoxfmtfor their own use). The only remainingbiomestrings in the repo are two inertbiome-ignoresuppression comments inapps/hash-frontend/src/pages/supply-chain/..config/mise/config.toml:lockfile = true— groundwork so the committedmise.lockstays maintained bymise install/mise use. Harmless until the lockfile exists (mise does not create one automatically;mise lockdoes).min_version2026.4.22→2026.6.13— see below.README.md: the two dev-facing mise version mentions updated to≥ 2026.6.13.Why
min_version = 2026.6.13(and not the Docker pin 2026.7.0)The Docker
MISE_VERSIONand the dev-facingmin_versionare separate knobs; the images can ride newer without churning every developer. The floor is the max of the hard requirements for working with the committed lockfile, all verified against the mise changelog:mise installand churns the committed lock: node source-lock outcomes (#10547, v2026.6.13), java lock identity (#9989, v2026.6.4),rust-toolchain.tomlidiomatic-file locking (#10309/#10319, v2026.6.3)mise lockfails loudly when active tools cannot resolve (#10586, v2026.6.13) — regenerating the lock on older versions can silently produce incomplete locksLockfile mechanics (verified against mise 2026.7.0 source/docs)
.config/mise/config.toml, the lockfile lives at.config/mise/mise.lock(lockfile_path_for_configuses the config's parent dir), so the existingCOPY .config/mise /etc/misecarries it into the images with no Dockerfile change; system-config lockfiles are supported (mise lock -gcovers "~/.config/mise/mise.lock and system config").config.ci.tomltools get a separatemise.ci.lockscoped to CI.mise lock -p linux-x64,linux-arm64,macos-x64,macos-arm64pre-generates entries for both Docker arches and both dev macOS arches. Under strict locked mode, install fails for a platform with no lockfile URL (no silent fallback); without strict mode, an unlocked platform falls back to normal resolution.npm:*/cargo:*/pipx:*tools are version-only in the lockfile (no pre-resolved URLs) and are explicitly exempt from the strict-mode URL check (supports_lockfile_url() == false); their downloads go to the npm registry/crates.io with those tools' own retry behaviour. The rust toolchain is likewise exempt (rustup manages it).mise.lockis committed, the follow-through isMISE_LOCKED=1(orlocked = true) for the Docker install layers so image builds hard-require pre-resolved URLs.Pre-Merge Checklist 🚀
🚢 Has this modified a publishable library?
This PR:
📜 Does this require a change to the docs?
The changes in this PR:
🕸️ Does this require a change to the Turbo Graph?
The changes in this PR:
mise.lockis not yet committed (see Blocked by) — strict locked installs are therefore not yet enabled in the Dockerfiles.🐾 Next steps
.config/mise/mise.lock(all four platforms), then addMISE_LOCKED=1to the Docker tool-install layers.biome-ignorecomments inapps/hash-frontend/src/pages/supply-chain/(inert; cosmetic only).🛡 What tests cover this?
❓ How to test this?
mise installlayer is a cache hit (no upstream downloads in the log).