Duplicate Rust-stdlib symbols across well-known native libraries when multiple 'shared tokio' libs are linked together

[proggeramlug]

Summary

Found while getting a real-world sst/opencode compile all the way to a working binary (companion investigation to #5918/#5919, #5922/#5924/#5927/#5923). After all import/namespace-resolution symbol issues were fixed, the final link still failed — first with ~5900, then (after #5920/#5921 landed on main, which helped) ~2150 duplicate-symbol errors from ld on macOS.

Root cause

opencode's dependency graph needs FIVE separate "well-known" native libraries simultaneously: http/https/http2 → perry-ext-http, fastify → perry-ext-fastify, ioredis → perry-ext-ioredis, net → perry-ext-net, ws → perry-ext-ws, each rebuilt "with shared tokio" per the well-known: routing log lines. Each of these is an independently-built Rust staticlib that bundles its OWN full copy of shared transitive dependencies (their exact codegen-unit object files, e.g. tokio-<hash>.tokio.<hash>-cgu.0.rcgu.o). libperry_stdlib.a ALSO bundles its own copies of the same crates (for its own fetch/https/websocket support). When multiple such libraries are linked together, every shared dependency they have in common — std, core, alloc, hashbrown, aho_corasick, regex_automata, tokio, hyper_util, h2, rustls, reqwest, ring, bytes, url, and perry's own perry_runtime codegen unit — ends up defined in more than one archive.

#5920/#5921 (already on main) added strip_bundled_runtime_from_well_known_lib, which safely drops duplicate perry_runtime-* member objects from a well-known lib when stdlib bundles the identical codegen unit (verified via two safety rules: Rule 1 — stdlib bundles the exact same codegen unit; Rule 2 — no sibling member in the same archive depends on a symbol only that copy provides). This fixed the perry_runtime duplicates specifically and reduced the failure from ~5900 to ~2150 duplicate symbols in opencode's case, but the SAME duplication shape recurs for every other shared crate above once TWO OR MORE well-known libraries both need a full HTTP-client stack (e.g. both http and fastify bundle tokio+hyper_util+h2+rustls+reqwest+ring).

Why there's no linker-flag escape hatch

Verified directly on this machine's toolchain (Xcode/CLT current as of 2026-07): neither -Wl,-multiply_defined,suppress nor -Wl,-ld_classic (with or without -multiply_defined) suppress the duplicate-symbol error anymore — both are reported as "obsolete"/"deprecated" and the link still fails. Unlike GNU ld/lld (which perry already uses -Wl,--allow-multiple-definition for on Android/HarmonyOS), Apple's current linker has no tolerance mode for this. The fix has to happen at the object/archive level before linking, mirroring what #5920/#5921 already does for perry_runtime.

What does NOT work: naively widening #5920's candidate set

I tried generalizing strip_bundled_runtime_from_well_known_lib's candidate filter from member.starts_with("perry_runtime-") to "every member" (relying on the existing Rule 1 + Rule 2 checks to gate safety). This broke the existing issue_5920_wrapper_bundled_runtime_async_starvation regression test (fire_and_forget_fetch_survives_recompile started failing with the same starvation symptom #5920 fixed).

Root cause of why the naive widening is unsafe: Rule 2's sibling_undefined set is computed as "the union of undefined symbol references from every member NOT in the candidate set." As the candidate set grows to include nearly every crate-derived member (std, core, tokio, etc. are present in almost every object in an HTTP-stack-bundling archive), the "sibling" set used to check for unsafe removals shrinks correspondingly — inter-candidate dependencies (candidate A referencing a symbol only candidate B defines, both of which are simultaneously being removed) are never checked at all, since Rule 2 only guards candidate-vs-non-candidate edges. A safe generalization needs either a smarter per-member reachability computation, or an iterative fixed-point approach (remove one "layer" of provably-safe duplicates at a time, recomputing sibling references between rounds) rather than a one-shot flat widening.

Suggested direction for a real fix

• Generalize the duplicate-detection to run per-crate (matching each member's crate-name prefix, e.g. tokio-, hyper_util-, h2-, rustls-, reqwest-, ring-, bytes-, url-, std-, core-) similarly to perry_runtime-, but iterate to a fixed point (repeatedly compute removable members and re-derive sibling_undefined from the shrinking "kept" set) instead of a single pass, so inter-candidate dependencies are still caught.
• Alternatively, dedupe at the ld-invocation level via -single_module/symbol-versioning-style tricks, or by post-processing with llvm-objcopy --keep-global-symbols per-archive against a computed "authoritative provider" map (whichever archive is linked first/is stdlib wins, in ALL other archives strip to only symbols not already provided elsewhere) — this needs the same fixed-point care as above.
• This is a genuinely deep, strip_dedup.rs-architecture-level task; recommend whoever picks it up start from the #5920 test's design (spawn a real async task, assert it doesn't starve) and add an analogous "N well-known libs + fastify + http simultaneously" fixture to catch regressions in either direction.

Workaround used for now

None found that's both safe and general — @fastify/*/@ioredis/commands removal from compilePackages was tried and abandoned (it just routes those packages to interpreted-JS fallback, which this build of perry has removed support for entirely: "JavaScript runtime (V8) support has been removed"). opencode's full binary currently cannot complete linking with --enable-wasm-runtime and its real dependency set; all symbol-resolution issues (the actual multi-bug investigation across #5918/#5922/#5924/#5927) are fully fixed and this is the last remaining blocker.

[proggeramlug]

Update: found a working (partial) fix, now in PR #5923.

Correction on the two "failed attempts" documented above: both were false negatives. The safety-check tool (issue_5920_wrapper_bundled_runtime_async_starvation) requires llvm-objcopy/llvm-nm/llvm-ar to actually be reachable (via PERRY_LLVM_OBJCOPY/PERRY_LLVM_NM/PERRY_LLVM_AR env vars or PATH) for the strip-dedup mechanism to run at all — without them, strip_bundled_runtime_from_well_known_lib (the existing #5920/#5921 fix) silently no-ops (wrapped in .unwrap_or_else with a "(non-fatal)" log line) and the test's own compiled binary reproduces the pre-#5920 starvation bug regardless of any change under test. Confirmed by running the SAME test against a completely untouched origin/main checkout: it also "fails" without the env vars configured, and passes once they're set. Neither of my two prior attempts were actually breaking anything — I just didn't have a working local LLVM toolchain install visible to the test's subprocess.

The actual fix, now verified correctly with the environment properly configured: generalizes strip_bundled_runtime_from_well_known_lib's two safety rules (stdlib bundles the identical codegen unit; no other kept member depends on a symbol only the candidate provides) from perry_runtime-* members specifically to every member, via a fixed-point iteration (a one-shot widening is NOT safe on its own — candidates can depend on each other, so each round recomputes references against the shrinking kept-set until stable).

Result on a real sst/opencode compile (with #5918/#5919 also applied): duplicate-symbol errors went from ~5900 → 0 for the ioredis/net/ws well-known libraries entirely, and from ~2150 → ~1382 for the http+fastify pair specifically (the two largest, most interconnected libraries, both needing a full reqwest/hyper_util/h2/rustls/tokio stack) — Rule 2 conservatively protects more members as a single archive's internal dependency graph grows, so this specific combination is substantially improved but not fully zero yet. Leaving this issue open for that remaining gap; PR #5923 has the working generalization plus three unrelated but similarly-shaped import-resolution fixes found in the same investigation.

Also worth a follow-up: the strip-dedup mechanism's silent no-op on missing LLVM tools is a genuine footgun — it produces a much-later, totally disconnected "N duplicate symbols" ld failure with zero indication dedup was skipped. Might be worth a louder warning (or hard error) when PERRY_LLVM_OBJCOPY/find_llvm_tool resolution fails entirely, distinguishing "dedup determined nothing removable" from "dedup couldn't even run."

[proggeramlug]

Update from PR #5923 (three additional fixes landed there, plus a root-caused but unfixed residual gap):

Fixed:

1. Generalized Spawned async tasks starve after recompile with fresh auto-optimize archives (wrapper-bundled perry-runtime copy splits event-pump state) #5920/fix(link): #5920 — dedup wrapper-bundled perry-runtime in wrappers-first links #5921's fixed-point dedup rules from perry_runtime-* codegen units to every shared dependency (strip_bundled_shared_deps_from_well_known_lib).
2. Matched codegen-units across the "shared tokio" (perry-ext-net: outbound TCP panics — LTO dead-strips tokio CONTEXT statics #507) build group — perry-stdlib-static/perry-runtime-static already override it to 16 (compilePackages: eventemitter3 fails to link — undefined _js_event_emitter_* symbols #5140), but the well-known ext crates (perry-ext-fastify/http/ioredis/net/ws) had no override and defaulted to the workspace's 1, causing rustc to partition shared dependencies like compiler_builtins differently across crates that should produce identical output. Confirmed via md5 this is now byte-identical (231/231 codegen units) where it previously diverged (e.g. 2.4 MB vs 551 KB for the same core unit).
3. Found and fixed a real bug in build_optimized_libs(): one of its three return paths (the "must build well-known libs from scratch," i.e. cold/cleared-cache path) hardcoded prefer_well_known_before_stdlib: false, silently skipping the dedup mechanism entirely on any build that couldn't reuse a warm auto-optimize cache. The other two paths correctly compute this from whether well-known libs are in play. This was silently disabling all of Spawned async tasks starve after recompile with fresh auto-optimize archives (wrapper-bundled perry-runtime copy splits event-pump state) #5920/fix(link): #5920 — dedup wrapper-bundled perry-runtime in wrappers-first links #5921/Duplicate Rust-stdlib symbols across well-known native libraries when multiple 'shared tokio' libs are linked together #5928's fixes whenever the cache was cold — confirmed by comparing duplicate-symbol counts with/without a stale cache directory present (2152 vs 1382 for the same program).

Root-caused but not fixed — the residual gap:

Even with all three fixes, a program needing opencode's specific combination (http+fastify+ioredis+net+ws simultaneously) still hits ~1382 duplicate symbols, unchanged from before any of this work. The reason: most of the colliding dependencies (tokio, hyper_util, rustls, ring, h2, std, core, ...) get folded into perry-stdlib-static's own linker-plugin-lto cross-crate merge, producing genuinely different compiled output than a plain (non-LTO) compile of the same crate in the well-known ext libs — not "the same code split differently," but two structurally different compilation strategies. No amount of settings-matching (codegen-units or otherwise) makes these byte-identical, because LTO's optimization decisions depend on the full set of code being merged together.

The fixed-point dedup's Rule 2 (protect a candidate if some other kept sibling still needs a symbol only it provides) operates at file granularity: a codegen unit bundles many functions, so if even one is genuinely unique, the whole file — including its other, actually-duplicate symbols — gets kept, and ld still flags those.

I tried exempting Rule 2's protection when a candidate is byte-identical (raw bytes, not just name) to its matched stdlib member — sound in principle (a byte-identical twin provably provides every symbol the candidate does), passed the full safety/regression suite, but never actually triggers for this combination since none of the colliding objects are byte-identical for the LTO reason above. Didn't keep it in the PR since it adds complexity with no demonstrated benefit for the case that motivated it — happy to revisit if a case emerges where it does help.

I also tried llvm-objcopy --localize-symbol-based symbol-level surgery in an earlier iteration (hide just the specific symbols a protected file shares with stdlib, keep the rest). That's more surgical but caused a real, confirmed regression: "there is no reactor running, must be called from the context of a Tokio 1.x runtime" — a genuine split-runtime-state bug inside tokio's own internals. Reverted.

What I think an actual fix looks like, without more certainty than I can get solo: either (a) drop perry-stdlib-static's linker-plugin-lto (risky — it exists specifically to solve #5140, so this needs another way to preserve those exports), or (b) give the well-known ext crates the same LTO treatment for just their shared deps so both sides merge equivalently (uncertain whether independently-LTO'd units would actually end up byte-identical even then, and I don't have cross-platform infrastructure to validate broadly). Flagging both as open questions rather than guessing further.

[proggeramlug]

The two fixes described above (codegen-units alignment + cold-build dedup bug) are now in a separate PR since #5923 merged before they were ready: #5980.

[proggeramlug]

One more concrete lead for whoever picks up the architectural fix, from continued investigation:

Ruled out as not-actually-safer: converting the duplicate symbols to weak linkage (llvm-objcopy --weaken-symbol) instead of whole-file removal. Checked — these symbols are currently strong/global (T in nm) in both copies. Weak-symbol resolution happens per-symbol, not per-file, so this would let the linker pick different crates' copies for different functions within what should be one coherent codegen unit (e.g. tokio's internals) — reintroducing exactly the split-runtime-state risk the whole-file granularity in the current dedup code is protecting against, and that issue #5920 exists to catch.

A more promising lead for eliminating the duplication at the source (rather than deduping after the fact): perry-stdlib already has partial infrastructure for delegating to well-known ext crates instead of bundling its own implementation — the external-http-client-pump/external-fastify-pump/external-ws-pump/external-net-pump feature family, and perry-stdlib/src/http.rs has a whole external_client_request module used instead of client_request_surface when external-http-client-pump is active. However, that only covers node:http/node:https. perry-stdlib/src/fetch/mod.rs and axios.rs each independently instantiate their own reqwest::Client with no equivalent delegation path — fetch() is a core JS global always compiled in regardless of which well-known ext libs are linked, so even a program that only imports fastify still gets a full reqwest/hyper/tokio/rustls stack bundled into stdlib for fetch()'s sake, which is the actual source of the duplication.

Extending the existing external-pump pattern so fetch()/axios also delegate their HTTP execution to a linked well-known ext lib (when present) instead of instantiating their own reqwest client would remove the duplication at the source rather than requiring linker-level dedup at all. This is real feature work across several files in perry-stdlib with behavioral-compatibility implications (fetch() has WHATWG-specific semantics that need to survive being routed through an ext lib built for node:http compatibility) — flagging as a concrete next step rather than attempting it blind.

[proggeramlug]

Final update on this investigation thread — the duplicate-symbol wall itself is now fully closed for opencode's real compile, and the compile got much further than expected once it was:

The duplicate-symbol wall (#5928 itself): closed for opencode's specific case not by a perry compiler change, but by removing the actual cause of needing fastify+http+ioredis+net+ws simultaneously — opencode's only real fastify consumer turned out to be an optional GitLab OAuth callback listener (opencode-gitlab-auth's dist/callback-server.js), stubbed with the user's explicit authorization (disclosed, narrow, reversible: GitLab OAuth login won't work, nothing else is affected). Once fastify wasn't needed, the whole well-known-lib duplicate-symbol collision this issue tracks doesn't arise for this program. The two perry-side fixes landed here (codegen-units alignment, cold-build dedup restoration) remain independently correct and valuable for other programs that do need multiple well-known libs simultaneously.

What that unblocked: with the duplicate-symbol wall out of the way, the compile proceeded much further and hit two more, entirely separate, previously-unreached perry bugs in sequence:

• Computed-key static class fields leak their synthetic name into cross-module external-global declarations #5984 (fixed, PR fix(compile): #5984 — computed-key static fields leak synthetic name into cross-module externs #5985): computed-key static class fields (drizzle-orm's pervasive static [entityKind] = "..." tagging) leaked their synthetic placeholder name into cross-module external-global declarations, causing undefined-symbol errors for ~10 drizzle-orm/effect-drizzle-sqlite classes.
• Cross-module class resolution misattributes module prefix for same-named class (ClientAbort.annotation → wrong module) #5987 (open, not yet fixed): a narrower, single-symbol issue — cross-module resolution misattributes the module prefix for a class (effect's ClientAbort) that's defined identically in two different files, similar in spirit to import_function_prefixes collision for namespace-reexport members (Context.a vs Option.a) #5922/import_function_origin_names collision for namespace-reexport members (Context.Service resolves to wrong suffix) #5924/import_function_prefixes/origin_names collide between a namespace member and an unrelated plain named import #5927/Computed-key static class fields leak their synthetic name into cross-module external-global declarations #5984 but for plain named imports rather than namespace re-exports. This is the only remaining blocker to a fully successful opencode compile as of this writing.

Also found and filed separately: #5986, a regression (unrelated to any of this work) that reintroduced #5924/#5927's exact symptom via a recent large upstream PR — flagging since it'll affect anyone relying on those fixtures being green on current main.