Two-tier component secret delivery + env declarations (#1550)

[kriszyp]

Closes #1550. Stacked on #1554 (kris/secrets-store-715) — the diff shows only the consumption side. Retarget to main after #1554 merges; do not merge #1554 via squash while this is stacked.

Summary

Consumption of the hdb_secret store, delivered in two tiers:

• Global tier (rows with empty grants): decrypted at the start of each component-load cycle and materialized into the real process.env, before applications load. Precedence: a pre-existing real env var always wins (logged). Child processes inherit these — same semantics as .env today. No isolation promise.
• Scoped tier (rows with non-empty grants): never in process.env — not inherited by child processes, invisible to env dumps. Exposed only through the per-component accessor: import { secrets } from 'harper' and scope.secrets. The accessor is a superset view: scoped rows granted to the component plus its declared global-tier names, so app code uses secrets.FOO uniformly and ops can later tighten a secret from global to granted without breaking the app.

Component configs declare env expectations in an env: block:

env:
  NODE_ENV: production                                        # string → inline literal into process.env (.env semantics, supports enc:v1:)
  STRIPE_SECRET_KEY: { required: true, description: Stripe }  # object → declaration, satisfied from the store
  SENTRY_DSN: { required: false }

An unsatisfied required: true declaration gates that component's load (reason: missing | ungranted | custody-unavailable); the instance keeps running and nothing of the gated component is registered. Literals are applied only after the whole block validates and the gate passes. get_components exposes the declared-but-unsatisfied set per component (unsatisfiedEnv: name/description/required/reason/tier — metadata only, never values) for Studio configure-me rendering.

Accessor binding (the investigated part)

• vm, compartment, and the default vm-current-context loader: import { secrets } from 'harper' (and CJS require('harper') inside the app loader) resolves to the per-scope synthetic harper module (security/jsLoader.ts getHarperExports) — exact binding. This includes the default mode: the harper specifier is intercepted by resolveModule before Node resolution, so the symlinked package is not consulted.
• native loader / bundler-loaded code / natively-loaded deps: those reach the process-wide singleton package, whose secrets export resolves the loading component via an AsyncLocalStorage context that scopedImport binds for the duration of each component load. Access from an ambiguous context (e.g. lazily at request time) fails loudly with an error naming the component-resolution problem — it never guesses.
• Recommended idiom (documented in the module header): destructure at module top level — const { STRIPE_SECRET_KEY } = secrets; — where binding is exact in every mode.

JS-level scoping is documented as a slowdown layer, not a security boundary (shared process; containers/uids are the boundary; no SES hardening claimed).

Degraded modes

• No custody registered (OSS core / material missing): required declarations report custody-unavailable; global rows log and stay out of env; node boots.
• hdb_secret table missing (pre-upgrade): store treated as empty, missing reasons note the store is unavailable; node boots.
• Late custody / changed secrets: heal on restart or reload cycle (each cycle re-reads the store; env-declaring component loads also refresh the snapshot so deploy validation in a long-lived worker gates against post-boot set_secret/grant_secret). No live re-materialization in v1.

Where to focus review

• The secrets export is also an ambient global (via _assignPackageExport). Its loud-failure traps mean util.inspect/console.log of a container holding it (e.g. console.log(global)) outside a load context throws. Deliberate trade-off for "never silently hand back another component's secrets"; symbols and then are exempted so await/inspect probes don't explode.
• Main-thread reload semantics: loadedPaths is never cleared in production, so already-loaded components early-return on main-thread reload; the declaration registries are therefore overwrite-on-reprocess (a cycle-level wipe would permanently empty get_components.unsatisfiedEnv — caught by the cross-model review leg).
• Per-load store scan: each env-declaring component load re-scans hdb_secret (small system table, load-time only) to keep out-of-cycle gates fresh.
• Gate containment: gate failure = status failed + log + early return; it deliberately does not install an ErrorResource at / (the pre-existing per-plugin catch does, which can clobber the root URL space — not repeated here).

Explicitly out of scope (follow-ups)

• sensitive: false plaintext rows in hdb_secret (ops-managed non-secret vars with the same grants/declaration/UI machinery) — touches feat: hdb_secret store with grant-scoped secret operations #1554's operations while in review.
• Live re-materialization on secret change (v1 = restart/reload heals).
• Truly process-level vars (NODE_OPTIONS, UV_THREADPOOL_SIZE) — need container-level injection (the P3 surface), not the store.
• env: in the root config is warn-and-ignore for now (root loads before custody registration/materialization; needs its own ordering story).
• Docs: user-facing (env: block + accessor); companion documentation PR to follow when this retargets to main.

Tests

unitTests/components/componentSecrets.test.js (42: materialization/precedence/retraction, literal + enc:v1: semantics, declaration shapes, all three gate reasons, accessor visibility/superset/enumeration/collision, ALS binding incl. concurrent loads, loud ambiguous failure, proxy invariants) and unitTests/components/componentSecretsGate.test.js (2: loader-level gate containment — component fails, instance survives, literals land on the passing path). Existing loadEnv behavior untouched.

Cross-model review (adjudicated; all fixes in follow-up commits):

• Codex leg — 6 findings, all confirmed and fixed: root-resource poisoning on gate failure, literal-before-gate env mutation, stale declaration registries, scope.secrets identity mismatch on RUN_HDB_APP paths, stale store snapshot on deploy-path loads, missing preventExtensions trap. Plus 1 valid residual on the first fix round (a cycle-level registry wipe vs. the never-cleared loadedPaths) — resolved with overwrite-on-reprocess semantics.
• Gemini leg — 7 findings: 4 confirmed and fixed (concurrent-load scan/decrypt herd + out-of-order snapshot overwrite → single-flight materializeGlobalSecrets; gate rejecting an env fallback the accessor would serve for granted-but-undecryptable rows → gate now matches; Object.prototype names masquerading as secrets under dynamic access → null-prototype views). 1 documented-as-designed (env literals share process.env cross-component with load-order visibility — inherent to .env semantics, now stated in the module docs). 2 dismissed with evidence: the "Blocker" claim that AsyncLocalStorage context is lost during dynamic-import ESM top-level evaluation is empirically false on our supported Node range (guarded by a permanent regression test covering top-level read, destructure, and post-top-level-await); the catch (error) primitive-throw concern doesn't apply (the guarded block only throws Error instances and matches the loader's existing pattern).

Generated by Claude (Fable) with cross-model review by Codex and Gemini.

[gemini-code-assist]

Code Review

This pull request introduces a component-facing secrets management system with global-tier secrets materialized into process.env and scoped-tier secrets accessible via a per-component secrets accessor, alongside an env: configuration block to validate environment expectations during component loading. The review feedback identifies a critical issue where the process-wide secrets Proxy can cause crashes during global object inspection (e.g., by debuggers or logging tools) when accessed outside of a component-loading context, and suggests returning safe, empty values for the has, ownKeys, and getOwnPropertyDescriptor traps in such scenarios.

[claude]

Reviewed; no blockers found.

[claude]

Suggestion (non-blocking): the comment says "Enforce the no-newline invariant at the injection point so it holds for every source," but only token is guarded — registry is not. A manually-edited config or a hypothetical bypass path with an embedded \n in the registry string would survive normalizeRegistryUrl's .trim() and inject extra .npmrc lines (e.g. https://host/\nregistry=https://evil.com/ → an injected registry= line). Apply the same guard to registry:

Suggested change

	if (/[\r\n]/.test(token)) {
	if (/[\r\n]/.test(token)) {
	throw new Error(`registry auth token for '${registry}' contains an illegal newline character`);
	}
	if (/[\r\n]/.test(registry)) {
	throw new Error(`registry URL for token entry contains an illegal newline character`);
	}