fix(correlation): unwrap seed envelopes in the correlation seeder's input reads

[Yigtwxx]

Summary

Follows #5896, whose "Out of scope" section named this file: scripts/seed-correlation.mjs reads its inputs with the same envelope-blind JSON.parse that #5870 was filed for, sits in the same Railway bundle, and registers the same military:flights:{v1,stale:v1} pair. There is no issue open for it.

scripts/seed-correlation.mjs:41-46:

for (let i = 0; i < INPUT_KEYS.length; i++) {
  const raw = results[i]?.result;
  if (raw) {
    try { data[INPUT_KEYS[i]] = JSON.parse(raw); } catch { /* skip */ }
  }
}

Seven of the nine INPUT_KEYS are written by contract-mode seeders, which store { _seed, data } (scripts/_seed-utils.mjs). So every field read in computeCorrelation (:702-728) was reading the envelope:

const protests   = protestData?.events      ?? (Array.isArray(protestData) ? protestData : []);
const outages    = outageData?.outages      ?? (Array.isArray(outageData)  ? outageData  : []);
const earthquakes= quakeData?.earthquakes   ?? (Array.isArray(quakeData)   ? quakeData   : []);
const stockQuotes     = data['market:stocks-bootstrap:v1']?.quotes ?? [];
const commodityQuotes = data['market:commodities-bootstrap:v1']?.quotes ?? [];
const cryptoQuotes    = data['market:crypto:v1']?.quotes ?? [];
const newsClusters    = (insights?.topStories ?? []).map(...);

{_seed, data}.events is undefined, Array.isArray(envelope) is false, so each one fell through to [].

Key	Writer	Written as	Effect here
military:flights:v1	seed-military-flights.mjs (local redisSet, no runSeed)	bare	works
military:flights:stale:v1	seed-military-flights.mjs	bare	works
unrest:events:v1	seed-unrest-events.mjs	envelope	protests = []
infra:outages:v1	seed-internet-outages.mjs	envelope	outages = []
seismology:earthquakes:v1	seed-earthquakes.mjs	envelope	earthquakes = []
market:stocks-bootstrap:v1	seed-market-quotes.mjs	envelope	stockQuotes = []
market:commodities-bootstrap:v1	seed-commodity-quotes.mjs	envelope	commodityQuotes = []
market:crypto:v1	seed-crypto-quotes.mjs	envelope	cryptoQuotes = []
news:insights:v1	seed-insights.mjs	envelope	newsClusters = []

Mapped onto the four domains (:761-783):

• military — the only surviving domain, and only because the flights pair is the one writer that never migrated to runSeed.
• escalation — collectEscalationSignals(protests, outages, newsClusters), all three empty.
• economic — collectEconomicSignals(allMarkets, newsClusters), all four empty.
• disaster — collectDisasterSignals(earthquakes, outages, protests), all three empty.

Why nobody saw it. The hasAnyData tripwire at :699-700 tests data[k] != null, and an envelope object is not null, so "No input data available in Redis" never threw. validateFn then failed the MIN_CORRELATION_CARDS floor whenever military also produced nothing, and runSeed resolved contractState = 'RETRY' (scripts/_seed-utils.mjs:2038-2040), which holds the previous cards alive without advancing _seed.fetchedAt. Exit 0, no alarm, stale cards — the same silent-degradation signature as #5870.

The irony at :704: the ?? data['military:flights:v1'] fallback exists to tolerate a raw-array shape. It would not have rescued an enveloped flights key either.

The fix

The pattern #5896 established, unchanged:

• unwrapEnvelope from scripts/_seed-envelope-source.mjs. It only unwraps when _seed.fetchedAt is a number, so the bare flights keys pass through byte-identical, as does the legacy top-level-array shape the Array.isArray fallbacks still handle.
• JSON.parse stays outside unwrapEnvelope. It accepts a raw string, but on a parse failure returns that string as data — which would register a malformed value as a found key and defeat the hasAnyData tripwire. Same reasoning as the comment at seed-cross-source-signals.mjs:225-235.
• The per-key freshness gate, because unwrapping without one revives preserved last-good envelopes into cards stamped computedAt: Date.now(). Every budget is the source seeder's own declared maxStaleMin rather than a number chosen at this call site — and the five keys shared with seed-cross-source-signals.mjs come out at exactly the budgets its own table already uses, which is a useful cross-check. The two military:flights keys are absent from the table on purpose: written bare, they carry no _seed for the gate to read.

No deploy manifest change. scripts/_seed-envelope-source.mjs is already listed in seed-bundle-derived-signals's watchPatterns in scripts/railway-services.json, next to scripts/seed-correlation.mjs itself, and is pulled in transitively by scripts/_seed-utils.mjs:11 which this seeder already imports.

Audit of the remaining scripts/ readers

Since the same defect had already appeared twice, I swept every scripts/ file that issues a Redis GET and parses the result, and checked each against its writer rather than trusting the key name:

Reader	Reads	Verdict
seed-correlation.mjs	7 cross-seeder contract-mode keys	the defect, fixed here
seed-hs2-chokepoint-exposure.mjs:210	comtrade:*:v1 from seed-comtrade-bilateral-hs4.mjs	clean — that writer uses a plain SET (:558), not runSeed, so the keys are bare
seed-forecast-bets.mjs:285	forecast:resolutions:v1, contract-mode	clean, but only just — collectOpenEnsembleIds unwraps ad hoc with Object.values(ledger.data ?? ledger) (:230-232), and the feed reads handle _seed explicitly in filterFreshFeeds/unwrapFeeds (:97-116)
seed-portwatch-port-activity.mjs:913	its own previous-version keys	clean, self-written
seed-wb-indicators.mjs:468-470	keys it wrote earlier in the same run	clean, self-written, bare
seed-military-bases.mjs, seed-webcams.mjs, seed-resilience-static.mjs, seed-comtrade-bilateral-hs4.mjs, lib/brief-embedding.mjs	own state / meta / cache keys	clean, self-written
seed-cross-source-signals.mjs, seed-regional-snapshots.mjs, regional-snapshot/_helpers.mjs	cross-seeder keys	already envelope-aware

So seed-correlation.mjs was the last envelope-blind cross-seeder reader in scripts/. seed-forecast-bets.mjs is worth a second look at some point — it is correct today, but its correctness lives in three separate consumers rather than at the read — and I have deliberately not touched it here.

Design decisions for maintainer review

• The freshness gate is included rather than deferred. Without it this fix would, on its first bad upstream day, turn a silent-empty domain into a confidently-wrong one. The budgets are derived, but if you would rather land the unwrap alone and gate separately, the two are cleanly separable.
• fetchInputData and INPUT_KEYS are now exported so the test can drive the real reader instead of a copy. computeCorrelation is left unexported — the field reads are pinned through fetchInputData's output, which is the seam the defect actually lives at.
• The hasAnyData tripwire is left as-is. With the unwrap in place it means what it says again. Making it stricter (say, requiring a minimum number of found keys) would be a behaviour change with its own alerting implications rather than a repair.
• The legacy Array.isArray fallbacks in computeCorrelation are left in place. They are dead for the keys that migrated, but they are what makes the read tolerant if any of these writers is ever rolled back, and removing them is not this PR's job. There is a test pinning that path.

Verification

tests/correlation-envelope-reads.test.mjs        16 pass, 0 fail   (new)
+ seeder-validation-floors, correlation-runtime-mode,
  cross-source-signals-envelope-reads,
  regional-snapshot-envelope-unwrap, seed-envelope-parity
                                                105 pass, 0 fail

Every guard is mutation-proven:

Mutant	Red
envelope unwrap removed (the pre-fix bare parse)	11
null-payload guard removed	1
freshness gate removed	2
JSON.parse folded inside unwrapEnvelope	1
unrest budget collapsed to the 30-minute default	1
earthquake budget widened to a day	1

No survivors.

Each of the seven enveloped fixtures asserts both directions: that the payload field survives the read, and that the pre-fix shape does not expose that field at all — so the bug stays pinned rather than merely fixed. Fixtures are driven through an asReadByTheSeeder() helper built on the real unwrapEnvelope, so a test cannot assert against a shape the reader could not produce. INPUT_KEYS itself is pinned, so adding a tenth key forces a decision about its freshness budget.

The existing tests/seeder-validation-floors.test.mjs and tests/correlation-runtime-mode.test.mts never touched the read path — the first covers only declareRecords/validateFn, the second greps the source text — which is why this was invisible.

Other gates:

npm run typecheck        clean
npx biome check          clean (2 files)
npm run lint:boundaries  no violations
check-unicode-safety     2652 files scanned, clean

npm run test:data: identical failure set to origin/main — 47 failing test names on both, comm diff empty in both directions.

scripts/audit-railway-watch-paths.mjs needs the railway CLI, which I do not have locally, so I verified the bundle claim structurally instead: scripts/_seed-envelope-source.mjs is present in the seed-bundle-derived-signals watchPatterns array in scripts/railway-services.json.

Out of scope

• The correlation epic (epic(correlation): entity-resolution-first correlation improvements #5981) and specifically feat(correlation): unify Railway correlation seeding with shared clustering #5986, which plans to replace this seeder's hand-copied clustering with the shared contract. That work is downstream of these reads and inherits whatever they produce; today it would inherit three empty domains. This PR does not touch clustering, scoring, thresholds or runtime modes.
• seed-forecast-bets.mjs's scattered unwrapping, described in the audit above. Correct today, structurally fragile, and its own change.
• seed-military-flights.mjs not using runSeed. It is the reason the military domain still works, so migrating it is a change that should be made deliberately and with this reader already envelope-aware — which it now is.
• No change to unwrapEnvelope, to any writer, or to the published card shape.

Type of change

• Bug fix
• New feature
• New data source / feed
• New map layer
• Refactor / code cleanup
• Documentation
• CI / Build / Infrastructure

Affected areas

• Map / Globe
• News panels / RSS feeds
• AI Insights / World Brief — the correlation cards behind the Military/Escalation/Economic/Disaster panels
• Market Radar / Crypto
• Desktop app (Tauri)
• API endpoints (/api/*)
• Config / Settings
• Other: scripts/seed-correlation.mjs (Railway derived-signals bundle)

Checklist

• Tested on worldmonitor.app variant — N/A. The change is inside a Railway seeder's Redis read path; exercising it in production needs Upstash and Railway credentials I do not have. Verified through the real reader with a stubbed Upstash pipeline, and with mutation proof that each guard has teeth. The observable effect to look for after deploy is the existing [Correlation] inputs: line at :735 reporting non-zero protests/outages/markets/news, and non-zero signals on the escalation, economic and disaster lines.
• Tested on tech.worldmonitor.app variant (if applicable) — N/A, no variant-specific behaviour.
• New RSS feed domains added to api/rss-proxy.js allowlist (if adding feeds) — N/A, no feeds added.
• No API keys or secrets committed
• TypeScript compiles without errors (npm run typecheck)

Documentation Alignment Checklist

N/A — no documentation claim is published or changed. The fix restores reads that were already specified; no methodology, API/MCP contract, generated doc or example changes. Listed for completeness:

• Claim ledger attached or linked — N/A, no documented claim changes.
• All required Audit Council role signoffs attached — N/A, no methodology or contract change.
• Generated docs regenerated from proto where applicable — N/A, no proto change.
• Fixture-backed examples recomputed — N/A, no published example depends on this seeder.
• Redis writers/readers enumerated for every documented key — the per-key writer table above covers all nine inputs. No key is added, removed, or written differently; the five outputs (correlation:cards-bootstrap:v1 plus the four per-domain keys) keep their existing shape and TTL.

[vercel]

@Yigtwxx is attempting to deploy a commit to the World Monitor Team on Vercel.

A member of the Team first needs to authorize it.

[Yigtwxx]

Closing and reopening only to re-trigger CI — no code change, head stays cf6ead80. variant-smoke-full failed on e2e/dashboard-news-request-budget.spec.ts, which this PR cannot reach; details in the comment that follows.

[Yigtwxx]

Context on the close/reopen above — it was a CI re-trigger, nothing changed. Head is still cf6ead80.

The first run after the rebase went red on variant-smoke-full, in
e2e/dashboard-news-request-budget.spec.ts:1055 ("an early scroll waits for slow-tier readiness and hydrates without a second gesture", #5876). It failed the gating assertion at line 1088:

post-mount panel priming must remain gated while the slow tier is pending
- Array []
+ Array [ "http://127.0.0.1:4173/api/market/v1/list-stablecoin-markets" ]

One stablecoin request escaped before slowBootstrap.release(). Both the first attempt and retry #1 failed the same way; the other 40 tests passed.

That can't originate here: this PR's diff is scripts/seed-correlation.mjs and tests/correlation-envelope-reads.test.mjs, neither of which is referenced from src/ or e2e/, so nothing in it reaches the browser bundle the smoke exercises. On the same base (06066aff1) and in the same window, variant-smoke-full passed on #6044, on #5897, and on main's own push run.

I don't have re-run rights on this repo, so close/reopen was the only way to re-trigger. The job passed on the second run against the identical SHA, and gate is green again. The checks list still shows the first run's red variant-smoke-full alongside the second run's green one; the commit status is the authoritative one.

Worth flagging rather than burying, though: that assertion is a timing race that reproduced twice in a row on one loaded runner, so it can take an unrelated PR red again.

For the record, the rebase itself: onto main (06066aff1), 141 commits behind before it, replays with no conflicts, diff unchanged at +324/-5 across 2 files, and npx tsx --test tests/correlation-envelope-reads.test.mjs is 16/16 on the rebased tree. No file this PR touches was modified upstream in those 141 commits.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
worldmonitor	Ready	Preview	Aug 9, 2026 5:26pm

[koala73]

I couldn't make changes to this PR because you maintainer edits disabled - it's now superseded by #6385, which includes this PR’s original commit and the follow-up hardening. Closing without merging.