Repository navigation
Regenerate TICON datums on every core - #245
Merged
Merged
Conversation
A forced datum regeneration spends about a second of CPU per station synthesizing the 19-year epoch, and the import ran all 4,838 stations in one process. `./import` now runs one shard per core (SHARDS overrides the count), each taking every nth station, and fetches GESLA before the shards start so they don't race to extract it. Each station depends only on its own data, so the output is identical to a single-process run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
commit: |
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Resolve the GESLA preparation race and validate that the shard count is a positive integer.
2 open findings
What changed in this PR
Parallelizes TICON imports across configurable shards and prefetches GESLA data.
Changes:
- Adds round-robin shard selection via
SHARDS/SHARD. - Launches parallel importer processes.
- Documents shard configuration and GESLA prefetching.
| File | Summary |
|---|---|
sources/ticon/README.md |
Documents parallel import behavior and shard configuration. |
sources/ticon/import.ts |
Filters stations by shard; cached fallback paths can race during GESLA preparation. |
sources/ticon/import |
Launches shard workers; needs safe GESLA preparation and validation for non-positive shard counts. |
🧠 Review effort: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
A cache-reuse import skips the up-front GESLA fetch, but a station with no cached datums or harmonics still calls `ensureGeslaData`, and every shard could then download and extract into the same paths at once. `unzip` also fills GESLA_DIR in place, so a shard could read it half-extracted. The download and the extract now go to per-process paths and are renamed into place. `./import` also rejects a SHARDS that is not a positive integer instead of importing nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
npm run import -w sources/ticonnow runs one process per core, each taking every nth station. A forced datum regeneration spends seconds of CPU per station synthesizing the 19-year epoch, and all 4,838 stations ran in a single process.SHARDSsets the count, andSHARDS=1runs one process. GESLA is downloaded before the processes start so they don't race to extract it.A forced regeneration of the 304 meds stations (
FORCE_DATUMS=meds) on this 20-core Mac:maininstalls)The per-core run's station files are byte-identical to the single-process run, both for the forced meds regeneration and for a run that reuses cached datums. Each station depends only on its own data. Round-robin assignment leaves some processes finishing early, since GESLA file sizes vary tenfold, but at 33 s that isn't worth a work queue.
Regenerating on
mainalso rewrites the name or region of 569 stations, with or without this change. That is #244.🤖 Generated with Claude Code