Skip to content

fix(backend): discover Ohio SoS artifacts by product label, not file name - #551

Merged
shu1513 merged 1 commit into
mainfrom
claude/ohio-live-run
Aug 6, 2026
Merged

shu1513 merged 1 commit into
mainfrom
claude/ohio-live-run

Conversation

@shu1513

@shu1513 shu1513 commented Aug 6, 2026 •

Copy link
Copy Markdown
Owner

Found during the first attended live run: the Ohio raw-data refresh could not resolve a single artifact.

The bug

listOhioSosPortalFiles searched every listing row for a *.CSV file name. The file-transfer page doesn't publish file names — it lists product labels:

label size id
Active Candidate List 122 KB 120
Candidate Contributions--2026 91023 KB 6768
PAC COVER PAGES 8317 KB 3431

Result: 303 rows listed across the four tabs, zero matched, all 17 required artifacts reported missing. The label-discovery path had never completed against the portal — the acquisition spike downloaded through a different route, so PR 4 shipped this untested against live HTML.

The fix

Each product maps to its exact label on its own list page (ohioSosProductFromListingRow). The cache file name is then derived from the matched product, so planOhioSosCycleDownloads, the download loop, and the manifest code are untouched.

Exactness and tab-scoping are load-bearing. The NEW tab lists per-committee files whose labels differ from the statewide annual only by a committee name between single dashes:

Candidate Contributions--2026                      ← statewide annual (133 MB)
Candidate Contributions-DAVE YOST FOR OHIO-2026    ← one committee (5 KB)

A prefix or substring match would silently install one candidate's file as the entire state's contributions. Matching is full-string, case-insensitive (the portal renders PAC COVER PAGES in caps), and scoped to the product's own list type. Tests pin all 11 label forms verbatim from the live portal, plus the per-committee trap from both the NEW and CAN tabs.

Verification (live, attended)

  • Discovery: 17/17 resolved, missing_file_names: [], ids matching the spike's recorded map.
  • Refresh: 17/17 downloaded, 0 failures, manifests written for the first time.
  • 31-U bundle un-staled: 28/28 detail reports (was 13, with 15 missing ≈ $5.9M invisible). Two reports fail reconciliation (481068415 −$86,455.17, 506135809 −$418.40) and quarantine per decision 4 instead of publishing.
  • Sync: 189/189 candidates, 0 failed; receipts $104,396,358.50; outside spending published for the first time — $9,034,464.14 (V-PAC oppose $8,211,114.50 + support $600,000; Ohioans for a Healthy Economy oppose $217,704.16); 0 ambiguous targets.
  • npm run typecheck + npm test green (6,362 passed).

One data note worth recording: CAC_CON_2026.CSV downloaded at 133 MB / 593,131 rows against a listed 91,023 KB. Verified legitimate rather than assumed — every row is RPT_YEAR=2026 across 341 distinct committees, and the row count scales with the byte count, so it's a post-filing-deadline batch and the portal's size column is stale.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes

    • Improved Ohio finance artifact discovery by matching exact portal labels and transaction years.
    • Prevented unrelated or committee-specific files from being selected for statewide processing.
    • Preserved artifact download details and modification dates during listing processing.
  • Documentation

    • Documented the first live Ohio finance run, including artifact downloads, report processing, reconciliation results, and remaining PDF work.
    • Added guidance for refreshing stale data after artifact availability changes.

…name

The file-transfer page lists product labels ("Candidate
Contributions--2026") and never publishes file names, but discovery
searched each listing row for a *.CSV name. Every one of the 17 required
artifacts was therefore reported missing and no refresh could run.

Each product now maps to its exact label on its own list page, and the
cache file name is derived from the matched product, so the download
plan, download loop, and manifest code are unchanged.

Exactness is load-bearing: the NEW tab lists per-committee files whose
labels differ from the statewide annual only by a committee name between
single dashes ("Candidate Contributions-DAVE YOST FOR OHIO-2026" vs
"Candidate Contributions--2026"), so a prefix match would install one
committee's file as the whole state's. Tests pin all 11 label forms
verbatim from the live portal plus that trap.

Verified end to end against the portal: 17/17 artifacts resolved, all 28
Form 31-U detail reports fetched, 189/189 candidates synced.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Aug 6, 2026 •

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 510defda-73a5-472c-9e29-dd486d5c58f9

📥 Commits

Reviewing files that changed from the base of the PR and between 30869ae and 970afb3.

📒 Files selected for processing (3)
  • backend/src/pipeline/ohioFinance/ohioSosArtifactAcquisition.ts
  • backend/tests/pipeline/ohioFinance/ohioSosArtifactAcquisition.test.ts
  • ohio_plan.md

📝 Walkthrough

Walkthrough

Ohio SoS artifact discovery now matches exact normalized portal labels to required products and transaction years. The pipeline derives cache filenames from resolved metadata, skips unrelated listings, and records the completed live refresh and synchronization results.

Changes

Ohio SoS artifact discovery

Layer / File(s) Summary
Label resolution and artifact acquisition
backend/src/pipeline/ohioFinance/ohioSosArtifactAcquisition.ts
The pipeline resolves list-scoped portal labels to product keys and transaction years. It skips non-required rows and derives cache filenames from artifact metadata.
Resolution validation and live-run record
backend/tests/pipeline/ohioFinance/ohioSosArtifactAcquisition.test.ts, ohio_plan.md
Tests cover statewide labels, annual years, case normalization, wrong-tab labels, committee-specific labels, and unrelated rows. The plan records the completed refresh and synchronization results.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant OhioSoSPortal
  participant LabelResolver
  participant ArtifactAcquisition
  OhioSoSPortal->>LabelResolver: listing cells and list type
  LabelResolver->>ArtifactAcquisition: product key and transaction year
  ArtifactAcquisition->>ArtifactAcquisition: derive cache filename
Loading

Possibly related PRs

  • shu1513/voteapp#534: Extends the same Ohio finance artifact-acquisition pipeline with planned label-based SoS discovery.
  • shu1513/voteapp#538: Refines listing-label resolution and artifact metadata handling in the same module.
  • shu1513/voteapp#545: Updates the same artifact-acquisition module and its tests.
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: discovering Ohio SoS artifacts by product label instead of file name.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch claude/ohio-live-run

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Repository owner deleted a comment from coderabbitai Bot Aug 6, 2026
@shu1513

shu1513 commented Aug 6, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai full review

@coderabbitai

coderabbitai Bot commented Aug 6, 2026 •

Copy link
Copy Markdown
✅ Action performed

Full review finished.

@shu1513
shu1513 merged commit f6ce938 into main Aug 6, 2026
6 checks passed
@shu1513
shu1513 deleted the claude/ohio-live-run branch August 6, 2026 07:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant