Skip to content

feat(courseware): extract each distinct scope text once and reuse it - #3040

Open
blarghmatey wants to merge 4 commits into
courseware-stale-headerfrom
courseware-content-address
Open

blarghmatey wants to merge 4 commits into
courseware-stale-headerfrom
courseware-content-address

Conversation

@blarghmatey

Copy link
Copy Markdown
Member

Stacked on #3038, for the task "Content-address courseware extraction so B2B course copies reuse one extraction". extraction_key hashes the platform, run and scope id, and B2B copies of a course share its text under new block ids, so every copy was sent to the model again. The local lake has 107,205 scopes holding 43,885 distinct scope texts (1.08 of 2.74 billion source characters), so about 60% of extraction input is copies.

This adds content_key, which hashes what a call reads apart from the course header (extractor id, prompt version, schema version, model, effort, scope kind and title, part and input_text_checksum) and leaves out the run and scope id. extract_scopes sends one call per content key, or takes a stored extraction with that key from the new reusable argument. Each copy still gets its own extraction row under its own extraction_key (stop_reason = 'reused', zero tokens) and its own claim and evidence rows, so covered_keys, rekey and the dbt models work unchanged (only two column descriptions changed). Evidence takes the copy's own source_ref by source id, since the model sees S01, S02 and not block ids, and verification and offsets carry over because the text is identical. A failed call leaves its copies out, so the next run retries them. local extract passes its stored tables as reusable, and the dry run marks copies reused.

A copy records the original's header_version, not its own, because its claims were made under the original's header and covered_keys hashes header_version into the header-keyed key. With its own header, a later --key-on-header run would count the copy as already extracted under that header and skip it. With the original's, is_header_stale from #3038 flags the copy and --key-on-header re-extracts it.

Measured with dry runs: xPRO MLx2 R14 to R16 reuses 190 of 288 scope parts and sends ~450k of ~1.28M input tokens. The task's figure of 6,224 runs holding 3,260 distinct texts didn't reproduce; the local lake has 6,222 distinct run checksums with ids and 5,823 without.

Testing: 279 ml tests pass, including local extract run twice over a run and its copy (one model call, the copy stored with the original's header and its own claims, evidence and scope_input row) and a --key-on-header run re-extracting a copy made under another header. ol-dbt unit-test passes.

Open questions for review:

  • Reuse ignores the course header across courses, consistent with the 2026-10-17 decision to keep the header out of the key. That means identical boilerplate in unrelated courses shares claims.
  • An untitled source's block id is its title in the message, which keeps about 7,000 more scopes from deduplicating. Fixing that changes the message, so it needs an EXTRACTOR_ID bump.

Not covered: the Dagster extraction asset in #3035 doesn't pass stored rows as reusable, so production gets none of the savings until it does.

https://claude.ai/code/session_01QJTJTyry3Ep28ndqEfqUgK

@github-actions

Copy link
Copy Markdown

🔎 ol-dbt impact — column-level blast radius

✅ No column-level downstream impact detected for the changed models.

Posted by ol-dbt impact (annotate-only — does not block merge).

…o course copies

The extraction key hashes the run and the scope id, and B2B copies of a course
share its text under new block ids, so every copy was sent to the model again.
On the local lake's sources, 107,205 scopes hold 43,885 distinct texts (1.08 of
2.74 billion source characters), so about 60% of extraction input is copies.

content_key hashes what a call reads apart from the course header (extractor
version, scope kind, title, split and source text) without the run or scope id.
extract_scopes now sends one call per content key and reuses a stored
extraction with that key (the new reusable argument). Each copy still gets its
own extraction row under its own extraction_key, with stop_reason 'reused' and
zero tokens, and its own claim and evidence rows, so covered_keys, rekey and
every dbt model work unchanged. Evidence takes the copy's own source_ref by
source id, since the model sees S01, S02 and not block ids; verification and
offsets carry over because the text is identical.

A dry run over xPRO MLx2 R14 to R16 marks 190 of 288 scope parts reused and
sends ~450k of ~1.28M input tokens. The Dagster asset (courseware-dagster-assets)
still has to pass stored rows as reusable.

Claude-Session: https://claude.ai/code/session_01QJTJTyry3Ep28ndqEfqUgK
A reused row took header_version from its own scope, but its claims were made
under the original's course header. covered_keys hashes header_version into the
header-keyed key, so a later key_on_header run counted the copy as extracted
under its own header and skipped it, and a third copy with that header could
reuse claims no call made under it. The row now carries the original's
header_version, so key_on_header re-extracts a copy whose header differs.

Also shares the per-row content key computation between reusable_content_keys
and _reusable_extractions, which the dry run and the real run both depend on.

Claude-Session: https://claude.ai/code/session_01QJTJTyry3Ep28ndqEfqUgK
The content-address tests called extract_scopes directly, so nothing ran a copy
through local extract now that it sits on the crash-safe write order, the schema
check and header-recording scope inputs. This runs local extract twice over a
run and its B2B copy and checks the copy is stored with no call, carries the
original's header_version, gets its own claims and evidence, and that its
scope_input row records the copy's own header, which is what lets
dim_courseware_extraction flag it as header-stale.

Claude-Session: https://claude.ai/code/session_01QJTJTyry3Ep28ndqEfqUgK
@blarghmatey
blarghmatey force-pushed the courseware-stale-header branch from 492cdc2 to 6797fa2 Compare October 18, 2026 13:50
@blarghmatey
blarghmatey force-pushed the courseware-content-address branch from 9d2f723 to 327ac70 Compare October 18, 2026 13:50
@blarghmatey
blarghmatey marked this pull request as ready for review October 18, 2026 22:18
test_courseware_content_address imported the stubs from ml_tests.test_courseware.
CI runs the pytest script, which does not put ml_tests on sys.path, so the module
failed to collect. Local runs went through the python module runner, which adds
the working directory and hid it. The stubs now live in a stub_llm_client fixture.

Claude-Session: https://claude.ai/code/session_01QJTJTyry3Ep28ndqEfqUgK

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant