Repository navigation
feat(courseware): extract each distinct scope text once and reuse it - #3040
Open
blarghmatey wants to merge 4 commits into
Open
blarghmatey wants to merge 4 commits into
blarghmatey wants to merge 4 commits into
Conversation
🔎 ol-dbt impact — column-level blast radius✅ No column-level downstream impact detected for the changed models. Posted by |
…o course copies The extraction key hashes the run and the scope id, and B2B copies of a course share its text under new block ids, so every copy was sent to the model again. On the local lake's sources, 107,205 scopes hold 43,885 distinct texts (1.08 of 2.74 billion source characters), so about 60% of extraction input is copies. content_key hashes what a call reads apart from the course header (extractor version, scope kind, title, split and source text) without the run or scope id. extract_scopes now sends one call per content key and reuses a stored extraction with that key (the new reusable argument). Each copy still gets its own extraction row under its own extraction_key, with stop_reason 'reused' and zero tokens, and its own claim and evidence rows, so covered_keys, rekey and every dbt model work unchanged. Evidence takes the copy's own source_ref by source id, since the model sees S01, S02 and not block ids; verification and offsets carry over because the text is identical. A dry run over xPRO MLx2 R14 to R16 marks 190 of 288 scope parts reused and sends ~450k of ~1.28M input tokens. The Dagster asset (courseware-dagster-assets) still has to pass stored rows as reusable. Claude-Session: https://claude.ai/code/session_01QJTJTyry3Ep28ndqEfqUgK
A reused row took header_version from its own scope, but its claims were made under the original's course header. covered_keys hashes header_version into the header-keyed key, so a later key_on_header run counted the copy as extracted under its own header and skipped it, and a third copy with that header could reuse claims no call made under it. The row now carries the original's header_version, so key_on_header re-extracts a copy whose header differs. Also shares the per-row content key computation between reusable_content_keys and _reusable_extractions, which the dry run and the real run both depend on. Claude-Session: https://claude.ai/code/session_01QJTJTyry3Ep28ndqEfqUgK
The content-address tests called extract_scopes directly, so nothing ran a copy through local extract now that it sits on the crash-safe write order, the schema check and header-recording scope inputs. This runs local extract twice over a run and its B2B copy and checks the copy is stored with no call, carries the original's header_version, gets its own claims and evidence, and that its scope_input row records the copy's own header, which is what lets dim_courseware_extraction flag it as header-stale. Claude-Session: https://claude.ai/code/session_01QJTJTyry3Ep28ndqEfqUgK
blarghmatey
force-pushed
the
courseware-stale-header
branch
from
October 18, 2026 13:50
492cdc2 to
6797fa2
Compare
blarghmatey
force-pushed
the
courseware-content-address
branch
from
October 18, 2026 13:50
9d2f723 to
327ac70
Compare
blarghmatey
marked this pull request as ready for review
October 18, 2026 22:18
test_courseware_content_address imported the stubs from ml_tests.test_courseware. CI runs the pytest script, which does not put ml_tests on sys.path, so the module failed to collect. Local runs went through the python module runner, which adds the working directory and hid it. The stubs now live in a stub_llm_client fixture. Claude-Session: https://claude.ai/code/session_01QJTJTyry3Ep28ndqEfqUgK
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #3038, for the task "Content-address courseware extraction so B2B course copies reuse one extraction".
extraction_keyhashes the platform, run and scope id, and B2B copies of a course share its text under new block ids, so every copy was sent to the model again. The local lake has 107,205 scopes holding 43,885 distinct scope texts (1.08 of 2.74 billion source characters), so about 60% of extraction input is copies.This adds
content_key, which hashes what a call reads apart from the course header (extractor id, prompt version, schema version, model, effort, scope kind and title, part andinput_text_checksum) and leaves out the run and scope id.extract_scopessends one call per content key, or takes a stored extraction with that key from the newreusableargument. Each copy still gets its own extraction row under its ownextraction_key(stop_reason = 'reused', zero tokens) and its own claim and evidence rows, socovered_keys,rekeyand the dbt models work unchanged (only two column descriptions changed). Evidence takes the copy's ownsource_refby source id, since the model sees S01, S02 and not block ids, and verification and offsets carry over because the text is identical. A failed call leaves its copies out, so the next run retries them.local extractpasses its stored tables asreusable, and the dry run marks copiesreused.A copy records the original's
header_version, not its own, because its claims were made under the original's header andcovered_keyshashesheader_versioninto the header-keyed key. With its own header, a later--key-on-headerrun would count the copy as already extracted under that header and skip it. With the original's,is_header_stalefrom #3038 flags the copy and--key-on-headerre-extracts it.Measured with dry runs: xPRO MLx2 R14 to R16 reuses 190 of 288 scope parts and sends ~450k of ~1.28M input tokens. The task's figure of 6,224 runs holding 3,260 distinct texts didn't reproduce; the local lake has 6,222 distinct run checksums with ids and 5,823 without.
Testing: 279 ml tests pass, including
local extractrun twice over a run and its copy (one model call, the copy stored with the original's header and its own claims, evidence and scope_input row) and a--key-on-headerrun re-extracting a copy made under another header.ol-dbt unit-testpasses.Open questions for review:
EXTRACTOR_IDbump.Not covered: the Dagster extraction asset in #3035 doesn't pass stored rows as
reusable, so production gets none of the savings until it does.https://claude.ai/code/session_01QJTJTyry3Ep28ndqEfqUgK