feat(courseware): detect answer keys with one rule and a model check

[blarghmatey]

What are the relevant tickets?

Stacked on #3032. Witan task Unify answer-key detection for courseware documents across Open edX and OCW, and add a model-side check.

Description (What does it do?)

Two heuristics keep answer keys out of int__courseware__extraction_sources. courseware_answer_document reads only an Open edX file's path, and #3031 has its own courseware_answer_key over an OCW resource's name, title and text. Each misses what the other catches, e.g. Open edX keys whose name says nothing but whose opening reads "Pre-Assessment Answer Key" or "Homework 2 Sample Solutions".

This replaces both with one rule set and adds a model check for what a rule can't see:

• courseware_answer_key(name, title, text_head, marker_count) in src/ol_dbt/macros/courseware_answer_key.sql, under feat(courseware): add OCW extraction scopes #3031's macro names and signature. Open edX documents go through it with their path minus the extension, no title, and their text. courseware_answer_documents.sql is deleted.
• courseware_ocw_resource_name and courseware_ocw_answer_resource hold feat(courseware): add OCW extraction scopes #3031's non-page predicate (a Solutions tag, or the rule for anything that isn't a video, a transcript or tagged Videos), so feat(courseware): add OCW extraction scopes #3031 can call them after its rebase.
• int__courseware__answer_document_review is the reviewer queue: every document the rule excludes on either platform (to spot teaching material dropped by mistake), plus the documents still read whose latest model verdict flags them (keys the rule missed).
• ml.lib.courseware.answer_check sends each document's opening and end (the first 4,000 and last 2,000 characters of a long one) to Haiku, by default, in batches of 20. The verdict is stored per source with the text's checksum and a check_version (a hash of the checker, prompt, schema, excerpt sizes, model and effort), so only changed text is sent again and a copy of a text already checked (a B2B course copy) reuses the verdict without a call. A batch that fails, or whose verdict ids repeat or fall outside the listing (e.g. counted from 1), is dropped and checked on the next run.
• local answer-check runs it, and --dry-run reports how many sources need a verdict and how many distinct texts would be sent, without building a client.

Rule changes against the two old heuristics:

• Open edX now also reads the first 300 characters of the text, and counts three or more answered "Solution:"/"Answer:" labels in a document named as graded work. It also gets the OCW name tokens a solution file carries glued to other letters (pset3sol, 034pfsoln) and "key" at the end after a number or an assessment (exam2_key).
• The OCW name rule that matched a sol, sols, ans or answer(s) token without the sol-gel and "to/for answers" exceptions is dropped. The word rules match sol, sols, ans and answers with those exceptions. A new name rule catches a session or problem-set number joined to sol (ses5_solis_wets).
• For both: "final" counts as graded work only as an exam (not ..._FINAL_updated or vFinal) or as a title that is only "Final". A key in the opening must head the text ("Answer Key for Homework 3" counts, "key concepts" and "answer key questions" are prose). A label counts only when something other than a blank line or a line of underscores follows it, because an exam leaves "Answer:" blank.

How can this be tested?

• ol-dbt unit-test -s "int__courseware__answer_document_review int__courseware__extraction_sources" passes. The review model's test covers rule cases on both platforms and the latest-verdict choice. The extraction sources test checks that an Open edX document whose name or opening says answer key is not read and a reading that works examples is.
• uv run python -m pytest ml_tests in dg_projects/ml: 263 passed, including the answer check's tests for reuse, re-checks on a new version or changed text, failed and misnumbered batches, and the dry run making no calls.
• prek run --from-ref courseware-openedx-static-refs --to-ref HEAD passes.
• 51 cases compared feat(courseware): add OCW extraction scopes #3031's OCW macro with this one on DuckDB. None of feat(courseware): add OCW extraction scopes #3031's own cases flip. ses5_solis_wets is now excluded, and exams with blank "Answer:" labels, sol_gel and "key concepts" are now read.
• On the local Open edX table (83,225 documents), everything the old path rule excluded (568) is still excluded, and 24 more documents (12 distinct paths) are. That was measured before the fix commit loosened three of the rules, so the count in the design doc needs a re-measure.

Additional Context

• A model flag only puts a document in the review queue. It doesn't drop it from extraction. Whether a flag should drop a document on its own is an open question.
• There is no seed for reviewer decisions yet.
• The answer check isn't wired into Dagster; the assets are on feat(ml): run courseware extraction, resolution, description and graph as Dagster assets #3035.
• Only document sources are checked by default.
• A bare singular "answer" in a name with no assessment word is now read, where feat(courseware): add OCW extraction scopes #3031's macro excluded it.
• OCW corpus counts are not re-measured.

After #3031 is rebased onto this branch it must delete src/ol_dbt/macros/courseware_ocw_answers.sql (the macro names collide) and call courseware_ocw_answer_resource. The OCW exclusion counts in design doc section 8 then need re-measuring.

https://claude.ai/code/session_01QJTJTyry3Ep28ndqEfqUgK

[github-actions]

🔎 ol-dbt impact — column-level blast radius

0 breaking, 0 surrogate-key regeneration, 1 warning, 2 info across 3 changed model(s).

Details

• ⚠️ macro: courseware_answer_key.sql → 2 downstream model(s)
• ℹ️ int__courseware__answer_document_review — check_version, checked_at, courserun_readable_id, is_excluded_by_rule, is_flagged_by_model, model_id, model_reason, platform, review_reason, source_name, source_ref, source_title, text_head → 0 downstream model(s)
• ℹ️ macro: courseware_answer_documents.sql → 0 downstream model(s)

_{Posted by ol-dbt impact (annotate-only — does not block merge).}