Repository navigation
Add the Claude Haiku 5.5 model card and metadata - #201
Merged
Merged
Conversation
The Models API lists claude-haiku-5-5 with created_at 2026-10-07, the
release date its model overview gives ("Released October 7, 2026"). A
request with no thinking parameter returns a thinking block (119 of 123
output tokens on the one-question probe), so it joins the thinking-default
Claude class. Unlike Sonnet 5.5 and Opus 5.5, the API accepts forced tool
use on this model, and the forced-tool probe returned no thinking block.
The card therefore takes the board's forced answer tool, as Claude Opus 5's
row does, and answers without extended thinking (eval_no_tools'
_tool_choice_for documents the mechanism). Gauntlet: 3/3 and 16/16 parsed
on the tool contract, $0.001 per scenario; two-scenario smoke: 36/36 rows,
no errors, $0.0025.
Priced at $0.10/$0.50 per 1M with $0.125 cache writes and $0.01 cache
reads, the pricing page's rates for prompts up to 100,000 tokens (read
2026-10-08). litellm's bundled backup map lacks the id, so eval_no_tools
registers it locally, as for Sonnet 5.5.
The display name joins MODEL_DISPLAY_NAMES and the app roster in the
release that folds the row into the board, since test_paper_results pins
the display names to the frozen 46-model roster.
Also corrects Sonnet 5.5's cache-read override from $0.20 to $0.10 per 1M.
The pricing page's footnote 2 prices Sonnet 5.5 and Opus 5.5 cache hits at
0.05x base input, and litellm's remote cost map now carries $0.10, so
test_local_claude_models_resolve_without_remote_cost_map failed on main
whenever the remote map loaded. Sonnet 5.5's board run billed no cache
tokens, so no recorded cost changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
MaxGhenis
added a commit
that referenced
this pull request
Oct 8, 2026
Renders the paper outside the sandbox with the installed Quarto 1.9.36 and re-pins it with freeze_snapshot.py --rendered-only (pdf sha256 9403564b...). The Louisiana audit README now records d994 (ruled 2026-10-06, after this release was built): the convention is kept as written, and both Louisiana references are excluded in the release after this one. Validation logs: render exit 0; app lint, 170 bun tests and the build exit 0; full pytest 1,651 passed, 9 skipped, 1 failed. The failure is test_local_claude_models_resolve_without_remote_cost_map[claude-sonnet-5-5], which fails on main too once litellm loads its remote cost map. PR #201 fixes it, and this branch takes the fix from main before it merges. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
Independent review: APPROVE at
Since the review: the 100-household board run finished 100/100, with 1,984 rows, no missing values or explanations, no errors, $0.13 total and a 5.1 s median latency per household. Its scores reach the board in the next data release. Merging on the gates: checks green, MERGEABLE, review approved at the current head. |
MaxGhenis
added a commit
that referenced
this pull request
Oct 9, 2026
Brings in #201 (Claude Haiku 5.5's model card), whose corrected Sonnet 5.5 cache-read price fixes the test_eval_no_tools failure on this branch's CI. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis
added a commit
that referenced
this pull request
Oct 9, 2026
…last answer (release dashboard-data-20261006) (#202) * Date the model response window from the last answer, not the release date The release drivers set the manifest's model_response_date to "2026-06-12 to <release date>". Release dashboard-data-20260930 recorded an end of 2026-09-30, but the last answer on that board (GPT-6.1 Sol) completed at 2026-09-29 22:36:32 UTC and no model answered on the 30th. freeze_snapshot.model_response_window now derives the end as the UTC date of the latest request_completed_at in the predictions the freeze gzips into the snapshot. main() derives it before its first write and passes it to build_manifest. The start stays configured (MODEL_RESPONSE_START), because the first waves' rows carry no timestamps and the run label gives their date. A driver that still sets MODEL_RESPONSE_DATE must name the derived window, or the freeze refuses. Both drivers stop setting it, and freeze_gpt61sol derives the window in its preflight so --dry-run reports it. The live manifest is not changed here: correcting it is a published-claim change queued for Max (cos decision d831). A strict xfail test records the live mismatch and fails once a freeze corrects the manifest, so that freeze must remove the mark. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Audit the state income tax withheld in federal SALT; propose three exclusions policyengine-us 2.15.17 fills the state income tax part of the federal SALT deduction with state_withheld_income_tax, an AGI-formula estimate of withholding, and no prompt states state income tax withheld or paid. This adds the audit (reference_audit/2026-10-05): a sweep of all 1,984 outputs under the reference, liability, net and zero readings (baseline reproduces all 1,928 scored references), the side readings the records quote, three proposed reference_depends_on_unlisted_input records (022 CA, 081 MA, 114 VA federal income tax), and the leaderboard impact scored through the analyze CLI on scratch copies. No reference, exclusion record or published number changes here. The exclusions change published scores and wait for Max's ruling. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Audit Louisiana's 2026 standard deduction behind two scored references scenario_051 and scenario_077 state income tax rest on policyengine-us's explicit 2026 Louisiana standard deduction of $12,835 (PR #8411), which no Louisiana publication states. The audit sources what Louisiana published, sweeps all 1,984 outputs under six candidate deductions on policyengine-us 2.15.17 with latest_final, and scores each option through the analyze CLI. It changes no reference. Before the 2026-07-03 freeze the Department of Revenue published only a provisional $12,875 (withholding tables and estimated-tax worksheet), saying the return amount would differ; RIB 26-019 (2026-09-28) set it at $12,838, which moves the references by $0.09 and no exact match. The audit recommends keeping both references and drafts the documentation; the ruling is Max's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Keep the sweep, scoring and check logs the audit README cites Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Audit optional employer pass-through in the payroll tax references; draft both records Four scored payroll_tax references (scenario_032 MN, 043 CO, 081 MA, 082 NY) count an employee share of a state paid-leave premium that the law lets the employer deduct but does not require. policyengine-us 2.15.17 counts the largest share the employer may deduct. The output asks for "mandatory employee state payroll taxes". reference_audit/2026-10-05-payroll classifies every state program in the payroll references from primary law (two adversarial reviews per program, none refuted), sweeps all 1,984 outputs with an output-scope adapter that drops the optional shares (four outputs move, nothing else), and drafts both candidate records: proposed_exclusions.json and proposed_regenerations.json. It scores each through the analyze CLI; the unchanged copy reproduces the published payload. Changes no reference or published number. Draft README; review pending. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Audit the Medicare Part B premium in medical expenses; propose excluding scenario_114's Virginia income tax policyengine-us 2.15.17 treats every Medicare-eligible person as enrolled (takes_up_medicare_if_eligible defaults to true) and adds the modeled Part B premium, $2,434.80 for 2026, to medical expenses. No prompt states Medicare enrollment or a premium. A sweep of all 1,984 outputs on the reference system with the premium at 0, and with enrollment off, moves exactly two: scenario_114 federal (10,729.61 -> 11,265.27, already proposed in #191) and Virginia (3,514.15 -> 3,654.15). No SNAP or eligibility output moves by any amount. Adds reference_audit/2026-10-05-medicare-part-b: the sweep, a per-household explanation, the drafted records (one for the Virginia output, and a fallback for the federal output if #191's record is not adopted) and the leaderboard impact under each combination with #191. Changes no reference or published number; the exclusion waits for Max's ruling. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Rest the payroll recommendation on the reference rules; record the independent review An eight-agent review of 8714770 (differential impact and sweep, mechanism, advocates for keep and regenerate with a judge, release checklist, claim audit) agreed on every number: 2,438 impact values, all 1,984 swept outputs bit for bit, and the 36 hand-computed payroll parts. The judge upheld exclusion but struck the draft's board-effect and model-behaviour arguments, which no rule uses. - README: the recommendation now rests on 2026-09-22/09-28 rule 4 and the r24, r25 and r14 precedents; regeneration is the strict reading of "mandatory". Adds both options' board effects side by side, the counter-evidence the reviewers found, the Acts 2026 c. 101 line mapping, the review summary and the release checklist. Fixes the claim audit's five errors and thirteen imprecisions. - proposed_exclusions.json: one shared unlisted input; the basis cites the rules and precedents (no engine input exists for the employer's choice). - payroll_mandatory_scope.py: docstring states its scope (DE, ME, VT and WA left for upstream). Sweep, proposals and impact rerun on the new hash; values unchanged. - verification/independent_reviews.json: the eight reports. Changes no reference or published number. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Name the payroll decision (cos d972) in the audit README Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Answer the take-up reading in the Part B records; add the legacy impact summary and independent reviews - State the prompt's "assume program take-up" sentence in both records and the README, and why it does not settle Medicare enrollment (the benchmark's DEFAULT_TAKEUP_INPUTS has no Medicare entry; the request asks eligibility only). - Name both ACA coverage tests in the mechanism section. - leaderboard_impact.py recomputes the freeze's legacy impact_summary_by_model.csv in every case (the unchanged copy reproduces the frozen file). The stale-weight check now reports that file, which reorders 10 of 46 models with 2.15.17's weights, so the finding is narrowed to "no change in the analyze payload". - Read #191's records from a sha256-checked copy at its head 8af912a, so this directory does not depend on #191's branch. - Add the four workflow reviewers' reports (verification/independent_reviews.json). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Correct scenario_031's Medicaid annotations to the engine's California disregard The case note, reference explanation and 43 row annotations for scenario_031 head_medicaid_eligible explained the reference (eligible) by subtracting a Medicare Part B premium from countable income. policyengine-us 2.15.17 subtracts no premium. It counts $23,853.47 of SSI unearned income and applies California's $230 monthly disregard in place of SSI's $20 exclusion. That gives $21,093.47 against $22,024.80 (138% of the 2026 guideline). Wording-only: every row stays llm_error with its subtype. The ledger (reference_audit/2026-10-05-medicaid-031-annotations/rewrites.json) is applied by scripts/apply_rewrites.py, which re-pins the manifest hashes, and tests/test_annotation_rewrites.py keeps it in force. engine_values.py asserts every figure on the reference system. payload_diff.py shows a re-export changes only this cell's text fields. The frozen run dir is untouched. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Point the Part B audit's annotation findings to #197 and the scenario_114 row check Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Apply the independent review's wording fixes to scenario_031's annotations Narrows 'holds only if the disregard is skipped' to the engine's limit, names gemini-3.5-flash's source for $23,853, drops 'about' before exact figures, rewords claude-fable-5, the case note's category clause and the explanation's limit phrase. Records immigration_status in engine_values.json and tightens payload_diff's reproduction check. Adds the review as verification/independent_review.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Keep CSV timestamps exact and test the real manifest builder's window Answers the hard-tier review of #188 (b6b385c): - model_response_window parses with float_precision=round_trip; pandas' default parser rounded the last double before 2027-02-01 UTC up to midnight and dated a January 31 answer February 1. New end-to-end CSV regression. - A test now calls the real build_manifest with a window no release has used and requires it in model_response_date and both reproducibility notes; a builder hard-coded to September 30 fails it (mutation f). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Add the release 20261006 driver, its spec and its tests scripts/release_20261006.py stages, exports and freezes release dashboard-data-20261006 from release 20260930 (read from git at 8b4c0ca): the eight ruled exclusion records and their adjudications, the record edits the three audits wrote, the annotation rewrites, and the derived response window. docs/release_20261006/spec.json holds its inputs. Gates: the base stage is 20260930's; the annotations differ from 20260930's only on the excluded and reworded outputs; two exports agree byte for byte; the payload differs from 20260930's only where the release allows; rebuilt with 20260930's exclusion record, the staged bundle reproduces 20260930's statistics exactly; the freeze leaves references, predictions and serving configuration as they were and changes the manifest only where the release allows. tests/test_release_20261006.py rebuilds each committed record from git and the spec, rescores the payload's rows independently, and checks the gates with Hypothesis properties. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Date the 2026-10-05 wave's decisions in UTC, document the driver, widen the in-force test - spec: the new wave's adjudications were written on 2026-10-06 UTC; - DATE_CONVENTIONS_NEW names Max's rulings of 2026-10-05 US Eastern time; - docs/release_20261006/design.md documents the driver's inputs, gates and order; - the in-force test strips any adjudication sentence before re-appending it; - rescore_sensitivity_summaries knows the word for 64. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * WIP: partial content edits written against the trial freeze Test pins, paper and docs prose, and a draft release note from the stopped content pass. Some are reviewed and some are not; the content pass after the fresh freeze finishes them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Add the release's annotation rewrite ledger and its evidence 306 whole-field rewrites in four groups, each drafted from its audit's evidence, checked by an adversarial verifier and revised: - salt (81): scenario_022 and scenario_081 federal income tax; the SALT state income tax is the engine's withholding estimate, not the household's; - s114 (67): scenario_114 federal and Virginia income tax; the withholding estimate and the imputed Medicare Part B premium; - payroll (151): the four paid-leave payroll outputs; optional employer pass-through shares, and scenario_032's and 043's note errors; - louisiana (7): scenario_051 and 077 case notes and rows; sourced $12,835 chain, no change to scored references. Each group's evidence file quotes the model explanations and engine values behind every changed number. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Record the d994 hold in the release spec * Freeze dashboard-data-20261006 and rescore sensitivity evidence * Pin the release note to the draft v2 contract and document the d994 hold * Render the paper, pin it, record d994's ruling and the validation logs Renders the paper outside the sandbox with the installed Quarto 1.9.36 and re-pins it with freeze_snapshot.py --rendered-only (pdf sha256 9403564b...). The Louisiana audit README now records d994 (ruled 2026-10-06, after this release was built): the convention is kept as written, and both Louisiana references are excluded in the release after this one. Validation logs: render exit 0; app lint, 170 bun tests and the build exit 0; full pytest 1,651 passed, 9 skipped, 1 failed. The failure is test_local_claude_models_resolve_without_remote_cost_map[claude-sonnet-5-5], which fails on main too once litellm loads its remote cost map. PR #201 fixes it, and this branch takes the fix from main before it merges. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Record the reviewed head Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Name the published rates behind Massachusetts's maximum; restore the pre-freeze tree The independent review of head 8b831f2 found that the release note and 32 annotation rewrites called $805.01 (0.46%) the most a Massachusetts employer may deduct. The payroll audit records a competing reading: a literal application of Acts 2026 c. 101 s. 45 would let a large employer deduct up to 0.772% ($1,351.02) for 2026, while the Department's rates page kept 0.46%. The texts now say "the most Massachusetts's published 2026 rates let the employer deduct": the ledger's scenario_081 payroll texts, its adjudication reasoning, a fifth record edit for the exclusion record, and the note. The exclusion and every score are unchanged. Also from the review: verify_payload now requires the staged payload's households and outputs to be the base's before it compares cells, with a test that a dropped cell or household is refused; the design note records d994's ruling instead of calling it pending. The freeze's outputs (5435bab) are restored to their pre-freeze state so prepare, export and freeze can run again on the corrected ledger. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Freeze dashboard-data-20261006 again on the corrected Massachusetts wording Payload f03441f64316997de843e82c27272442c794148a121c25e1343fbddbe38574a0, 127,224,818 bytes. Against the reviewed head 8b831f2 the freeze changes 31 row annotations, one case note, one adjudication's reasoning and one exclusion record's alternative reading, all for scenario_081's payroll output, and the hashes that follow. The analysis files and the sensitivity evidence are byte-identical, so no score moved. The paper is rendered and pinned again (pdf sha256 f3ccebc8...). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Point the effects record at the corrected payload; drop the session handoff drafts The summary CSV the effects record binds is byte-identical after the re-freeze, so every expected effect still holds; only the payload hash it names changes. The handoff drafts were one session's working notes, not repo documentation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Record the validation logs of the corrected freeze Full pytest on the merged head: 1,662 passed, 9 skipped, exit 0. App lint, 170 bun tests and the build: exit 0. Paper render: exit 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Record the reviewed head Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qualify Massachusetts's caps in the two remaining exclusion-record sentences The round-2 review found two record sentences that still stated Massachusetts's limits without the published-rates qualification: scenario_081's federal-tax note ('the largest share the employer may deduct') and its payroll record, which attributed the 100% family and 40% medical caps directly to s. 6(c). Acts 2026 c. 101 ss. 25-26 and 45 can be read to swap those caps for 2026. The note now names the published 2026 rates, and a sixth record edit qualifies the caps the same way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Restore the pre-freeze tree for the round-3 freeze Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Freeze dashboard-data-20261006 on the round-3 record edits Payload aa34e5c9ea926848dc7af460a98f17021bc42dae54a8015c9988a85e79a5d462, 127,225,036 bytes. Against round 2 only scenario_081's two exclusion-record sentences change; the analysis files and sensitivity evidence remain byte-identical to the first freeze, so no score moved. The release asset is replaced and verified against the pointer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Record the round-3 validation logs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Record the reviewed head Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis
added a commit
that referenced
this pull request
Oct 10, 2026
…6 (release dashboard-data-20261010, 47 models) (#208) * WIP: start the Haiku 5.5 release driver from finish_gpt61sol.py Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Pin Claude Haiku 5.5's finished run inputs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Stage the ten exclusions Max ruled on 2026-10-06 in the Haiku 5.5 driver docs/haiku55/spec.json lists d1022's eight records (pinned by sha256 in reference_audit/2026-10-05-reference-adversary/proposed_changes.json) and d994's two Louisiana records (reference_audit/2026-10-05-louisiana/ proposed_exclusions.json, reference_law_published_after_freeze: the 2026 return amount Louisiana published, $12,838, came after the freeze), with their adjudication classes and reasoning. - install-exclusions writes release 20261006's 64 records plus the ten into the stage and rebinds the record in stage.json. - adjudicate-exclusions decides the ten from each case's bound Opus 5.5 verdict, restating scenario_051's 2026-09-22 regeneration in place. - Triage accepts exactly those entries and the 2026-10-06 wave in the date conventions. - Export requires 1,910 scored outputs per model and, with 20261006's record put back, byte-identical incumbent modelStats. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Name Claude Haiku 5.5 and release 20261006 in the driver's messages Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Describe a later-law exclusion's frozen value as the engine's own figure The excluded-output note said the frozen value of a reference_law_published_after_freeze record "used the engine's projection". Louisiana's $12,835 (d994), the first such record, is a computation from published CPI-U, not a projection, so the note now says "the engine's own figure", which fits both. The Louisiana records state the ruling as the earlier records do ("decision d994"). Adds Claude Haiku 5.5's display name to the paper and app rosters for the release that folds it in. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Test the local Claude registration in a fresh interpreter without the remote cost map The card review of #201 found that the parametrized registration test reads litellm's already-loaded map, which the remote fetch can fill, so it passes without the local entry. This test disables the remote map in a subprocess, checks the bundled backup lacks the newest ids, and requires each locally registered model at its override prices. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Add the Haiku 5.5 release's freeze driver, tests and design note; fix three driver bugs - scripts/freeze_haiku55.py adapts freeze_gpt61sol.py. It rebuilds the payload from the receipt-bound stage, checks the installed exclusions, and writes the snapshot (74-record exclusion record), annotations, pointer and version label. It appends this release's wording amendments to release 20261006's committed list rather than replacing it. - tests/test_finish_haiku55.py ports the GPT-6.1 Sol driver tests and covers the new steps (reworded cases, spec records, exclusion build, ruled adjudications, date conventions, install, the scope gate), with Hypothesis properties. tests/test_freeze_haiku55.py does the same for the freeze, on fixtures only. - docs/haiku55/design.md: procedure, gates and invariants. Driver fixes the new tests caught: - a restated scenario_051 was rebuilt with release 20261006's judge fields, so triage, export and the freeze would refuse it; - reworded_since_seed compared rows with their pandas index, so a reordered file would read as reworded; - a re-export after the freeze was refused at the committed exclusion record, which the freeze replaces with the release's. The spec's record_edits now name each engine defect's upstream fix (#9928, #9946, #9948, merged 2026-10-09; Ohio #10020, open, #9925 related). The later-law docstring no longer calls the frozen value a projection. 592 passed, 2 skipped, 1 strict xfail (the spec's adjudications_written_on, set when the adjudication step runs). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Write the judge provenance record from the stage's sidecars --step provenance builds docs/haiku55/judge_provenance.json from each re-opened case's sidecar: its hashes, its sidecar fields with addresses withheld, and its group by declared account (the pb-judge setup-token login, or subfleet lane claude-18's token for the cases judged after that login's weekly limit). It then runs verify_judge_provenance on the result, so export's gate holds by construction. Tests cover the grouping, the refusals and, locally, the live stage. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Pin the Haiku 5.5 driver to release 20261006 as shipped BASE_COMMIT is #202's merge on main (9ce4ade), BASE_SHA256 the shipped payload aa34e5c9..., and the exclusion pin the shipped record (92741dfd..., whose two scenario_081 sentences the release's review reworded). Reference explanations, values, scenarios and predictions are unchanged, so every staged prompt and verdict still binds. The design note's base table follows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Record the 2026-10-06 wave's decision date and the judge provenance The spec names 2026-10-09 as the day the ten ruled outputs' decisions were written (adjudicate-exclusions). docs/haiku55/judge_provenance.json lists the 231 new Opus 5.5 verdicts: 210 on the pb-judge setup-token login and 21 on subfleet lane claude-18's token after that login's weekly limit. Each was judged isolated with a token login, and every transcript passes the runner's checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Add the reference builder for a second engine upgrade (2.15.17 to a newer policyengine-us) Generalized from reference_audit/2026-09-28/scripts/build_references_latest.py (#182). Reads release 20261006 from git, recomputes all 1,984 outputs with the pinned pre-freeze conventions, and gates every move against an actions file: approved scored moves, regenerated engine-defect exclusions, new exclusions, and rechecked excluded outputs that keep their decided values (rule 5). Narratives regenerate the explanation of each changed output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Let the paper's reference facts follow any number of engine upgrades EngineUpgrade and its accessors key the paper's figures to the last upgrade, keep the September (2.15.17) facts pinned to that revision, and rebuild the exclusion record and references as they stood on any date. A mock second upgrade (tests/second_engine_upgrade.py, labelled mock data) exercises them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Add --step install-references to the Haiku 5.5 driver Gates a reference build against release 20261006 before writing anything, renders the audit it gives in memory, installs it in place, sets aside only the verdicts whose prompts change, and holds every later step (exclusions, adjudications, triage, export) to the installed build. The freeze refuses an upgraded stage until it learns to carry one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Aim a drifted regeneration at its fix modules' pre-fix value, not a stale record A 20261006 engine-defect record decided on 1.755.4 keeps that engine's corrected value, which other engine changes have since moved (scenario_005 federal: record 107,198.34; the fix module on 2.35.3 gives 108,525.58). A regeneration now names its audited target: the record's alternative_value (kind record), or its fix modules' corrected value on an older engine that still has the defect, from a committed evidence file (kind fix_modules). The builder applies the modules again on the new engine and refuses if they still move the output; the driver re-derives the target from the evidence file and the modules committed at BASE_COMMIT. Evidence mode writes that file. Also: copy r19_irs_sales_tax_2025.json beside the conventions it serves, and write the judge provenance record with indent=1 as docs/gpt61sol's is. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Record the fix modules' pre-fix evidence on policyengine-us 2.37.2 24 engine-defect outputs, each on 2.37.2 with the pinned conventions alone and with its audited fix modules after them. Every module still moves its output beyond $1 there, so each defect is present on 2.37.2; the corrected values are the targets a later engine's regeneration must land on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Add a pre-merge check of a policyengine-us checkout against the fix evidence Runs the builder's regeneration gate early: each evidence item must land within $1 of its corrected value, and its fix modules must leave it unchanged, on the policyengine-us that PYTHONPATH puts first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Decide the outputs an engine upgrade newly excludes docs/haiku55/spec.json's upgrade_adjudications holds one item per output the installed build newly excludes (the Indiana county cells), in the ruled adjudications' shape. adjudicate-exclusions appends each one's entry, built from its item, its build record and its case's bound verdict and dated by the record; the date conventions then name the upgrade's wave, with the day the spec says its decisions were written. Triage's gate holds the staged record to exactly those entries, so every excluded output stays decided. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Point the paper's September upgrade facts at that upgrade Renders byte for byte as before. The upgrade facts move to r.september_upgrade and r.excluded_outputs_by_engine_version_phrase, so a second engine upgrade can add its own without changing them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Name the upstream fixes behind WI 064 and VA 039 state, by bisecting their merges Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Rule the engine upgrade into the release spec Max, 2026-10-09: wait for the fixed engine and regenerate the cells it fixes. The spec now names d1022's four defect cells as regenerated by the upgrade, decides the two Indiana county outputs the upgrade newly excludes (upgrade_adjudications), and dates those decisions. The Idaho approval's basis cites policyengine-us #9810 and Idaho Code 63-3082. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Read a regeneration's removed record under the builder's key The builder and policybench.paper_results name it "record"; the driver read "removed_record", which only its mock build wrote, so a real build failed install-references. A local test now runs the 2.37.2 rehearsal build through load_build. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Reuse an earlier build's narrative when its writer inputs are unchanged A rebuild on a newer engine rewrites every changed output's narrative, and judges' prompts render it, so each rewrite costs a verdict. --reuse-from keeps the earlier narrative when the value, cause, grounding (its engine's name aside), PolicyEngine variable and trace are byte-identical and the narrative does not name the earlier engine; the reused rows are listed beside the output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Name the build's own engine in the drafted Indiana county record Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Decide triage flags from the spec; scope-export the upgrade without the old annotations On the upgraded references the judge flagged WI 064 state, regenerated by #9801: it applied a $500 Wisconsin capital loss limit. The 2025 Wisconsin Schedule WD limits a net capital loss to $3,000 ($1,500 married filing separately) or less, as policyengine-us does, so the reference stands. The spec's triage_adjudications now carries such decisions (affirmed references only, on re-opened cases, dated by a wave the release names, read only with an upgrade installed); adjudicate-exclusions appends each from its case's bound verdict after dropping the regenerated records' decisions, and the gate holds it exact. A dropped case's new entry is new, not a restatement. The upgrade's scope export puts release 20261006's references back but keeps the stage's annotations, which follow the new references, so it now checks the dashboard schema without requiring an annotation on every wrong answer; it is read only for modelStats. The judge provenance record describes the working stage that carries the upgrade. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Freeze a stage that carries an engine upgrade of the references freeze_haiku55.py now freezes an upgraded stage as strictly as a plain one: it re-gates the installed build and the recorded drops, requires the export receipt to bind every retained build file, holds the staged, committed and manifest reference pins to the build (scenarios stay release 20261006's), takes the build's explanations, exclusion record and counts (69 records, 1,915 scored outputs on the 2.37.2 rehearsal), names exactly the manifest fields the last upgrade may change, counts exclusions by their recorded engines in the version text, and checks the adjudication record with the upgrade's new-exclusion and triage decisions. freeze_snapshot.py follows the sidecar's last engine_upgrade revision, not the first. A stage without an upgrade freezes byte for byte as before. Built on a Subfleet build lane (GPT-6.1 Sol); its final regression run was stopped under a disk alert and re-run here. The scope export keeps calling release_20261006.export_payload, now with require_failure_annotations=False. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Answer the machinery review: pin fix-module siblings, bind approvals and tolerances, correct eight narratives From the GPT-6.1 Sol review of the upgrade machinery (findings 3-8; 1 and 2 were already fixed in the scope export and triage commits): - A fix module that loads a sibling (r01, r02 and r03's v2 modules) now pins it: the builder and the evidence record each sibling's bytes as committed at the base, and the driver re-derives them. The 2.37.2 evidence is re-run with them; its values are unchanged. - Non-finite engine, module, evidence and action values are refused. - The installer requires an approval for every scored move beyond $1, and holds each regeneration to its own tolerance (at most $1). - Narrative reuse checks the earlier narratives file is that build's (each changed row once, for the US, no error, at the build's value), and keeps a narrative naming the engine only when the engine is the same. - A narrative states its amount only as a dollar figure equal to the cent; a bare "2026" no longer passes for $0. - Eight narratives that misstated a figure or the mechanism behind the change are hand-corrected from the engine's own values (hand_corrected_narratives.json, with the reasons in its .md). Also re-pins the spec to #200's merged proposed_changes.json (each record now names decision d1022; CO 043's reading is reworded), names d994 on the two Louisiana records, and drops an unverified Idaho statute citation from the drafted approval basis. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Record WI 064's answered flag in the design note, not as a decision The case's re-judged verdict, after its narrative was corrected, raises no flag, so the release records no triage decision; the design note keeps the earlier verdict's $500 capital-loss hypothesis and why the reference stands. The judge provenance record follows the re-judged stage. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Stage Claude Haiku 5.5's tool_choice: auto sensitivity run The board row forces the answer tool, which leaves Claude's extended thinking off (the onboarding probe's forced-tool response carries no thinking block); the 2026-10-08 re-run with tool_choice: auto declares the tool and leaves it to the model. Its predictions (results/local/haiku55-stage/auto-run, model renamed claude-haiku-5.5-thinking) are committed as a deterministic gzip, and the by-variable and rescore scripts take it. Its scores, would-rank and by-variable asset are written by the rescore after the 47-model board is frozen. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Draft the Haiku 5.5 release note and paper changes on the 2.37.2 rehearsal Both are drafts: every bracketed value is the rehearsal's and becomes a fact a test recomputes from the frozen release; the note and paper proper replace them once the release on the fixed policyengine-us is frozen. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Add the paper's accessors for the 2026-10-06 rulings and the second upgrade's counts ruled_records gathers the records the 2026-10-06 rulings decided, from the exclusion record and from a later upgrade's regenerated_exclusions, and refuses one that names no ruling; the counts split them by ruling and by whether the upgrade regenerated them. Word forms of the last upgrade's counts use digits above ten. Tested on the real 2.37.2 rehearsal build (local) and the mock second upgrade, whose 2026-10-06 record now names its ruling. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Name each regeneration's upstream fix from a committed map upstreams.json maps the 20261006 engine-defect root causes to the 2026-10-09 fix pull requests, with the bisected merges (#9801 for WI 064, #9633 for VA 039 state) as per-output entries; actions_from_cells.py --upstreams applies it, so the build on the final engine needs no hand edits of the draft. On the 2.37.2 rehearsal it names the same fixes the reviewed actions did. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * PRACTICE FREEZE on the policyengine-us 2.37.2 rehearsal (replaced at T) The 47-model snapshot frozen from results/local/rehearsal-2372/stage, so the paper, the note, the sensitivity rescore and the older releases' test pins can be brought up to a frozen release before the fixed policyengine-us ships. The freeze on that release replaces every file here; the squash merge keeps only it. Not for publication. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Address the delta review's findings 1-5 1. The driver binds each regenerated output to its one reviewed action: the revision must restate the action's tolerance, upstream, basis and target (kind, fix modules, evidence), the action's alternative_value must be the record's, and the action's tolerance is the one enforced, for landing and for module inertness. 2. The freeze verifies every fix module the final engine_upgrade pins: its path is the module's, its sha256 the bytes committed there, no module twice, and a later upgrade keeps the earlier upgrade's pins and conventions. 3. policybench/fix_module_closure.py derives a fix module's local dependency closure from its syntax tree, shared by the builder and the driver. Path(__file__).with_name("<literal>") is the one supported loader; a module reaching local files any other way (the c13v3_* wrappers' _HERE / f"{name}.py") is refused rather than pinned short. On every committed module it agrees with literal discovery or refuses. 4. Narrative reuse requires finite source values; "nan" no longer passes. 5. NY 082's hand narrative: the child credit phases out at federal AGI ($117,652.65); the care credit's percentage is about 29.354%. Also ruff format/check across the release branch, and drop a duplicated paper_results accessor. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Rescore the sensitivity runs on the 47-model board and add Haiku 5.5's - sensitivity_by_variable.py and rescore_sensitivity_summaries.py on the frozen board; Claude Haiku 5.5's tool_choice auto run scores 90.386 (would rank #11) against its 80.950 board row (#37). - claude-thinking-2026-08.md: every score, rank and per-program table re-read from the frozen board; a Claude Haiku 5.5 section. The scores on all 1,984 outputs move with the references, and a new test recomputes them. - The app's sensitivity chips take the rescored values and a Haiku 5.5 entry; the app tests' release pins move to this board, and the engine sentences' tests handle any number of engines. - A test holds ENGINE_UPGRADE_RECHECK to the frozen sidecar. - The freeze's pin check reads committed bytes from the script's own repository, so a scratch ROOT in tests still sees them; the freeze tests read the version list as release 20261006 left it. Values are the 2.37.2 rehearsal's; they are re-derived at the final freeze. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Carry the release chain's records and tests to release 20261009 - pyproject/uv.lock pin policyengine-us to the reference engine (2.37.2 in the rehearsal) and policyengine-core to the builder's 3.32.29. - scripts/date_haiku55_judge_verdicts.py: writes the verdicts this release's restated decisions name (docs/haiku55/judge_verdicts_20261009.json) and brings verification/judge_verdicts.json up to it, as release 20261006's driver did: the 2026-10-05 wave's release is now #202's commit, and this release's 2026-10-06 and 2026-10-09 waves join uncommitted. - tests/test_adjudications.py: earlier releases' checks read their records and evidence at their commits; the counts move to this record; a new test checks this release's restatements, drops and additions against 20261006's record. - The September 22 audit's tests know a later engine upgrade can regenerate a record; the 2026-09-29 installer and driver tests read release 20260929's commit; a re-judged case supersedes the 2026-10-05 wording rewrites its new verdict writes, without the corrected error returning. - reference_audit/2026-10-09-engine-upgrade/scripts/sweep_timing.py: the 2026-10-09 move's timing record and publication check, refusing an engine that was not the newest release when the sweep began. Counts are the 2.37.2 rehearsal's; they are re-derived at the final freeze. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Move the paper, notes and cost tests to release 20261009 Release 20261006's differential tests run on its committed files (_release_20261006_full); notes on it recompute from its commit; the frozen release's board snapshot is its manifest's; the pins move to the 47-model board (rehearsal values, re-derived at the final freeze). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Hold the snapshot tests to the real builder's setup (delta review finding 6) The freeze mocks record the setup build_references_upgrade.py writes on a later upgrade (inherited modules with paths, latest_final.py and the sales-tax table, its builder note), and a test checks it against the real 2.37.2 rehearsal build. The manifest test accepts that setup, with the stated-hours alias from the earlier upgrade and the inherited conventions. The snapshot audit counts move to the rehearsal's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Write the release's paper text, card, guide and note - paper/index.qmd: the October 2026 upgrade row and paragraph name the restored outputs' fixes, the changed reference and the new exclusions from the sidecar (new paper_results accessors, which refuse an unnamed root cause); a paragraph on the 2026-10-06 rulings (the reference adversary's eight records and Louisiana's two); the abstract and the audit classes gain the third exclusion class, law published after the freeze; the sentence on mechanically derived exclusions excepts the 2026-10-06 records; the judges sentence adds the October 9 addition. Rendered. - docs/benchmark_card.md and docs/paper.md: the engine, counts and the two October paragraphs. - app/src/notes: the Claude Haiku 5.5 release note, every figure a fact that tests/test_notes.py recomputes from the frozen release. - tests: the card's engine test reads each move's timing record; the paper's September timing test names the September engine. Rehearsal values (policyengine-us 2.37.2); the timing sentences and final numbers are written at the final freeze. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * PRACTICE re-pin of the rendered paper on the 2.37.2 rehearsal (replaced at T) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * State the October sweep's timing in the paper from its timing record engine_upgrade_timing_sentence reads the 2026-10-09 audit's verification/sweep_timing.json: the sweep's day, the engine's PyPI upload time and what the publication check found. It is empty until the record is written, and refuses a check that moved outputs or a record of another engine. MOCK-record tests cover each case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Count scored and excluded outputs apart in the publication check A newer policyengine-us release that fixes a defect behind an excluded output moves that output's computed value without touching a scored reference. The check records both counts; the paper's sentence states the scored outputs are unchanged and how many excluded ones move, and still refuses a release that moves a scored output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Derive the upgrade's day and release-named files in tests The final build's sweep falls on a later UTC day than the rehearsal's, so the tests read the engine upgrade's day from the sidecar, find the release note and the verdict evidence by name pattern, and the evidence file takes its name from the driver's release tag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Count engine-defect records whose defect is now fixed apart A record the upgrade rechecked whose new-engine value is its corrected value has a fixed defect and stays excluded for the unstated input its note names. The abstract and the limitations no longer call those defects unfixed: they count with the unstated inputs, and the audit paragraph and the card say how many there are. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Expect the action-binding refusal where it now fires first Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Say the earlier models' rates fall, and how restored targets are known The note's last paragraph claimed the new references raise every earlier model's rate. On the release with the IRA fix they fall: the restored outputs release 20261006 had excluded are ones few models get right. The paragraph now states the fall, its cause and the counterfactual, with facts recomputed from the frozen predictions, and its test fails until the frozen release bears the claim out. The paper now says how many restored outputs are held to the corrected value their record carries and how many to the value the audit's fix modules give on a later engine that still has the defect. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Drop the superseded note and paper drafts The release note and the paper carry the text now; the drafts held rehearsal numbers. Two comments named the engine move by a day the release may not bear. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Bring the design note up to the engine upgrade The note predated the ruling to wait for the fixed engine. It now describes the upgrade: which engine, the build and its actions, the two kinds of regeneration target, the install, and what later steps and the freeze hold an installed build to, with the tests that pin each gate. The open items are resolved and removed, and statements that tests and scripts were planned or uncommitted are gone. Three comments it listed as stale are corrected in the driver, the restate script and the driver's tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Say which restored outputs return to scoring and which stay scored The note and the card counted every regenerated record as an output PolicyBench had stopped scoring. Four of them, the October 6 engine defects, were still scored in release 20261006; the new engine fixes them, so they stay scored instead of being excluded. The note now splits the restored outputs into the ones release 20261006 excluded and those four, and its test checks the split against release 20261006's record. The note test's per-output match count compared the payload's exact mark with 1.0, but the payload writes a hit as 100.0, so it counted no match. It now compares with 100.0 and checks the scale. The fixes behind the restored outputs are also available as a phrase (engine_upgrade_restored_fix_phrase), which the paper's sentence uses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Hold the note's facts at the pre-T rehearsal's expected values Every fact is rewritten from the frozen release at T; until then they are the expected values from the rehearsal on the code T will carry, so the draft reads consistently. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Tie the publication check to the release's build (pre-T review 1, 6) sweep_timing.py check now takes the reference build directory instead of two free paths. It refuses a build whose sidecar does not name the sweep's engine, pin its CSV and name the committed builder; whose reference CSV and exclusion record are not the committed snapshot's; or whose computed.csv does not give every scored reference. The record pins each input by sha256, and check exits non-zero, after writing it, when the newest release moves a scored output. Both steps refuse a sweep whose output does not follow its engine (wheel upload, then install, then output), so an older file cannot backdate the sweep. The record lists newer releases whose wheels are all yanked. The timing sentence dates the engine's upload when it was not on the sweep's day, and the card test checks its timing record last, so a missing record no longer hides the September and exclusion-group checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Match the builder's target rule; separate unfixed defects (pre-T review 4, 5, 11) engine_upgrade_restored_targets now judges a fix-module target the way the builder's regeneration_target does: the modules must move the output beyond the exact-match tolerance on their engine, so a flag moved from 0 to 1 counts, and a regenerated flag must equal its target. The refusal and the 'because' clause the builder has no counterpart for are gone. The paper's September audit paragraph no longer says every engine-defect record stays excluded until a fixed engine replaces it: it states how many the reference engine still has the defect behind, and that an output also resting on an unstated input stays excluded. The October table row names how many restored outputs were decided on 2026-10-06. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Run the note's claim checks on recomputed facts (pre-T review 2) The facts-equality check failed on the practice snapshot and hid every check after it. It is now its own test, with the direction of the last paragraph. The claim checks run on the facts recomputed from the frozen release, and new ones cover the sentences nothing checked: the dependent's ,000 of wages and own return, Louisiana's price-index computation and September 28 bulletin, the unstated Indiana county, Idaho's permanent building fund tax, the serving claims (from the model card) and the unchanged conventions. 'The cause is' now rests on the incumbents' answers and the household weights being release 20261006's. The note's gain is the difference of the rates it prints. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Test the card's hand-written counts against the frozen records (pre-T review 3) The card's October paragraph (what the move restores, from where, the fixes behind them, the scored change and the new exclusions) and its exclusion breakdown (excluded outputs and households, engine-defect records, the unfixed ones and their root causes, the fixed-but-kept sentence, unstated inputs and later law) are now recomputed by a test. The root causes the card names as not fixed must be ones the reference engine still has. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Hold the judge evidence to the decisions it names (pre-T review 7) scripts/date_haiku55_judge_verdicts.py now refuses, before writing, a decision whose judge, classes, day or reference flag depart from the verdict it names (a flag set from an earlier run needs its judge_reference_suspect_source), and a set of new waves other than the 2026-10-06 rulings' and the engine upgrade's day. The adjudication test checks the flag of each restated decision and reads the 'current' evidence of every new-wave decision. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Refuse more ways a fix module can reach local files (pre-T review 8) The closure now also refuses directory listings (glob, rglob, iterdir, listdir, scandir, walk), the module's own location reached another way (__spec__, __loader__, an attribute .__file__, inspect, globals(), vars()) and imports of this repository's own package. Of the committed fix modules this refuses only r14_unlisted_weekly_hours_v2_v2.py, which imports policybench.scenarios and is no engine-upgrade target; no module the upgrade's evidence applies is refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Pin regeneration targets at the freeze; test the mock against every real build (pre-T review 9, 10) The freeze now checks that each regenerated output held to the audit's fix modules publishes the committed bytes of every module and sibling its target pins, and its committed evidence file. The mock-builder test compares the mock setup with the committed snapshot's engine upgrade from 2.15.17, so it runs everywhere once a freeze commits one, and with every local build. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Name later law among the sensitivity doc's exclusions; test its gain claim (pre-T review 11) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Run the publication check's sweep itself; require builder and sidecar (pre-T re-review 1-3) A file's birth time cannot show which engine computed it, so check no longer takes a check sweep's output: it runs the sweep, a first pass of this audit's builder in the venv holding PyPI's newest release, which the builder refuses unless the venv holds that release, and records what ran. The build's sidecar must name the committed builder and be the frozen sidecar byte for byte, and every computed and reference value must be finite before any comparison. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Close the re-review's nits (pre-T re-review 4-7) The fix-module closure also refuses a path built from anything but __file__ (Path(stem + '.py'), the working directory, the home directory) and opening a file whose name is not a literal; of the committed modules this refuses only r18_hold_all_projections.py, which no evidence applies, and the docstring now says what a static read cannot see. The freeze's regeneration-target check no longer lets an absent file and an absent pin compare equal. The note test checks the unforced probe its comparison rests on, and the card-count test refuses a fixed-but-kept sentence when there are no such records. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Check the imported engine is the wheel; keep the serving probes (pre-T review round 3) A version label can sit beside older code: an editable install whose checkout moved, or a source tree ahead of the wheel on the import path. Both timing steps now run a check under the venv's interpreter, in the sweep's environment: the version, that policyengine_us is imported from the installed package, that it is not editable, that every package file matches the wheel's RECORD sha256, and that no unrecorded module sits beside them. On the 2.37.2 venv it verifies 18,280 files in 3 s. The venv path is resolved once, so a relative --check-venv means the same venv for discovery, the check and the sweep, and the scratch directory is created if absent. The two serving probes behind the note's thinking claims are committed as a minimal fixture (docs/haiku55/serving_probes.json: request settings, response block types, usage), and the note test reads it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Revert "PRACTICE re-pin of the rendered paper on the 2.37.2 rehearsal (replaced at T)" This reverts commit 6a758b8. * Revert "PRACTICE FREEZE on the policyengine-us 2.37.2 rehearsal (replaced at T)" This reverts commit 41ba29f. * Roll the release's day to 2026-10-10 The release is built on the policyengine-us release that carries #10032, which reaches PyPI on 2026-10-10 UTC: the tag, snapshot date, upgrade wave day, note, judge-evidence file and the card and test names follow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Check the engine in the process that computes (pre-T review round 4) The engine check and the builder now run in one process under the venv's interpreter (ENGINE_RUNNER via run_on_engine), launched with -P and a fresh, empty bytecode-cache prefix: no script or working directory joins the import path, and no __pycache__ beside the sources is read, so every module compiles from the verified source. The check also refuses unrecorded importable files (sourceless bytecode, native extensions) and accepts sha384 and sha512 RECORD hashes. sweep_timing.py run runs the reference build the same way. Regressions run the real runner on MOCK installs: stale unchecked-hash bytecode that a plain interpreter executes, a package beside the builder script, an unrecorded native extension, each permitted hash algorithm, and a missing scratch parent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Refuse symlinks in the engine; take only fresh receipts and outputs (pre-T review round 5) The engine check refuses any symlink in the installed package: a linked directory hid its files from the scan and could shadow a recorded module, so installs must be copies (uv's default link modes). sweep_timing.py run refuses an existing receipt or an out-dir already holding computed.csv, and refuses after the run unless the builder wrote a new one; path arguments are resolved in the parent, which a new test showed mattered: the child runs in the repository root, so a relative --out-dir named another directory there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Read the builder's arguments as the builder does (pre-T review round 6) sweep_timing.py run now parses the builder's arguments with the builder's own flags (equals forms, abbreviations, last occurrence), refuses unknown ones, resolves every path and passes them on in one canonical form. The file it holds to freshness is the one the builder writes: --computed-csv when given, else <out-dir>/computed.csv. A test keeps the replica's flags equal to the builder's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Correct the reference narratives of all 24 outputs the 2.38.6 move changes The narrative writer's drafts were checked against each output's trace by an independent reviewer; these texts state the final engine's value and figures, and the notes give what each draft got wrong. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Record the judges of the cases the 2.38.6 references re-opened Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Keep CA 099's state income tax excluded: California disallows educator expenses The release's judge flagged the regenerated reference: California does not conform to the federal educator expense deduction (FTB Instructions for Schedule CA (540), Section C, line 11), and policyengine-us 2.38.6 still allows it in California AGI, so the reference is about $26.90 low. The output keeps its record (the build's actions list it under excluded_outputs_rechecked), and its corrected narrative comes out: the release installs 23 corrected narratives, not 24. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Record the judge of CA 099's re-opened case Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Pin policyengine-us 2.38.6, the references' engine Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Freeze release dashboard-data-20261010: Claude Haiku 5.5 on policyengine-us 2.38.6 references Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Date the release's judge verdicts; rescore the sensitivity runs on the 2.38.6 board Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Count fixed engine defects by the audit's evidence; separate a further defect An engine-defect record whose output later engine changes moved is judged fixed against the corrected value the audit's committed fix-module evidence gives, not only the record's own value: CA 005 and CT 120 federal income tax land on theirs, so the IRA phase-out they rest on is no longer counted unfixed. A record whose recorded defect is fixed but whose reference rests on a further defect the engine has (CA 099's California income tax) is its own kind: counted with the defects upstream has not fixed in the abstract and limitations, and said so in the paper. A landed record that names neither reason is refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * State the 2.38.6 release in the card, guide, app and note The card's engine, October move and exclusion breakdown, the paper guide's counts, the app's engine constant and the release note's facts follow the frozen release; the note's match count is the lower median (half of the newly scored outputs are answered within $1 by at most one model). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Render the paper on the 2.38.6 references and pin it Re-pins the rendered PDF and web files in the manifest and paperSnapshot, syncs the app's serving config with the frozen copy, and commits the sweep timing and publication-check record the card and paper cite. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Pin the tests to the 2.38.6 board and drop a stale judge wave The judge-verdict dating script built on the working tree's evidence, so a rehearsal pass's 2026-10-09 wave (release dashboard-data-20261009) survived into the T evidence beside the real 2026-10-10 wave. The script now keeps release 20261006's waves and rewrites this release's on every pass, with a regression test, and the evidence is regenerated from the T stage (only that wave's six lines change; the restatements file is byte-identical). Test pins move from the rehearsal to T: 70 adjudications (14 regenerated records' decisions dropped), 58 excluded outputs in 42 households, 1,926 scored per model, 659 parse failures (644 less four on the new exclusions plus 19 on the 14 restored outputs), 718 contract violations, 8,001/7,997/ 2,138 audit counts, GPT-6 Sol's 95.6% headline, and Fable 5.1's program rates (federal 67/84 after seven restored outputs and two removed). The restored-target mock tests now count the sidecar's own two fix-module targets beside the mocked ones, and the judge-day check for the 2026-10-06 wave is back to the 2026-10-08 and 2026-10-09 runs it reviewed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Answer the final review: CI pins, the 14-of-18 split, and the actions file Final review (20261010-032143-pb-haiku-final-review) and the PR's first CI run: - Blocker: test_reference_exclusions and test_snapshot_artifacts still held rehearsal counts. They now pin T's: 58 exclusions (42 unlisted input, 14 engine defect, 2 later law; 38/18/2 by engine), 8,001 annotated rows, 7,997 misses, 10,139 below full bounded score, 7,342 llm_error and 659 parse failures. CI also failed test_freeze_haiku55's rehearsal snapshot dates (now the spec's 2026-10-10) and its other-tag test, which the date roll had left naming the driver's own tag (it now derives the day after). - sweep_timing.py read st_birthtime directly, which Linux's stat lacks, so every timing test crashed on CI. birth_time() now refuses on such a platform rather than date the record by a modification time, and the tests use a labeled MOCK modification-time source there only (they create each dated file once). Checked under a simulated Linux stat locally. - Should-fix: the paper said the move returns all 18 fixed outputs to scoring. A new accessor, engine_upgrade_restored_split_sentence (a pure restored_split_sentence with a Hypothesis test), says 14 that earlier releases excluded return and the four ruled on 2026-10-06 while still scored stay scored; the table cell says the same. The 2026-10-06 paragraphs in the paper and card now say PolicyBench ruled to exclude ten. Paper re-rendered and re-pinned. - Nit: the build's final_actions.json is now committed byte for byte (the sidecar's actions_sha256 ef23b50e...), as the 2026-09-29 upgrade's is. Its note predates CA 099's move to the rechecks and says 19 and 16; the new README's erratum says so, and a test holds the arrays at 18 and 17 against the sidecar. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 12, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the serving card for Claude Haiku 5.5 (
claude-haiku-5-5, released 2026-10-07), so it can run on the canonical board. Its scores join the published board in the next data release, so this PR changes no published artifact.What the onboarding found (2026-10-08)
claude-haiku-5-5, created_at 2026-10-07T18:00Z, max_tokens 128,000, 1M input. Model overview: "Released October 7, 2026"; adaptive thinking on by default (effort medium).THINKING_DEFAULT_CLAUDE_MODELS.tool_choice: {type: tool}. The forced-tool probe returned no thinking block. Under the benchmark card's rule (a forced answer tool unless the provider rejects it), the card selects the tool contract. The row therefore answers without extended thinking, as Claude Opus 5's does;_tool_choice_fordocuments the mechanism.policybench onboard): tool-3var PASS 3/3 (2s), tool-full PASS 16/16 (4s); estimate $0.001 per scenario.Changes
config.py:MODELS["claude-haiku-5.5"], and a price override of $0.10/$0.50 per 1M, with $0.125 cache writes and $0.01 cache reads (the pricing page's rates for prompts up to 100k tokens, read 2026-10-08).eval_no_tools.py: local litellm registration (the bundled backup map lacks the id) and the thinking-default class.model_cards.py: the card (tool contract, no chunking, 16,384 budget, class timeout 300s, 128k provider cap).paper_results.py/app/src/modelMeta.ts: release date 2026-10-07.MODEL_DISPLAY_NAMESand the app roster in the release that folds the row into the board:test_paper_resultspins display names to the frozen 46-model roster, as it did for Sonnet 5.5.Also: a Sonnet 5.5 cache-read price fix (unblocks CI)
test_local_claude_models_resolve_without_remote_cost_map[claude-sonnet-5-5]fails on current main whenever litellm loads its remote cost map. That map now prices Sonnet 5.5 cache reads at $0.10 per 1M, and the pricing page's footnote 2 agrees (0.05x of $2). The repo override said $0.20. This PR corrects it to $0.10. Sonnet 5.5's board run billed 0 cached and 0 cache-write tokens (results/local/adds202609/sonnet55/run/predictions.csv), so no recorded or published cost changes.Tests
test_claude_haiku_5_5_is_a_public_default,test_claude_haiku_5_5_override_reconstructs_token_costs(4 cases),test_claude_haiku_5_5_reasons_by_default_but_not_under_the_forced_tool,test_claude_haiku_5_5_release_date_is_recorded, and theEXPECTEDrow intest_model_cards. Haiku is also added to the local-registration parametrization.tests/test_model_cards.py,tests/test_paper_results.pyandtests/test_eval_no_tools.pynow pass: 255 passed.🤖 Generated with Claude Code