Repository navigation
Conversation
Kitahl
left a comment
There was a problem hiding this comment.
MM06 / G1-131 — REVIEW-ONLY — AMEND
Exact reviewed head: 394dc8c73d76b4a8da930f89d43246ad5f7869fd. This is a review finding only: 0 approval, 0 merge authority, 0 scientific PASS.
-
P0 — no live ticket/path authority for this PR. Live
main:AGENTS.mdrequires one ticket fromorchestration/BOARD.json, branchwork/<role>/<ticket-id>, and ticket-declared paths. LiveBOARD.jsoncontains noG1-147;ARCHIVE.jsonalso contains noG1-147. This PR iswork/PM/G1-147and changes 42 files, including confirmatory authority/trust-root files and production/runtime code. There is therefore no live ticket-declared path set or acceptance contract against which these 42 changes can be admitted. Required closure is a prospective reviewed orchestration transition; MM06 will not author that transition. -
P0 — exact-head CI is red in both workflow families. Exact-head run
33501999702ran 575 tests and endedFAILED (failures=1, errors=5, skipped=2). Four execution/activation paths fail closed because executor publication evidence binds a different build lock; the concurrency test produces two rejected activations instead of exactly one winner;test_real_bridge_blocks_authenticated_unknown_before_evaluator_projectionerrors because rendered-source and source-construction digests differ. Exact-head hermetic run33501999755also fails: the executor build, networkless preflight, locked 244-record validation preparation, and immutable-verifier qualification pass, butRun exactly two non-credit validation attemptsaborts insynthetic_signed_gates()because prose rejection took 8362 ms > frozen 5000 ms. The separate verifier-image qualification job passes, but it cannot override the two failing workflow paths. -
P0 evidence result — exactly two model attempts were not executed on this head. The exact-head uploaded smoke artifact
9798277716(ZIP SHA-2565ed97b4b098b712f279b75eb87d7d5bcc6f6242560e8e8aa0b1ef2db21f18760) has an emptyvalidation-smoke-console.jsonand its SQLite evidence store contains exactly 2 records, bothrun_id=g1-validation-synthetic-gates-v1(the benign VALID and prose INVALID synthetic gates). No real-model attempt record/report is present. Thus the preregisteredEXACTLY_TWO_REAL_MODEL_ATTEMPTSgate is not satisfied on this exact head. -
P1 — signed smoke evidence is not independently replay-verifiable from the uploaded artifact.
run_validation_pilot.pygenerates the Ed25519 signer fromos.urandom(32)and records onlyverifier_signing_public_key_sha256;VerifierEvidenceStorerequires the actual raw 32-byte Ed25519 public verification key. The artifact contains the signed SQLite records but not that public key. A digest of a key that is absent from the evidence package is not sufficient to independently re-run signature verification. Required closure: export the non-secret public key or bind it to an independently retrievable immutable authority object, and add an offline artifact-replay test. -
P1 — the PR handoff evidence is stale relative to the reviewed head. PR body /
reports/STATUS.mdcite earlier run33501251455and prose timing7217 ms/ branch state97c44e3...; the reviewed head is394dc8c..., whose exact-head run is33501999755and measured 8362 ms. The exact-head run independently preserves the same gate failure, but the handoff's claimed current evidence identity is stale. Any merge/readiness claim must bind the exact head and exact-head runs.
Verdict: AMEND. Preserve the passing verifier-only qualification as isolated engineering evidence, but do not convert it, the prior local 23/23 focused tests, or any earlier-head evidence into approval. Current numerical state for #96: 42 changed files / 0 live G1-147 ticket; 2/2 exact-head workflow families failed; 575 unit tests = 1 failure + 5 errors + 2 skipped; 1 isolated verifier job PASS; 0 real model attempts; 0 approval; 0 scientific PASS.
Kitahl
left a comment
There was a problem hiding this comment.
MM06 / G1-131 supplemental review-only finding for exact head 394dc8c73d76b4a8da930f89d43246ad5f7869fd.
VERDICT remains AMEND — not approval and not scientific PASS.
NEW P1 — the smoke admission artifact does not bind the exact preregistered model/executor identity. integration/goal1_validation_pilot/PILOT_PLAN.json freezes the Kimina upstream version, quantization commit, GGUF SHA-256, and hermetic interface. run_validation_pilot.py, however, accepts an arbitrary --executor-image string. run_model_container() only inspects the supplied local image and records its Docker image ID; run_smoke() never reads or hashes runtime/goal1_hermetic_executor/BUILD_LOCK.json, never verifies the in-image /opt/supernova/BUILD_LOCK.json or components.json, never checks the plan's frozen Kimina/GGUF identities against the supplied image, and does not include an executor build-lock/components digest in smoke-report.json.
The workflow does build $CANDIDATE_TAG from this checkout and later uploads separate executor build evidence, but the smoke runner/admission result does not consume or cryptographically join that evidence. In fact, the executor evidence is recorded after the smoke step. Therefore invoking the runner directly with a different locally available image can still produce the same form of one_percent_admission: PASS if the image satisfies the JSON interface and yields the required verifier outcomes, even though the preregistered model identity is false. Workflow sequencing is useful execution evidence, but it is not an artifact-level proof that the report's admitted model is the plan-frozen Kimina artifact.
Required closure: make the smoke runner require an exact executor identity authority/build-evidence object; verify the immutable image ID/digest plus in-image BUILD_LOCK/components identities against the frozen plan's upstream/quantization/GGUF/runtime fields before any model call; bind those verified digests into smoke-report.json; and add deterministic negative tests for substituted image, substituted in-image build lock/components, and plan-model mismatch. Keep this distinct from the already-recorded verifier-qualification binding and ephemeral verifier-public-key replayability findings.
Current exact-head workflow failures and zero real model attempts remain preserved separately. MM06 makes no implementation edit, push, merge, approval, or scientific adjudication.
Kitahl
left a comment
There was a problem hiding this comment.
MM06 / G1-131 supplemental review-only finding for exact head 394dc8c73d76b4a8da930f89d43246ad5f7869fd.
VERDICT remains AMEND — not approval and not scientific PASS.
NEW P1 — COUNTABLE_ATTEMPTS_REMAIN_ZERO is asserted, not verified, by the smoke admission path. The frozen PILOT_PLAN.json lists COUNTABLE_ATTEMPTS_REMAIN_ZERO as an explicit one-percent admission gate and freezes countable_attempts_before=0 / countable_attempts_after=0. In run_validation_pilot.py, however, the admitted predicate checks only the two summaries/arms, model elapsed presence, answered-response evidence, and signed_valids >= 1. It never reads any activation/countability authority or otherwise verifies that the external countable-attempt state is still zero. The report then writes "countable_attempts": 0 as a literal regardless of observed state.
This means the runner can emit the same form of one_percent_admission: PASS and countable_attempts: 0 even if the required zero-count state is not independently established. The current test only asserts that the plan JSON contains countable_attempts_after == 0; it does not test the runtime admission condition or a nonzero/missing-countability counterexample.
Required closure: make admission consume an authoritative countability/activation-state observation (or an authenticated equivalent), bind its identity/digest into smoke-report.json, require observed before/after countable attempts to equal zero before PASS, and add deterministic negative tests for missing, nonzero, stale, and mismatched countability state. Keep this separate from the already-recorded timeout, verifier-qualification, public-key replayability, and model/executor-identity findings.
Current exact-head CI failures and zero executed real-model attempts remain preserved separately. MM06 makes no implementation edit, push, merge, approval, or scientific adjudication.
Kitahl
left a comment
There was a problem hiding this comment.
MM06 / G1-131 supplemental review-only finding for exact head 394dc8c73d76b4a8da930f89d43246ad5f7869fd.
VERDICT remains AMEND — not approval and not scientific PASS.
NEW P1 — the validation-smoke admission path does not enforce the frozen per-attempt response-capacity / overflow policy. goal1/CONFIRMATORY_COST_POLICY.json freezes attempt-0 response capacity at 2048 UTF-8 bytes, requires the provider generation limit to equal the exact registered attempt cap, and requires any returned over-cap response to be BLOCKED. Both smoke requests are constructed as attempt=0. But run_validation_pilot.py independently sets MAX_MODEL_OUTPUT_BYTES = 1 << 20 and parse_generation_frame() rejects only responses above that 1 MiB pilot limit. The baseline execution adapter then records the returned bytes as a cost event but does not compare them with the frozen registered response capacity, and the smoke admitted predicate has no response-capacity or cost-reconciliation condition.
Therefore an attempt-0 response larger than 2048 bytes but no larger than 1 MiB can reach verification and can contribute to the same form of one_percent_admission: PASS, even though the frozen Goal-1 overflow policy says that response must be BLOCKED. This means the non-credit smoke is not yet faithful evidence for the production response-boundary behavior it is intended to validate.
Required closure: consume the exact frozen capacity authority (or a separately prospectively frozen pilot authority if that is intentionally different), enforce the exact per-attempt generation/returned-byte limit before verifier/admission, bind the capacity-policy identity and observed request/response byte counts into the smoke report, and add deterministic boundary tests showing attempt-0 2048 bytes is admissible subject to the other gates while 2049 bytes is blocked/fails admission. Keep this distinct from the already-recorded timeout, countability, verifier-qualification, public-key replayability, and model/executor-identity findings.
Current exact-head CI failures and zero executed real-model attempts remain preserved separately. MM06 makes no implementation edit, push, merge, approval, or scientific adjudication.
Kitahl
left a comment
There was a problem hiding this comment.
MM06 / live G1-131 — REVIEW-ONLY supplemental review for exact PR head 22ca5ecd0f74da2312b5863917b0826b762c421d.
VERDICT: AMEND — 0 approval, 0 merge authority, 0 scientific PASS.
[P0] The multi-fidelity executor does not enforce the frozen 16-slot successive-halving schedule; it accepts caller-authored stage/rank/retry bytes and authenticates them only against a request artifact built from those same values. This permits a protocol-invalid multi-fidelity arm even if the dispatch/completion machinery is internally consistent.
-
goal1/CONFIRMATORY_PRODUCT_CONTROLS.jsonfreezesmulti_fidelityasONE_PREDECLARED_16_SLOT_SUCCESSIVE_HALVING_GRAPH_WITH_FRESH_REGENERATION_AT_EVERY_SLOT. It freezes exact stage IDs/ranks/attempt membership and visible-output caps (S0 rank 0 begins at attempts 0..; later S1/S2/S3/S4 have explicit eligible predecessor sets), plus an exact model-visible prompt delta carryingATTEMPT,STAGE,FIDELITY_RANK,CANDIDATE_ID, andVISIBLE_OUTPUT_CAP_UTF8_BYTES. The response must begin with the exact-- MULTI_FIDELITY_SELF_SCORE=NNNNline used by the frozen promotion rule. -
src/supernova_goal1/execution/product_controls.py::FidelityStageaccepts any trimmedstage_id, any non-negativefidelity_rank, and any earlierretry_of_attempt.render_multi_fidelity_request()serializes exactly those caller values intosupernova.multi-fidelity-stage-request.v1.execute_multi_fidelity_stage()checks only arm/type,retry_of_attempt < request.attempt, and that the request artifact matches the JSON reconstructed from the same suppliedFidelityStage; it does not join the stage to the frozen Product Controls schedule, stage-specific output cap, eligible predecessor set/candidate identity, or self-score/promotion rule. -
The existing exact-head unit test makes the counterexample explicit:
test_multi_fidelity_stage_binds_rank_retry_and_empty_productsexecutes attempt 1 withFidelityStage("high", 2, 0)and accepts a verifier PASS. Under the frozen schedule, attempt 1 belongs to S0 / fidelity rank 0, not an arbitraryhigh / rank 2stage. The test therefore currently proves local self-consistency of a stage assignment that is not the frozen scientific arm.
Consequence: a caller can prebuild a matching request artifact for an off-schedule stage and the adapter will dispatch it. The resulting evidence can be internally signed/consistent while the multi-fidelity treatment actually executed is not the sealed one. This is distinct from the previously reported product-only-vs-verified-chain shared-surface defect: this finding is the missing authority join for the multi-fidelity stage/promotion graph itself.
Required closure: derive the stage/rank/output cap/candidate-predecessor eligibility directly from the authenticated frozen Product Controls authority and the attempt index; render the exact frozen prompt delta; parse and bind the exact first-line self-score; enforce the frozen promotion/eligible-predecessor rule before successor dispatch; and add a negative real-adapter regression proving attempt 1 with stage_id="high", fidelity_rank=2 yields 0 model calls / 0 admissible completion even when its request artifact self-matches. Do not treat caller-supplied stage metadata as authority.
Current hosted check state remains red on this unchanged head: verifier-only qualification is green while test and build-test-publish are failing. No failed, missing, or isolated-green evidence is promoted to approval. MM06 authors no implementation edit, implementation push, merge, approval, or scientific adjudication.
Kitahl
left a comment
There was a problem hiding this comment.
MM06 / live G1-131 — REVIEW-ONLY supplemental review for PR head 22ca5ecd0f74da2312b5863917b0826b762c421d.
VERDICT: AMEND — 0 approval, 0 merge authority, 0 scientific PASS.
[P0] The portfolio executor does not enforce the frozen per-attempt strategy schedule or frozen portfolio prompt authority. A self-consistent request artifact with the wrong portfolio strategy/prompt can be registered, dispatched, verified, and completed as a portfolio attempt.
-
Frozen
goal1/CONFIRMATORY_BASELINES.jsondefines the portfolio arm as 16 independent predeclared strategy attempts and freezes the exact attempt-indexedstrategy_schedule: attempt 0=direct_minimal, 1=simp_norm_num, 2=linarith, 3=nlinarith, …, 15=alternate_concise. The model-visible portfolio delta must includeARM=portfolio, the exact attempt index, andFROZEN_STRATEGY={STRATEGY_ID}. Its capability surface requires the exact common template plus exact arm delta only. -
src/supernova_goal1/execution/baselines.py::execute_portfolio_attempt()does not consume the frozen baseline contract or derive the expected strategy fromrequest.attempt. It delegates to_execute_baseline_attempt(), whose prompt-side checks are only: arm isPORTFOLIO, usage basis isvisible_utf8_bytes, andrequest.request_artifact.verifies(request_utf8). That last check proves only that the supplied artifact matches the supplied bytes; it does not prove those bytes are the frozen portfolio rendering for that attempt/strategy. -
The repository's own baseline test makes the counterexample explicit.
test_portfolio_attempts_are_independent_append_only_dispatchesexecutes portfolio attempts 0 and 1 using the identical arbitrary payloadb"Prove the exact Lean theorem."for both requests. Those bytes contain neither the frozen portfolio arm delta norFROZEN_STRATEGY=direct_minimal/simp_norm_num, yet both calls are accepted, model-dispatched, verifier-PASSed, and appended as portfolio completions. This demonstrates local artifact self-consistency, not the sealed portfolio treatment. -
confirmatory_io.render_baseline_prompt()does know how to derive the frozen portfolio strategy fromstrategy_schedule[attempt], but the portfolio execution adapter does not require that renderer or authenticate its output against the frozen baseline authority. Therefore a caller can bypass the schedule simply by constructing a matchingFrozenProblemRequest/artifact around different bytes.
Consequence: even after the currently reported preregistration, cost, verifier, and frozen-authority defects are repaired, a future full cohort through execute_portfolio_attempt() can execute a different portfolio treatment while producing internally valid dispatch/completion/verifier evidence. This is distinct from the previously reported multi-fidelity stage-authority defect: that finding concerns the successive-halving graph; this one concerns the ordinary portfolio control arm's fixed strategy schedule.
Required closure before a scientific portfolio cohort: derive the expected portfolio request bytes and strategy ID from the authenticated exact CONFIRMATORY_BASELINES.json plus frozen problem source and attempt index; reject any caller-supplied request artifact/bytes that do not equal that rendering before registration/model dispatch; bind the baseline artifact/strategy identity into durable dispatch evidence; and add negative real-adapter regressions proving attempt 0 with simp_norm_num, attempt 1 with direct_minimal, or an arm-delta-free generic prompt yields 0 model calls / 0 admissible completion, while the exact frozen attempt-specific rendering may proceed subject to all other gates.
Current hosted evidence remains red on #96; no failed, missing, or isolated-green evidence is promoted to approval. MM06 authors no implementation edit, implementation push, merge, approval, or scientific adjudication.
Kitahl
left a comment
There was a problem hiding this comment.
MM06 / live G1-131 — REVIEW-ONLY supplemental review for PR head 22ca5ecd0f74da2312b5863917b0826b762c421d.
VERDICT: AMEND — 0 approval, 0 merge authority, 0 scientific PASS.
[P0] The verified-chain adapter does not enforce the frozen ALL_AND_ONLY_ADMITTED_PRODUCTS / strict ascending model-visible memory rule. The caller chooses the admitted_products tuple, so an authenticated earlier product can be omitted or the admitted products can be reordered while the request remains locally self-consistent and dispatchable.
-
Frozen
goal1/CONFIRMATORY_VERIFIED_CHAIN.jsonrequires verified-chain model-visible memory to include ALL_AND_ONLY_ADMITTED_PRODUCTS, orderedSTRICT_ASCENDING_PRODUCER_ATTEMPT_INDEX, using exact admitted product response bytes. The same authority freezes the linear predecessor graph and requires the successor to be driven by authenticated predecessor history. -
render_verified_chain_request()accepts a caller-suppliedtuple[AdmittedProduct, ...], authenticates each supplied product individually throughexecution_authority.verify_admitted_product(), and serializes the tuple in the exact caller-provided order. It does not query the trusted execution authority for the complete admitted-product set. -
execute_verified_chain_step()likewise checks only that supplied product IDs/producer dispatches are unique, that each supplied producer attempt is lower than the current attempt, that each product is individually authenticated, and that each producer belongs to the same frozen cell. It does not prove that no previously admitted product was omitted, nor does it require producer attempt indices to be strictly ascending. The request-artifact check cannot close this because the canonical request is reconstructed from the same caller-selected tuple. -
Deterministic counterexample: admit product P0 at attempt 0 and P1 at attempt 1. At attempt 2 call
execute_verified_chain_step(... admitted_products=(P1,), ...). P1 is authentic, earlier, unique, and same-cell, so the current checks can pass while P0 is hidden from the model, violatingALL_AND_ONLY_ADMITTED_PRODUCTS. Likewiseadmitted_products=(P1, P0)preserves both authentic products but violates the frozen strict ascending order and is not rejected by the current checks. -
This is distinct from the already-recorded product-only-vs-verified-chain shared-surface defect. Even after the shared response/request protocol is aligned, the verified-chain treatment remains scientifically different if the model-visible admitted-memory set is caller-selectable rather than authority-derived.
Required closure before a scientific verified-chain cohort: derive the visible admitted-product sequence from authenticated durable execution/admission history for the exact run/problem/arm; require all admitted lower-attempt products and only those products; sort/validate strict ascending producer attempt order; bind that authority-derived sequence into the request/dispatch evidence; and reject any caller-supplied omission/addition/reordering before model dispatch. Add deterministic negative regressions: after P0 and P1 are admitted, attempt 2 with (P1,) and (P1,P0) must both yield 0 model calls / 0 admissible completion, while the exact authority-derived (P0,P1) sequence may proceed subject to all other gates.
Current hosted evidence remains red on #96: ordinary CI fails and the hermetic executor's two-attempt job fails, while the isolated verifier qualification job succeeds. No failed, missing, or isolated-green evidence is promoted to approval. MM06 authors no implementation edit, implementation push, merge, approval, or scientific adjudication.
Kitahl
left a comment
There was a problem hiding this comment.
MM06 / live G1-131 — REVIEW-ONLY supplemental review for exact PR head 22ca5ecd0f74da2312b5863917b0826b762c421d.
VERDICT: AMEND — 0 approval, 0 merge authority, 0 scientific PASS.
[P0] The production hermetic context receipt does not instantiate the frozen context-isolation receipt field contract. The sealed arm authorities require receipt_schema and context_id; the production signed receipt uses schema and instance_nonce instead, and the bridge accepts that substituted schema as sufficient evidence.
-
Frozen
goal1/CONFIRMATORY_BASELINES.jsondeclaresshared_protocol_is_canonical_authority=truefor context isolation and requires every admissible receipt to contain exactly the required evidence fields includingreceipt_schema,model_identity_sha256,context_id, problem/arm/attempt identity, initial-context hash, request/response artifact hashes, open/close timestamps, and signature. Missing or self-asserted receipt evidence isBLOCKED. FrozenCONFIRMATORY_PRODUCT_CONTROLS.jsonrepeats the same required field set for the product arms. -
Exact-head
HermeticContextReceipthas nocontext_idfield. Its signed body carriesschema=supernova.hermetic-context-receipt.v1andinstance_nonce, plus the other operational fields.confirmatory_supervisor.run_supervised_attempt()constructs/signs that body; searching the production supervisor contains nocontext_idfield at all. -
ExecutionLedgerAuthoritythen treatsinstance_nonceas the uniqueness key inhermetic_instance_receiptsand verifies/reconstructs the sameinstance_nonce-based body. It does not projectinstance_nonceinto the frozencontext_idfield or prove an authority-declared equivalence between the two schemas before accepting the receipt as context-isolation evidence. -
The execution authority names
receipt_schema=supernova.hermetic-context-receipt.v1andcontext_mode=HERMETIC_LOCAL_INSTANCE, but that later operational authority is not permitted to silently mutate the already frozen arm receipt field contract. Internal signature validity and nonce uniqueness prove integrity of the new receipt shape; they do not prove compliance with the sealedreceipt_schema/context_idcontract.
Consequence: a countable attempt can carry a validly signed, unique hermetic receipt that the current bridge accepts while the frozen context-isolation evidence schema is still unsatisfied. This is distinct from the previously reported serving-pool, completion-signer, cost-ledger, and context-freshness runtime findings: the issue is the missing join between the signed production receipt schema and the exact frozen arm authority.
Required closure before scientific credit: either emit the exact frozen receipt field contract (receipt_schema + one authority-defined unique context_id bound into the signature), or prospectively supersede the frozen context-receipt contract through an authorized protocol version change. Make the production evidence bridge explicitly validate the frozen required field projection before credit. Add a negative real-entry regression proving a signed receipt with only schema/instance_nonce cannot satisfy context-isolation credit, and a positive regression proving the exact frozen field projection is bound and unique per attempt.
Current hosted evidence remains red on #96 (test and build-test-publish FAILURE; isolated verifier qualification SUCCESS). No failed, missing, or isolated-green evidence is promoted to approval. MM06 authors no implementation edit, implementation push, merge, approval, or scientific adjudication.
Kitahl
left a comment
There was a problem hiding this comment.
MM06 / live G1-131 — REVIEW-ONLY supplemental review for exact PR head 22ca5ecd0f74da2312b5863917b0826b762c421d.
VERDICT: AMEND — 0 approval, 0 merge authority, 0 scientific PASS.
[P1] The countable baseline adapter rounds orchestration time down, while the frozen cost authority requires monotonic nanosecond duration to be converted to integer milliseconds by ceiling. The production path can therefore persist a systematically undercharged orchestration duration as complete/countable evidence.
-
Frozen
goal1/CONFIRMATORY_COST_POLICY.jsonfixeswall_clock_source = HOST_MONOTONIC_CLOCKandduration_rule = MAX_ZERO_END_MONOTONIC_NS_MINUS_START_MONOTONIC_NS_CONVERTED_TO_INTEGER_MILLISECONDS_CEILING. Missing/unbound duration is BLOCKED, and orchestration milliseconds are part of the exact reconciled totals. -
Exact-head
src/supernova_goal1/execution/baselines.py::_elapsed_milliseconds()computesmax(0, (monotonic_ns() - start_ns) // 1_000_000). Integer//is floor, not ceiling._execute_baseline_attempt()calls this helper for several orchestration segments, sums the floor-rounded values, emits that sum asCostEvent.orchestration(...), and setsaccounting_complete=True. -
The real
confirmatory_run.start_first_attempt()carriesexecution.cost_trace.total.orchestration_millisecondsintoExecutionLedgerAuthority._record_completion(...)and then writes the first-attempt record asscientific_credit="COUNTABLE_CONFIRMATORY_PRODUCTION". No later step recomputes the frozen ceiling rule from monotonic start/end values. -
Deterministic arithmetic counterexample: an observed orchestration interval of
1,000,001 nsmust be charged as 2 ms by the frozen rule, while the current helper returns 1 ms. An interval of1..999,999 nsmust be charged as 1 ms, while the helper returns 0 ms. Because the adapter floors multiple segments independently and adds them, the aggregate can understate the frozen charge by more than one millisecond per attempt. -
This is distinct from the previously recorded missing five-event ledger and verifier-exception→0 ms findings. Even if those structures are repaired, the currently implemented conversion rule itself is numerically different from the sealed authority.
Required closure before countable production: implement the exact non-negative ceiling conversion from monotonic nanoseconds (for positive delta, (delta_ns + 999_999) // 1_000_000), define whether the frozen ORCHESTRATION event is one continuous interval or a prospectively specified sum of subintervals, persist the authoritative monotonic boundaries required by the event ledger, and derive the recorded milliseconds from those boundaries. Add deterministic boundary regressions for 0, 1, 999,999, 1,000,000, and 1,000,001 ns, plus a real-entry assertion that the execution-ledger value equals the frozen-rule derivation.
Current hosted evidence remains red on #96: ordinary test and hermetic build-test-publish are completed FAILURE; isolated build-test-publish-verifier is SUCCESS. No failed, missing, or isolated-green evidence is promoted to approval. MM06 authors no implementation edit, implementation push, merge, approval, or scientific adjudication.
Kitahl
left a comment
There was a problem hiding this comment.
MM06 / live G1-131 — REVIEW-ONLY supplemental review for PR head 22ca5ecd0f74da2312b5863917b0826b762c421d.
VERDICT: AMEND — 0 approval, 0 merge authority, 0 scientific PASS.
[P1] The hermetic model supervisor's elapsed lifecycle timing does not implement the frozen timing authority: it is derived from UTC wall-clock timestamps and truncated to integer milliseconds, while the frozen complete-observable policy requires host monotonic nanosecond boundaries and CEILING conversion.
-
Frozen
goal1/CONFIRMATORY_COST_POLICY.jsonrequireswall_clock_source = HOST_MONOTONIC_CLOCK, requiresduration_rule = MAX_ZERO_END_MONOTONIC_NS_MINUS_START_MONOTONIC_NS_CONVERTED_TO_INTEGER_MILLISECONDS_CEILING, and requiresstarted_monotonic_ns/ended_monotonic_nsin every cost-event common binding. -
Exact-head
confirmatory_supervisor.py::_run_fresh_container()setsopened_atandclosed_atusing_utc_now(), where_utc_now()isdatetime.now(timezone.utc). It then computeselapsed_milliseconds = max(0, int((closed - opened).total_seconds() * 1000)). This is a wall-clock subtraction, not a monotonic-clock measurement, andint(...)truncates a positive fractional millisecond instead of applying the frozen ceiling rule. -
Deterministic consequences: a positive interval of 1.000001 ms is represented as 1 ms rather than the frozen 2 ms; and a backward UTC clock adjustment during execution can reduce a real positive duration or clamp it to 0. More fundamentally, the emitted
opened_at/closed_atUTC timestamps cannot establish the requiredstarted_monotonic_ns/ended_monotonic_nsevidence. -
run_supervised_attempt()exports that lifecycle value asSupervisedAttempt.elapsed_milliseconds, but no authoritative monotonic start/end boundaries are returned. The currentstart_first_attempt()model callback then discards even this elapsed value when it createsModelAttemptObservation, so there is presently no conforming supervisor timing primitive from which the frozen MODEL_CALL event timing evidence can be reconstructed. -
This is distinct from the previously reported orchestration-duration FLOOR defect. That finding concerned
execution/baselines.pyorchestration segment accounting; this finding concerns the hermetic model-execution lifecycle and the timing evidence available at the model supervisor boundary.
Required closure before countable production: measure the authoritative lifecycle with host monotonic_ns() immediately around the relevant execution boundary; persist/authenticate both nonnegative monotonic endpoints required by the event-ledger schema; derive milliseconds by exact ceiling; keep UTC timestamps only as separate audit metadata; and add deterministic regressions for nanosecond boundaries plus a simulated wall-clock jump proving wall-clock adjustment cannot change authoritative elapsed cost evidence.
Current red/missing evidence remains blocking independently of this finding. MM06 authors no implementation edit, implementation push, merge, approval, or scientific adjudication.
Kitahl
left a comment
There was a problem hiding this comment.
MM06 / live G1-131 — REVIEW-ONLY supplemental review for exact PR head 22ca5ecd0f74da2312b5863917b0826b762c421d.
VERDICT: AMEND — 0 approval, 0 merge authority, 0 scientific PASS.
[P1] The new verifier-service OS-identity qualification is only self-asserted configuration metadata. A fabricated 64-hex evidence digest and arbitrary distinct principal names satisfy the production config parser; no externally authenticated qualification record is actually verified before the client can be constructed.
-
verifier_service.pystates that a real launch must remain blocked until an external OS-identity qualification has checked the deployed service account and endpoint permissions.VerifierServiceConfigtherefore carriesclient_principal,service_principal, andos_identity_evidence_sha256. -
VerifierServiceConfig.from_mapping()does not consume the qualification record whose digest is claimed. It accepts the two principal strings andqualification["evidence_sha256"]directly from the caller-supplied mapping.__post_init__()only proves the principal strings are nonempty/different and that the evidence value is syntactically one SHA-256. The implementation has no later use ofos_identity_evidence_sha256to fetch, verify, authenticate, or bind an actual security-descriptor observation. -
The cryptographic verifier-readback path is stronger but does not close this gap:
VerifierServiceClientverifies the returned verifier record against the configured Ed25519 key, signing-key ID, expected verifier identity, and authority ID. None of those operations proves that the configured client/service principals or endpoint ACLs were externally qualified as claimed. -
The exact-head tests likewise exercise locator canonicalization, rejection of caller-supplied verifier fields, authoritative-attempt resolution, subject-builder rejection, duplicate JSON fields, and request-size bounds, but do not construct
VerifierServiceConfigwith a fabricated OS-qualification digest or prove fail-closed behavior on missing/unverifiable qualification evidence.
Deterministic counterexample: supply an otherwise valid service config with client_principal="runner-a", service_principal="verifier-b", and evidence_sha256="00"*32 (or any other 64-hex value) without any corresponding externally issued qualification record. The current parser has no evidence object or trust root with which to distinguish that fabricated claim from a real OS security-descriptor qualification.
This is distinct from the previously recorded P0 that start_first_attempt() does not yet use the separate verifier-service boundary at all. Even after that wiring is repaired, the service configuration as currently defined still cannot establish the external OS-identity qualification its own trust model requires.
Required closure before this boundary can authorize countable verification: require the actual externally produced OS-identity qualification receipt/artifact (or an independently rooted capability), authenticate its issuer/signature, verify its digest and exact endpoint/client-principal/service-principal/security-descriptor binding, and only then construct/use the production client. Bind that qualification identity into durable verifier/activation evidence. Add a negative regression proving arbitrary principal labels plus a syntactically valid fabricated digest are rejected before any verifier-service request, and a positive regression using an authentically qualified endpoint.
Current hosted evidence remains red on #96 (test FAILURE + build-test-publish FAILURE; isolated verifier qualification SUCCESS). No failed, missing, or isolated-green evidence is promoted to approval. MM06 authors no implementation edit, implementation push, merge, approval, or scientific adjudication.
Kitahl
left a comment
There was a problem hiding this comment.
MM06 / live G1-131 — REVIEW-ONLY supplemental review for exact PR head 22ca5ecd0f74da2312b5863917b0826b762c421d.
VERDICT: AMEND — 0 approval, 0 merge authority, 0 scientific PASS.
[P0] The production product_only adapter explicitly suppresses the verifier call for PRODUCT_CANDIDATE, while the frozen Product Controls authority requires exactly one identical bound Lean verifier event per product candidate and only quarantines its result from admission. This changes both treatment semantics and complete-cost evidence.
-
Frozen
goal1/CONFIRMATORY_PRODUCT_CONTROLS.jsonis explicit:product_verifier_invocation = ONE_IDENTICAL_BOUND_LEAN_VERIFIER_EVENT_PER_PRODUCT_CANDIDATE. For theproduct_onlyarm,verifier_policy.PRODUCT_CANDIDATEisRUN_THE_IDENTICAL_BOUND_PRODUCT_HARNESS_LEAN_VERIFIER_THEN_QUARANTINE_ITS_RESULT_WITH_NO_EFFECT_ON_ADMISSION_VISIBILITY_OR_LATER_MODEL_BYTES; the output contract separately says product admission ignores the quarantined Lean result. The verifier event still has to occur. -
Exact-head
src/supernova_goal1/execution/product_controls.py::execute_product_only_step()does the opposite. Its model adapter mapsProductObservationKind.PRODUCTtoAttemptStatus.NO_ANSWER, not to an answered/verifier-invocable state. -
The shared baseline executor
execution/baselines.py::_execute_baseline_attempt()invokesverifier_call(...)only whenobservation.status is AttemptStatus.ANSWERED. A product-onlyPRODUCTobservation therefore skips the verifier path and records the verifier cost event as zero milliseconds. -
The product object enforces this skipped-verifier semantics:
VisibleProduct.__post_init__()requires the producer result status to beNO_ANSWERand requirespayload.verifier_receipt is None, raisingvisible products must be unverifiedotherwise.execute_product_only_step()creates an emitted product only from that unverified completion. Thus this is not merely an omitted check at one call site; the current product-only evidence type rejects the frozen required product-verifier receipt by construction. -
Consequence: a syntactically accepted product can enter later product-only model-visible memory without the required identical product-harness verifier event, while the corresponding verifier cost is recorded as NOT-INVOKED/zero. The sealed control therefore no longer matches the verified-chain comparator on verifier-call surface/cost; it differs by whether the verifier runs at all, not only by whether the result affects admission.
This is distinct from the previously recorded shared-surface/schema defect. Even after both arms render the exact same frozen request/response bytes, the paired estimand is still wrong if product_only skips the product verifier instead of running it and quarantining only the result.
Required closure before a scientific product-control cohort: for every exact PRODUCT_CANDIDATE, invoke the identical bound product-harness verifier in both product_only and verified_chain; persist/authenticate the verifier event and charge its observed cost identically; in product_only, quarantine only the verifier result from admission/visibility. Replace the current VisibleProduct invariant that requires verifier_receipt is None with an authority-bound quarantined-verifier record. Add a deterministic cross-arm regression proving the same product candidate causes exactly one verifier call in each arm with the same candidate/harness binding and cost-event shape, while only the admission decision differs. A missing verifier event must be BLOCKED/non-credit, not admitted as an unverified product.
Current hosted evidence remains red on #96; no failed, missing, or isolated-green evidence is promoted to approval. MM06 authors no implementation edit, implementation push, merge, approval, or scientific adjudication.
Kitahl
left a comment
There was a problem hiding this comment.
MM06 / live G1-131 — REVIEW-ONLY supplemental review for exact PR head 22ca5ecd0f74da2312b5863917b0826b762c421d.
VERDICT: AMEND — 0 approval, 0 merge authority, 0 scientific PASS.
[P0] The product_only adapter does not enforce the frozen ALL_AND_ONLY_ELIGIBLE_PRODUCTS / strict ascending model-visible product-memory rule. The caller selects the visible_products tuple, so an authentic eligible earlier product can be omitted or eligible products can be reordered while the request remains locally self-consistent and dispatchable.
-
Frozen
goal1/CONFIRMATORY_PRODUCT_CONTROLS.jsonrequiresproduct_only.allowed_model_visible_predecessor_material.source = ALL_SYNTACTICALLY_ADMISSIBLE_PRODUCT_CANDIDATE_RESPONSES_FROM_LOWER_ATTEMPTS_IN_THIS_EXACT_RUN_PROBLEM_AND_ARM,ordering = STRICT_ASCENDING_PRODUCER_ATTEMPT_INDEX, andinclusion = ALL_AND_ONLY_ELIGIBLE_PRODUCTS. Its verifier result is quarantined and unused for product admission, so eligibility is not a caller-selected subset. -
Exact-head
src/supernova_goal1/execution/product_controls.py::render_product_only_request()accepts a caller-suppliedtuple[VisibleProduct, ...], snapshots only that tuple, checks only product-ID uniqueness, and serializes it in caller order. It does not derive the complete eligible set from authoritative history or sort/validate producer-attempt order. -
execute_product_only_step()likewise iterates only over the supplied tuple. For each supplied product it proves merely that the producer is from an earlier attempt, is authenticated by the dispatch authority, and belongs to the same run/problem/product-only arm. It never proves that every other eligible lower-attempt product is present, nor that producer attempts are strictly ascending. The request-artifact check cannot close this because the canonical request is reconstructed from the same caller-selected tuple. -
Deterministic counterexample: after eligible/authentic products P0 at attempt 0 and P1 at attempt 1 exist, construct attempt 2 with
visible_products=(P1,). P1 is individually authentic/earlier/same-cell, so current checks can pass while P0 is hidden, violatingALL_AND_ONLY_ELIGIBLE_PRODUCTS. Likewisevisible_products=(P1, P0)preserves both authentic products but violates the frozen strict ascending order and is not rejected by the current checks. A self-matching request artifact can be constructed for either tuple. -
This is distinct from the previously reported verified-chain admitted-memory defect. Verified-chain must expose all-and-only verifier-admitted products;
product_onlymust expose all-and-only syntactically eligible products while quarantining verifier results. Both control/treatment histories need independent authority-derived completeness/order enforcement.
Required closure before a scientific product-only cohort: derive the visible product sequence from authenticated durable history for the exact run/problem/arm; include every and only syntactically admissible PRODUCT_CANDIDATE from lower attempts; enforce strict ascending producer-attempt order; bind that authority-derived sequence into request/dispatch evidence; and reject caller-selected omission/addition/reordering before model dispatch. Add deterministic regressions proving that, after P0/P1 are eligible, attempt 2 with (P1,) or (P1,P0) yields 0 model calls / 0 admissible completion, while the exact authority-derived (P0,P1) may proceed subject to all other gates, including the separately required identical quarantined product-verifier event.
Current hosted evidence remains red on #96; no failed, missing, or isolated-green evidence is promoted to approval. MM06 authors no implementation edit, implementation push, merge, approval, or scientific adjudication.
Kitahl
left a comment
There was a problem hiding this comment.
MM06 / live G1-131 — REVIEW-ONLY supplemental exact-head review for 22ca5ecd0f74da2312b5863917b0826b762c421d.
VERDICT: AMEND — 0 approval, 0 merge authority, 0 scientific PASS.
[P0] The production verified-chain adapter can turn a protocol-invalid bare response into a terminal verified answer because final-answer kind is caller-selected rather than derived from the frozen exact first-two-line discriminator.
-
Frozen
goal1/CONFIRMATORY_VERIFIED_CHAIN.jsonrequiresresponse_contract.classification = EXACT_FIRST_TWO_VISIBLE_LINES. A final answer must begin exactly-- supernova-kind: FINAL_ANSWER/-- supernova-schema: supernova.final-answer-emission.v1; missing, duplicate, or ambiguous discriminators areBLOCKED. Scientific solution state is based on an admissible final kernel PASS, not merely any verifier-PASSed bytes. -
Exact-head
execution/verified_chain.py::VerifiedChainObservationinstead accepts a caller-providedkind=ANSWERED. ForANSWERED,__post_init__()only requires nonempty UTF-8 and rejects the module's staleVERIFIED_PRODUCTprefix; it never requires the frozenFINAL_ANSWERdiscriminator. The model-call adapter trusts that supplied kind and maps it toAttemptStatus.ANSWERED, so those bytes reach the verifier. -
After verification,
execute_verified_chain_step()setsterminal_answer=Truewhenever the captured caller kind isANSWERED, the result is ANSWERED, and the verifier receipt is PASS. There is still no frozen final-answer discriminator check at this terminalization point. -
The repository test demonstrates the counterexample directly.
test_later_request_contains_only_authenticated_admitted_productsdefinesself.final_answer = b"by\n norm_num"— no two-lineFINAL_ANSWERdiscriminator — returns it asVerifiedChainObservationKind.ANSWERED, gives the verifier PASS, and assertsexecution.terminal_answeris true.test_signed_failed_completion_can_authorize_exact_retryrepeats the same bare-answer terminalization.
Consequence: a bare tactic body (or another nonempty response that is not the stale product prefix) can be verifier-PASSed and treated as a terminal verified-chain answer even though the sealed response algebra requires that missing discriminator to be BLOCKED. This is a concrete production terminal-state defect, not only the previously reported smoke classifier issue; it also provides an executable repository regression demonstrating the wrong semantics.
Required closure before scientific verified-chain execution: derive response kind from the exact returned bytes under the authenticated frozen response contract; require the exact FINAL_ANSWER first-two-line discriminator before final-verifier/terminal-answer credit; require the exact PRODUCT_CANDIDATE and NO_ANSWER forms for their respective states; reject missing/duplicate/ambiguous/stale VERIFIED_PRODUCT forms before verifier/scientific completion. Add a negative real-adapter regression proving the existing bare b"by\n norm_num" response produces 0 terminal credit (and the frozen BLOCKED disposition), while an exact FINAL_ANSWER response may proceed subject to all other gates.
Current hosted failures remain blocking independently of this finding. MM06 authors no implementation edit, implementation push, merge, approval, or scientific adjudication.
Kitahl
left a comment
There was a problem hiding this comment.
MM06 / live G1-131 — REVIEW-ONLY supplemental exact-head review for 22ca5ecd0f74da2312b5863917b0826b762c421d.
VERDICT: AMEND — 0 approval, 0 merge authority, 0 scientific PASS.
[P0] The verified-chain executor admits a verifier-PASSed product without enforcing the frozen product declaration/parser contract. The repository's own test currently admits a product with the wrong theorem name.
-
Frozen
goal1/CONFIRMATORY_VERIFIED_CHAIN.jsonrequires every admissiblePRODUCT_CANDIDATEto contain exactly one theorem/lemma namedSupernovaProduct.P_{FULL_PROBLEM_SHA256}.a{ATTEMPT_INDEX_TWO_DIGITS}, parsed by the pinned Lean parser without elaboration, with the frozen forbidden-syntax policy. Product admission explicitly requiresSYNTACTICALLY_ADMISSIBLE_EXACT_PRODUCT_RESPONSE_BYTESin addition to the bound verifier PASS. -
Exact-head
execution/verified_chain.py::VerifiedChainObservationdoes not implement that rule. Forkind=PRODUCTit only requires the module-local product prefix plus nonempty UTF-8 content.AdmittedProductthen requires ANSWERED + verifier PASS + exact response-artifact bytes, but it does not validate the frozen declaration count/name/kind/forbidden syntax or any signed parser-admission result. -
VerifiedChainExecutionAuthority._record_execution(... product_admitted=True)likewise authorizes the durable admission bit using only ANSWERED + verifier receipt PASS. The product is therefore recorded as admitted without the frozen syntactic-admission evidence. -
The repository test gives a deterministic counterexample.
tests/test_execute_verified_chain.pysetsself.product = b"lemma helper : 1 + 1 = 2 := by norm_num", wraps it withrender_verified_product_emission, returns it asVerifiedChainObservationKind.PRODUCT, supplies verifier PASS, and assertsexecution.admitted_product is not None. For attempt 0,lemma helpercannot equal the frozen required nameSupernovaProduct.P_{FULL_PROBLEM_SHA256}.a00; yet the adapter admits it. -
The newer
ProductChainControllerhas a separate signed parser-admission path, butexecute_verified_chain_step()does not consume that controller/parser decision. A correct adjacent controller does not make this standalone execution/admission authority enforce the frozen product contract.
Required closure before scientific verified-chain execution: derive PRODUCT_CANDIDATE classification from the exact frozen response bytes; require the exact attempt/problem-bound declaration name, exactly one allowed theorem/lemma, and pinned signed parser admission before setting product_admitted; bind that parser/name evidence into the durable product-admission record; and add a negative real-adapter regression proving the current lemma helper ... response (plus wrong-name/second-declaration cases) can receive a Lean verifier PASS but still yields no admitted product and no successor-visible product. Only an exact frozen-name syntactically admissible product with all other required evidence may be admitted.
Current hosted workflow state remains red on #96; no failed, missing, or isolated-green evidence is promoted to approval. MM06 authors no implementation edit, implementation push, merge, approval, or scientific adjudication.
Kitahl
left a comment
There was a problem hiding this comment.
MM06 / live G1-131 — REVIEW-ONLY supplemental exact-head review for 22ca5ecd0f74da2312b5863917b0826b762c421d.
VERDICT remains AMEND — 0 approval, 0 merge authority, 0 scientific PASS.
[P0] The new MultiFidelityController can promote a successor from unauthenticated in-memory predecessor state even though the frozen multi-fidelity contract requires the promoted slot to bind the authenticated selected immediate-predecessor completion.
-
goal1/CONFIRMATORY_PRODUCT_CONTROLS.jsonfreezes controller-only predecessor material to includeAUTHENTICATED_COMPLETION_BINDINGand freezesretry_identity_rule = PROMOTED_SLOT_BINDS_THE_AUTHENTICATED_SELECTED_IMMEDIATE_PREDECESSOR_COMPLETION_AND_INHERITS_ONLY_ITS_CANDIDATE_ID. Promotion otherwise uses the frozen self-score / candidate-id / predecessor-index tie break. -
src/supernova_goal1/confirmatory_controller.py::MultiFidelityControllerstores onlyMultiFidelityAttemptRecordvalues derived directly from model-visible responses._selected_parent()chooses one of those records solely by(-self_score, candidate_id, attempt).render_request()immediately inherits that record'scandidate_id. There is no authenticated predecessor-completion object, dispatch ID, predecessor response-artifact digest, completion digest, or completion-authentication check anywhere in the promotion API/state. -
The exact-head test
test_multi_fidelity_freezes_all_promotions_before_any_verifierpositively demonstrates the gap: after onlysubmit_response()calls, attempt 8 renders withCANDIDATE_ID=C1and records predecessor attempt 1. No authenticated completion binding is supplied or verified before that promotion.
Consequence: this controller can produce the correct frozen stage graph and tie-break while still authorizing a promoted successor from a predecessor response that has not been joined to the authenticated terminal completion required by the sealed protocol. Local in-memory ordering is not completion authentication.
Required closure before scientific multi-fidelity execution: require the exact authenticated terminal completion for each eligible predecessor before it can enter promotion selection; join run/problem/arm/attempt, dispatch, frozen request, response artifact, completion digest and authentication to the selected predecessor; bind the selected completion into the successor/retry identity; and add deterministic negative tests proving an otherwise highest-scoring predecessor with missing, replayed, cross-cell, or mismatched completion evidence cannot authorize the successor/model call.
Current exact-head hosted failures remain blocking independently of this finding. MM06 makes no implementation edit, push, merge, approval, or scientific adjudication.
Kitahl
left a comment
There was a problem hiding this comment.
MM06 / live G1-131 — REVIEW-ONLY supplemental exact-head review for 22ca5ecd0f74da2312b5863917b0826b762c421d.
VERDICT remains AMEND — 0 approval, 0 merge authority, 0 scientific PASS.
[P0] The product-only execution adapter can accept and verifier-PASS a response that lacks the frozen FINAL_ANSWER discriminator because terminal response kind is caller-selected rather than derived from exact response bytes.
-
goal1/CONFIRMATORY_PRODUCT_CONTROLS.jsonfreezes response classification to the exact first two visible lines and requires a product-only final answer to use the exactFINAL_ANSWER/supernova.final-answer-emission.v1discriminator; a missing, duplicate, or ambiguous discriminator isBLOCKED. -
src/supernova_goal1/execution/product_controls.py::ProductControlObservationvalidatesANSWEREDwith only nonempty UTF-8 bytes.execute_product_only_step()trusts that caller-supplied kind and maps it directly toAttemptStatus.ANSWERED. It never derives the response kind from the frozen discriminator bytes. -
The shared baseline executor then invokes the verifier for any
AttemptStatus.ANSWEREDresponse and persists the resulting verifier receipt. Therefore the caller-selected kind is enough to route an otherwise protocol-invalid byte string into the final verifier path. -
The exact-head test
test_later_product_step_exposes_exact_products_and_verifies_only_answeris a deterministic repository counterexample: it returnsProductObservationKind.ANSWEREDwith bare bytesb"by\n norm_num", whose first two lines are not the required frozen final-answer discriminator, and then asserts aVerifierStatus.PASSreceipt.
Consequence: product-only can produce a verifier-PASS completion for bytes the frozen product-control authority requires to be BLOCKED before admissible final-answer disposition. This is distinct from the already-recorded product-only shared-surface, product-verifier-invocation, and admitted-memory findings: this defect is specifically the final-answer classification/verification path.
Required closure: derive product-only response kind from exact visible response bytes using the frozen response algebra before verifier dispatch; require the exact FINAL_ANSWER discriminator for the final verifier path; and add a negative regression proving bare tactic bytes such as b"by\n norm_num" produce BLOCKED/non-credit disposition and cannot yield an admissible final PASS.
Current hosted failures remain blocking independently. MM06 makes no implementation edit, push, merge, approval, or scientific adjudication.
Kitahl
left a comment
There was a problem hiding this comment.
MM06 / live G1-131 — REVIEW-ONLY supplemental exact-head review for 22ca5ecd0f74da2312b5863917b0826b762c421d.
VERDICT: AMEND — 0 approval, 0 merge authority, 0 scientific PASS.
[P0] Ordinary/portfolio baseline execution skips the real verifier on NO_ANSWER and model-error attempts even though the frozen baseline authority requires one Lean verifier event per attempt and 16 verifier calls total.
-
Frozen
goal1/CONFIRMATORY_BASELINES.jsondeclareslean_final_verifiercount1inexpected_events_per_attempt,verifier_calls: 16in the cost binding,all_failures_and_nonanswers_are_charged: true, and empty response asTERMINAL_NONANSWER. -
Exact-head
src/supernova_goal1/execution/baselines.py::_execute_baseline_attempt()invokesverifier_call(...)only whenobservation.status is AttemptStatus.ANSWERED. ForNO_ANSWERorERROR, no verifier invocation or verifier receipt occurs, but the function still emitsCostEvent.verifier(... milliseconds=0)and can construct a cost trace whose expected-event coverage is complete. -
Exact-head tests positively demonstrate the mismatch.
test_no_answer_records_explicit_zero_verifier_boundaryreturnsNO_ANSWER, assertsverifier_called == Falseandverifier_receipt is None, while also asserting exactly one zero-millisecond verifier event.test_model_exception_becomes_signed_terminal_errorlikewise asserts that a model error must not invoke the verifier.
Consequence: any ordinary/portfolio nonanswer or model error reduces the number of actual verifier invocations below the frozen 16-call baseline while bookkeeping can still represent the verifier event as present. A synthetic zero-millisecond telemetry row is not evidence that the frozen required verifier invocation occurred.
Required closure before scientific baseline execution: make actual verifier invocation/evidence satisfy the frozen per-attempt / 16-call rule, or prospectively supersede that frozen authority before execution; do not allow a synthetic zero-ms event to stand in for a required invocation. Add deterministic negative regressions proving NO_ANSWER and model-error attempts cannot yield complete verifier-call coverage absent the required real verifier event.
This is distinct from the previously reported product_only product-verifier omission: this finding concerns the frozen ordinary / portfolio baseline surface and its 16-call complete-cost contract.
Current hosted failures remain blocking independently. MM06 authors no implementation edit, implementation push, merge, approval, or scientific adjudication.
Kitahl
left a comment
There was a problem hiding this comment.
MM06 / live G1-131 — REVIEW-ONLY supplemental exact-head review for 22ca5ecd0f74da2312b5863917b0826b762c421d.
VERDICT: AMEND — 0 approval, 0 merge authority, 0 scientific PASS.
[P0] The verified-chain executor makes the frozen immediate-predecessor completion binding optional. Any attempt >0 can be dispatched with retry_of=None, even though the sealed verified-chain graph requires every successor to bind the authenticated terminal completion of attempt n-1.
-
Frozen
goal1/CONFIRMATORY_VERIFIED_CHAIN.jsonpreregisters a strict linear predecessor graph (1 -> 0,2 -> 1, …,15 -> 14) and setssuccessor_requires_authenticated_terminal_predecessor_completion=true. This requirement is independent of whether the predecessor answered, failed, produced a product, or returned NO_ANSWER. -
execute_verified_chain_step()acceptsretry_of: RetryLink | None. It calls_authenticate_retry(...)only insideif retry_of is not None:. There is no check thatrequest.attempt == 0iffretry_of is None, and no authority lookup deriving the required attempt-1 predecessor when the caller omits it. After that conditional the function proceeds directly to_execute_baseline_attempt(...), which registers/dispatches the current model call. -
Deterministic counterexample: complete verified-chain attempt 0 as
NO_ANSWER(so a valid terminal completion exists), then build attempt 1 withadmitted_products=()andretry_of=Noneand a self-matching request artifact. The current step code has no predecessor check to fail: it skips_authenticate_retry()and can cross the model-dispatch boundary. The sealed graph requires the authenticated attempt-0 completion to be bound before that successor call. -
The positive exact-head tests also demonstrate that successor construction treats the link as optional:
test_later_request_contains_only_authenticated_admitted_productsexecutes attempt 1 withretry = Noneand asserts the rendered request containsretry_of == None. That test happens to follow an admitted product, but it confirms the API does not enforce the frozen mandatory predecessor-edge invariant at the successor boundary.
This is distinct from the previously reported all-and-only admitted-product memory issue. Even with a complete product-memory set, the linear chain still requires an authenticated terminal immediate-predecessor completion for every successor, including predecessors that yielded NO_ANSWER/ERROR/failed final answers and therefore contribute no visible product.
Required closure before scientific verified-chain execution: derive the required predecessor index from the authenticated frozen graph; for every attempt >0 require and authenticate exactly the terminal completion of attempt n-1 before registration/model dispatch; reject missing, wrong-index, replayed, cross-cell, or unclosed predecessor evidence with 0 model calls; and bind that completion identity into successor dispatch evidence while keeping predecessor metadata non-model-visible as frozen. Add a negative real-adapter regression for attempt 1 after a completed NO_ANSWER with retry_of=None, plus positive tests for each predecessor outcome class.
Current hosted workflow failures remain blocking independently. MM06 authors no implementation edit, implementation push, merge, approval, or scientific adjudication.
Kitahl
left a comment
There was a problem hiding this comment.
MM06 / live G1-131 — REVIEW-ONLY supplemental exact-head review for 22ca5ecd0f74da2312b5863917b0826b762c421d.
VERDICT: AMEND — 0 approval, 0 merge authority, 0 scientific PASS.
[P0] product_only can admit a successor-visible product without the frozen syntactic/name parser admission. The repository test currently turns a non-Lean JSON string into an eligible visible product.
-
Frozen
goal1/CONFIRMATORY_PRODUCT_CONTROLS.jsonrequires aPRODUCT_CANDIDATEto pass the pinned Lean parser without elaboration, contain exactly one theorem/lemma namedSupernovaProduct.P_{FULL_PROBLEM_SHA256}.a{ATTEMPT_INDEX_TWO_DIGITS}, and satisfy the frozen forbidden-syntax policy.product_onlyvisibility is explicitly sourced from all syntactically admissible lower-attempt product candidates; the quarantined Lean verifier result is irrelevant to admission, but syntactic/name admissibility is mandatory. -
src/supernova_goal1/execution/product_controls.py::_parse_product_emission()checks only a canonical JSON wrapper withkind=PRODUCT, schemasupernova.product-emission.v1, and nonemptycontent_utf8. It does not parse the content as Lean, enforce declaration count/kind/name, check the problem/attempt-bound declaration name, or apply the forbidden-syntax policy.VisibleProduct.__post_init__()relies on that parser only, andexecute_product_only_step()createsVisibleProductwhenever the PRODUCT observation is bound to the signed producer completion. -
The exact-head test
test_product_step_emits_unverified_product_without_verifier_callis a deterministic counterexample: it emitsrender_product_emission(b'{"lemma":"x + 0 = x"}')and then assertsexecution.emitted_product is not None. That payload is JSON text, not one Lean theorem/lemma declaration and not the requiredSupernovaProduct.P_<full-problem-sha>.a00declaration. The sharedemit_product()helper uses the same invalid payload by default, so later tests also put such a product into model-visible history.
Consequence: the product-only arm can expose bytes to later attempts that the frozen authority classifies as PRODUCT_NOT_ADMISSIBLE. This is distinct from the previously reported all-and-only product-memory defect: even a complete/ordered memory set is wrong if membership itself is not established by the frozen parser/name policy. It is also distinct from the verified-chain product-admission finding; this is the control arm's independent eligibility gate.
Required closure before scientific product-only execution: classify the exact frozen PRODUCT_CANDIDATE response bytes, run the pinned parser admission, require exactly one allowed theorem/lemma with the exact problem/attempt-bound name and no forbidden syntax, and only then create an eligible successor-visible product. Bind that admission evidence to the durable producer/product record. Add negative real-adapter regressions for the current JSON payload, wrong declaration name, second declaration, and forbidden syntax; each must produce no eligible/visible product and no successor-visible memory. Preserve the separately required identical quarantined product-verifier event.
Current hosted failures remain blocking independently. MM06 authors no implementation edit, implementation push, merge, approval, or scientific adjudication.
Kitahl
left a comment
There was a problem hiding this comment.
MM06 / live G1-131 — REVIEW-ONLY supplemental exact-head review for 22ca5ecd0f74da2312b5863917b0826b762c421d.
VERDICT: AMEND — 0 approval, 0 merge authority, 0 scientific PASS.
[P1] The ordinary/portfolio baseline pre-dispatch adapter does not enforce the frozen attempt domain {0..15}. A self-consistent attempt 16 request can be registered and can cross the model-call boundary even though the frozen baseline authority declares exactly 16 slots and classifies an unregistered attempt as BLOCKED.
-
Frozen
goal1/CONFIRMATORY_BASELINES.jsonfixesattempts_per_problem_arm=16, exactattempt_indices=[0,...,15], andretry_policy.unregistered_attempt=BLOCKEDfor both ordinary and portfolio. -
The request/artifact/dispatch primitives enforce only non-negativity.
ScheduledChatArtifactEnvelope._attempt(),FrozenProblemRequest._attempt(), andDispatchEntry.create()/_n(... attempt_index ...)all accept any integer >=0; none caps the attempt at 15 or joins it to the frozen baseline attempt set. -
execution/baselines.py::_execute_baseline_attempt()revalidates the request, then checks only arm,visible_utf8_bytes, and thatrequest_utf8matches the caller-supplied request artifact. It does not load/authenticate the frozen baseline attempt set or rejectrequest.attempt > 15. It then generates a completion signer, callsDispatchAuthority.register(...), constructsBaselineDispatch, and immediately invokes_safe_model_call(...). -
DispatchManifestprevents duplicate logical keys but likewise does not constrain an attempt index to the sealed arm domain. Therefore a caller can construct a matching ordinary or portfolio request/artifact atattempt=16; all local type/identity checks can remain internally valid and the adapter can append that off-protocol logical slot before model dispatch.
Consequence: local dispatch/request consistency is not sufficient evidence that a baseline call belongs to the frozen 16-slot scientific arm. A protocol-invalid extra model call can occur before any later integration layer has an opportunity to reject it, contaminating the sealed call cardinality even if downstream credit is eventually denied.
Required closure before scientific baseline execution: derive/validate the exact allowed attempt set from authenticated CONFIRMATORY_BASELINES.json (and, for production, the exact frozen operator/protocol slot) before registration/model dispatch; reject attempts outside 0..15 with 0 model calls / 0 dispatch registration / 0 admissible completion. Add deterministic real-adapter regressions for ordinary and portfolio attempt 16 and a boundary positive case for attempt 15. Do not rely on a downstream ledger rejection after the model call.
Current hosted failures remain blocking independently: ordinary CI and the hermetic executor path are red while verifier-only qualification is green. MM06 authors no implementation edit, implementation push, merge, approval, or scientific adjudication.
Kitahl
left a comment
There was a problem hiding this comment.
MM06 / live G1-131 — REVIEW-ONLY supplemental review for exact PR head 22ca5ecd0f74da2312b5863917b0826b762c421d.
VERDICT: AMEND — 0 approval, 0 merge authority, 0 scientific PASS.
[P0] The new confirmatory arm controllers treat caller-supplied JSON mappings as the frozen scientific authority without authenticating them to the sealed protocol's exact baseline/product-control artifacts. A same-ID mutated contract can therefore change model-visible treatment bytes or the multi-fidelity schedule and still be accepted as “frozen.”
-
The sealed protocol freezes
CONFIRMATORY_BASELINES.jsonby exact Git blob2ab23d89b22dd7c21963da1e3543bf2dc0e39193, and freezesCONFIRMATORY_PRODUCT_CONTROLS.jsonby exact Git blobab2b298e29022f931ec141bdba50485b0967ad3fplus shared-surface projection SHA-256f68045d4d9554b0639b4abae86658c6bacee3f29ef1cc4b5c4f5deac7b654ed7. -
ProductChainController.__init__()accepts an arbitraryMapping[str, object]and authenticates it only bystatus == "FROZEN"andcontract_id == "goal1-confirmatory-product-controls-v1". It does not compare exact bytes/blob/projection identity to the sealed authority.render_request()then directly renders from that accepted mapping. -
MultiFidelityController.__init__()is weaker: it accepts arbitrary baseline/product mappings and only requiresbaselines["status"] == "FROZEN"andproducts["status"] == "FROZEN", followed by shape checks andset(slots) == {0..15}. Its stage IDs, ranks, output caps, promotion graph, and prompt templates are then derived directly from those caller-provided mappings. No sealed blob/projection identity is verified. -
Deterministic counterexample: deep-copy the real Product Controls object, keep
statusandcontract_idunchanged, altershared.product_chain_prompt_template, and instantiateProductChainController; the constructor accepts it andrender_request(0)emits the altered model-visible prompt. Likewise, keep bothstatusfields unchanged but change the S0 visible-output cap or an eligible-predecessor set while preserving all 16 attempt indices;MultiFidelityControlleraccepts the mapping and renders/executes the altered schedule. Internal shape consistency therefore does not prove frozen-authority identity.
Required closure before these controllers can authorize scientific calls: consume a validated frozen-authority capability rooted in the sealed protocol, or independently verify the exact baseline/product-control raw artifact identities and shared-surface projection before controller construction; bind those verified identities into controller/dispatch evidence; and add deterministic negative tests proving same-status/same-ID mutations of the prompt template, stage output cap, and promotion graph are rejected before request rendering/model dispatch. Do not use string fields such as FROZEN or contract_id as authentication.
Current head-associated hosted evidence remains red independently: verifier-only qualification is green while ordinary CI and the hermetic executor path fail. No failed, missing, or isolated-green evidence is promoted to approval. MM06 authors no implementation edit, implementation push, merge, approval, or scientific adjudication.
Kitahl
left a comment
There was a problem hiding this comment.
MM06 / live G1-131 — REVIEW-ONLY supplemental review for exact PR head 22ca5ecd0f74da2312b5863917b0826b762c421d.
VERDICT: AMEND — 0 approval, 0 merge authority, 0 scientific PASS.
[P0] The new exact product-chain classifier does not enforce the frozen missing_duplicate_or_ambiguous_discriminator = BLOCKED rule. A response with a valid first FINAL_ANSWER discriminator plus a second discriminator is still classified as FINAL_ANSWER and can become a verifier-PASSed solve.
-
Frozen
goal1/CONFIRMATORY_PRODUCT_CONTROLS.jsondefines classification from the exact first two visible lines and explicitly setsmissing_duplicate_or_ambiguous_discriminatortoBLOCKED. This is part of the product-chain shared surface and therefore applies to bothproduct_onlyandverified_chainsemantics. -
Exact-head
src/supernova_goal1/confirmatory_io.py::classify_product_response()only usesresponse.startswith(PRODUCT_PREFIX)/response.startswith(FINAL_PREFIX). Once the first prefix matches, it never checks whether another product/final discriminator occurs later in the response. In the FINAL branch, the remaining_FORBIDDEN_FINAL_TOKENScheck does not reject the discriminator comment lines. -
Deterministic counterexample:
FINAL_PREFIX + FINAL_PREFIX + b"norm_num\n"(orFINAL_PREFIX + PRODUCT_PREFIX + b"norm_num\n"). The current classifier enters the first FINAL branch and returnsConfirmatoryResponseKind.FINAL_ANSWER.build_verification_subject()then places the exact response after:= by\n; both discriminator copies are Lean comments, so the final tactic remains verifier-invocable.ProductChainController.complete_verifier_slot()setsfinal_solvedfrom onlyresponse_kind == FINAL_ANSWERplus verifier PASS. Thus a duplicate/ambiguous discriminator can cross the verifier boundary and, if the proof succeeds, become a solved attempt even though the frozen response algebra requires BLOCKED before scientific disposition. -
The current exact-head
tests/test_confirmatory_io.pyvalidates normal PRODUCT/FINAL/NO_ANSWER prefixes but has no negative duplicate/ambiguous-discriminator regression. This finding is distinct from the previously reported caller-selectedANSWEREDdefects in the older product/verified-chain adapters: even the newer classifier/controller path that derives kind from bytes still accepts protocol-invalid duplicated discriminator material.
Required closure before scientific product-chain execution: classify the entire response for discriminator multiplicity/ambiguity before verifier construction; require exactly one allowed discriminator pair in the exact permitted position; reject any second PRODUCT/FINAL/NO_ANSWER discriminator occurrence as BLOCKED with no admissible solve/product; and add deterministic regressions for duplicate FINAL, duplicate PRODUCT, and mixed FINAL+PRODUCT forms. A duplicate-discriminator final response must never reach final_solved=True, even if the remaining Lean tactic would verifier-PASS.
Current hosted failures remain blocking independently: ordinary CI and the hermetic executor path are red while verifier qualification is an isolated green job. No failed/missing/isolated-green evidence is promoted to approval.
MM06 authors no implementation edit, implementation push, merge, approval, or scientific adjudication.
Kitahl
left a comment
There was a problem hiding this comment.
MM06 / live G1-131 — REVIEW-ONLY supplemental exact-head review for 22ca5ecd0f74da2312b5863917b0826b762c421d.
VERDICT: AMEND — 0 approval, 0 merge authority, 0 scientific PASS.
[P0] The newer ProductChainController can authorize the next product-chain request after a local NO_ANSWER record without the authenticated terminal predecessor completion required by the frozen Product Controls authority. This leaves the product_only control path able to advance from unauthenticated predecessor state.
-
Frozen
goal1/CONFIRMATORY_PRODUCT_CONTROLS.jsonapplies its predecessor-completion rule to attempts 1..15 and requires attempt i to bind the authenticated terminal completion of attempt i-1 regardless of outcome. Required evidence includes predecessor attempt, dispatch ID, frozen request SHA, response-artifact SHA, completion SHA, and completion authentication; registration without terminal completion does not authorize the successor, and missing/invalid/replayed/cross-cell binding isBLOCKED. -
Exact-head
src/supernova_goal1/confirmatory_controller.py::ProductChainControllerhas no predecessor-completion capability or authentication input.render_request()checks only that no verifier slot is pending andattempt == len(_records). It renders the successor directly from local controller state. -
The
NO_ANSWERpath makes the bypass deterministic without any verifier object:submit_response()classifiesNO_ANSWER, appends a localProductAttemptRecord, and returnsNone. That immediately makes_pending is Noneandlen(_records) == 1, sorender_request(1)is permitted even though no authenticated attempt-0 terminal completion, dispatch identity, response-artifact binding, completion digest, or completion authentication has ever been supplied to this controller. -
Minimal counterexample against the public API:
render_request(0);submit_response(0, exact_NO_ANSWER_bytes);render_request(1). The third call is authorized by local sequence state alone. Under the frozen shared predecessor rule, it must be BLOCKED until the authenticated attempt-0 terminal completion is joined.
This is distinct from the earlier old-adapter finding where execute_verified_chain_step(... retry_of=None) made a verified-chain predecessor optional. This finding is the newer ProductChainController path and specifically leaves the product_only control surface able to advance without the shared authenticated predecessor completion.
Required closure before scientific product-chain execution: make every attempt >0 consume and authenticate exactly the terminal completion of attempt n-1 before successor rendering/registration/model dispatch; join the frozen required fields and same-run/problem/arm identity; reject missing, wrong-index, replayed, cross-cell, or unclosed predecessor evidence with 0 successor model calls. Add a deterministic product_only regression proving attempt 1 after exact attempt-0 NO_ANSWER cannot even render/dispatch until the authenticated predecessor completion is supplied, plus positive cases for all predecessor outcome classes.
Current hosted failures remain blocking independently. MM06 authors no implementation edit, implementation push, merge, approval, or scientific adjudication.
Kitahl
left a comment
There was a problem hiding this comment.
MM06 / live G1-131 — REVIEW-ONLY supplemental exact-head review for PR head 22ca5ecd0f74da2312b5863917b0826b762c421d.
VERDICT: AMEND — 0 approval, 0 merge authority, 0 scientific PASS.
[P0] ProductChainController.complete_verifier_slot() can replay a signed verifier result from one attempt into a later attempt when the verification subject bytes are identical, because it ignores the verifier binding's attempt/dispatch/request identity. This can mark a later slot verifier-invoked/solved without a verifier event bound to that slot.
-
The frozen Product Controls shared surface requires exactly one bound Lean verifier event per
FINAL_ANSWERand requires all 16 attempts to execute/reconcile even after an earlier verified pass. -
VerifierBindingalready carries the identities needed to prevent replay:run_id,protocol_dispatch_id,actual_dispatch_id,dispatch_entry_sha256,frozen_request_sha256,attempt_result_sha256,problem_id,arm_id,attempt_id,candidate_id, and the source/candidate/theorem digests. -
The controller discards most of that binding.
_subject_matches()compares onlycandidate_source_sha256,source_construction_sha256,theorem_statement_sha256, andtheorem_target_set_sha256.complete_verifier_slot()then takesverification.result.status, records the current local attempt, stores the supplied verifier record SHA, and setsfinal_solvedfromFINAL_ANSWER + PASS. It never checks thatverification.binding.attempt_id, run/arm/problem, dispatch IDs, frozen request, or attempt-result identity belong to the pending controller slot. -
The replay is deterministic for repeated final answers.
build_verification_subject()does not include attempt identity in aFINAL_ANSWERsubject; with no admitted products, identical exact final-answer bytes at attempts 0 and 1 produce the same candidate/source/theorem digests. A legitimate PASSProductionVerificationissued for attempt 0 can therefore satisfy_subject_matches()again for the identical attempt-1 subject. The controller can append attempt 1 withverifier_invoked=True, the sameverifier_record_sha256, andfinal_solved=Trueeven though no verifier record was issued for attempt 1. -
This is distinct from the already-recorded predecessor-completion defect. Even if the attempt-0 terminal predecessor completion is properly authenticated before attempt 1, the verifier evidence used to adjudicate attempt 1 still must be bound to attempt 1 rather than replayed from attempt 0.
Required closure before scientific product-chain execution: require the verifier binding to match the exact current run/problem/arm/attempt, registered dispatch, frozen request/response artifact/candidate identity, and current attempt-result identity before accepting a verifier slot; reject a verifier record whose binding belongs to any other attempt/dispatch even when candidate/source bytes are identical. Add a deterministic regression: complete attempt 0 with a legitimate PASS for exact final bytes R, submit the same R at attempt 1, replay the attempt-0 ProductionVerification, and prove rejection/no attempt-1 solve; only a separately bound attempt-1 verifier record may complete that slot.
Current hosted evidence remains red independently: ordinary CI and the hermetic executor path fail while verifier-only qualification is isolated green evidence. MM06 authors no implementation edit, implementation push, merge, approval, or scientific adjudication.
Kitahl
left a comment
There was a problem hiding this comment.
MM06 / live G1-131 — REVIEW-ONLY supplemental exact-head finding for 22ca5ecd0f74da2312b5863917b0826b762c421d.
VERDICT: AMEND — 0 approval, 0 merge authority, 0 scientific PASS.
[P0] MultiFidelityController.complete_verifier_slot() permits cross-attempt replay of one signed verifier record when two multi-fidelity attempts have identical verification-subject bytes. This can mark a later slot verified without the frozen per-attempt verifier event bound to that slot.
-
Frozen
CONFIRMATORY_PRODUCT_CONTROLS.jsonrequires multi-fidelity to complete all 16 model calls, thenRUN_EXACTLY_ONE_FINAL_LEAN_VERIFIER_ON_THE_FULL_VISIBLE_RESPONSEper attempt, withmulti_fidelity_verifier_calls = 16. -
VerifierBindingalready carries the anti-replay identities: run/experiment, protocol and actual dispatch IDs, dispatch-entry SHA, frozen-request SHA, attempt-result SHA, problem, arm,attempt_id, candidate identity, plus source/theorem digests. -
The controller does not consume those slot identities. Shared
_subject_matches()checks only candidate-source SHA, source-construction SHA, theorem-statement SHA, and theorem-target-set SHA.MultiFidelityController.complete_verifier_slot(attempt, verification)then applies the supplied verifier status/record SHA to the caller-selected local attempt and marks that attempt verified. It does not requireverification.binding.attempt_id, run/arm/problem, dispatch IDs, frozen request, or attempt-result identity to match the selected slot, and it does not require verifier-record uniqueness across slots. -
Deterministic replay: submit the same exact valid multi-fidelity visible response bytes at S0 attempts 0 and 1. Their
VerificationSubjects are identical because subject construction contains frozen source/theorem identities plus candidate response bytes, but not the multi-fidelity attempt/candidate-slot identity. After promotion freeze, obtain one legitimateProductionVerificationfor attempt 0.complete_verifier_slot(0, V0)accepts it. Callingcomplete_verifier_slot(1, V0)then presents the identical subject to_subject_matches(), so the same signed verifier record can be stored for attempt 1 even though no attempt-1-bound verifier event occurred. -
This is a distinct surface from the previously reported
ProductChainControllerreplay issue: multi-fidelity has a separate 16-call post-freeze verifier phase and a separate frozen cost cardinality. One signed call must not satisfy two of those slots.
Required closure before scientific multi-fidelity execution: require every verifier completion to match the exact current run/problem/arm/attempt, registered dispatch, frozen request, response/candidate artifact, and attempt-result identity; reject a verifier record bound to any other slot even when subject bytes are identical; enforce one unique bound verifier record per multi-fidelity attempt. Add a deterministic regression using identical responses at attempts 0 and 1: replaying the attempt-0 ProductionVerification into attempt 1 must be rejected, and verification-phase completion must remain false until a separately bound attempt-1 verifier record exists.
Current red/missing evidence remains blocking independently. MM06 authors no implementation edit, implementation push, merge, approval, or scientific adjudication.
Kitahl
left a comment
There was a problem hiding this comment.
MM06 / live G1-131 — REVIEW-ONLY supplemental exact-head review for 22ca5ecd0f74da2312b5863917b0826b762c421d.
VERDICT: AMEND — 0 approval, 0 merge authority, 0 scientific PASS.
[P1] The production activation nonce is irreversibly consumed before the real pre-dispatch preflight is complete. A deterministic failure in any later benchmark/model/verifier/resource check can therefore burn the one-shot activation with 0 model calls, despite confirmatory_run.py stating that the nonce is consumed only immediately before the first frozen dispatch.
-
src/supernova_goal1/confirmatory_run.pydocuments: “The activation nonce is consumed only immediately before the first frozen dispatch.” Butstart_first_attempt()callsDurableActivationAuthority.read_active()/activate_once(...)near the top of the entry point, before validating the operator-plan first slot, loading/verifying the benchmark report/source, rendering the frozen request, reading the cost policy, constructing the execution ledger, loading the model/executor bindings and receipt key, validating the verifier publication, or loading the verifier-evidence key. -
DurableActivationAuthority.activate_once()makes the activation durable by inserting the singleton activation row in a transaction and then returnsread_active(). Its own state-machine contract says the committed row is immutable. A later exception therefore does not roll activation back. -
Deterministic counterexample: use a fresh valid run directory and valid activation/operator secrets, but make a post-activation prerequisite unavailable — for example the locked benchmark report file or
execution-ledger-secret.raw.activate_once()can commit first; the later loader then raises beforeexecute_ordinary()and before the model callback. A retry followsread_active()because the activation row already exists. Result: the externally scarce one-shot activation has been consumed even though the promised first-dispatch boundary was never reached and model-call count remains 0. -
This is distinct from the previously reported cross-directory activation replay, preseeded activation-row bypass, and post-dispatch crash-recovery findings. Those concern namespace/authentication or recovery after dispatch registration. This finding is the activation linearization order before any scientific dispatch.
Required closure before countable production: either complete and authenticate every deterministic frozen-artifact/resource/preflight condition that may fail before consuming the activation nonce, or define one crash-safe authority state machine in which activation reservation and first frozen dispatch are atomically/recoverably joined. Add a real-entry negative regression that injects a post-activation prerequisite failure and proves 0 model calls plus no irreversibly consumed activation (or an explicitly recoverable reserved state), and a positive regression proving nonce consumption occurs only when the first frozen dispatch is ready to cross its authorized boundary.
Current head-associated hosted evidence remains red independently: ordinary CI fails and the hermetic executor path fails while verifier qualification is isolated green evidence. No red, missing, or isolated-green evidence is promoted to approval.
MM06 authors no implementation edit, implementation push, merge, approval, or scientific adjudication.
Kitahl
left a comment
There was a problem hiding this comment.
MM06 / live G1-131 — REVIEW-ONLY supplemental exact-head review for 22ca5ecd0f74da2312b5863917b0826b762c421d.
VERDICT: AMEND — 0 approval, 0 merge authority, 0 scientific PASS.
[P1] The new arm controllers do not durably consume a model slot when the exact returned response is BLOCKED by the frozen response-safety contract. A malformed/over-cap response raises before controller state changes, so the same attempt can be rendered again and retried even though the frozen arm permits exactly 16 model calls and declares unregistered retry/post-hoc calls BLOCKED.
-
Frozen
goal1/CONFIRMATORY_PRODUCT_CONTROLS.jsonfixesmodel_calls_per_arm = 16, charges failures/nonanswers/malformed scores, declaresunregistered_retry_product_promotion_or_cost_event = BLOCKED, and classifies over-limit/invalid-UTF-8 product-chain responses asBLOCKED. Unused budget cannot authorize post-hoc calls. -
In
ProductChainController.submit_response(),classify_product_response(...)runs before_recordsor_pendingare modified. Missing/ambiguous discriminator, invalid UTF-8, or over-cap bytes raise. After that exception,_pendingis stillNoneandlen(_records)is unchanged, sorender_request(attempt)accepts the same attempt again. -
MultiFidelityController.submit_response()has the same state-transition gap:classify_baseline_response(...)runs before the attempt record is appended. An over-cap/invalid/forbidden response raises without consuming the slot, leavingattempt == len(_records)true and permitting the same frozen slot to be rendered again. -
Deterministic public-API counterexample:
render_request(0); submit a response that the frozen classifier rejects (for product-chain, e.g. bareb"norm_num\n"; for multi-fidelity, an over-cap response); catch theValueError; callrender_request(0)again. The second render is accepted because no terminal BLOCKED record/dispatch identity was committed for the first returned response.
This finding does not claim the existing monotonic DispatchAuthority duplicate-key check is broken; that separate authority can reject a duplicate logical registration when it is actually joined. The defect here is that these newly introduced “frozen arm state machines” themselves do not bind a rejected terminal response to an irreversibly consumed model slot, so their public state permits post-response reissue and provides no evidence that an integration using them cannot turn a BLOCKED response into an extra call.
Required closure before these controllers authorize scientific calls: once a model call has produced any terminal provider response/error for a slot, transition that slot exactly once to an authenticated terminal state, including BLOCKED/INCOMPLETE where required; never make classifier failure equivalent to “attempt never happened”; bind the state machine to the monotonic dispatch/completion authority so a second render/dispatch for the same slot is impossible; and add deterministic regressions proving a rejected product-chain or multi-fidelity response leaves the arm at one consumed attempt and a retry/re-render cannot authorize another model call.
Current hosted evidence remains independently red (test and build-test-publish FAILURE; verifier-only qualification SUCCESS). MM06 authors no implementation edit, implementation push, merge, approval, or scientific adjudication.
Kitahl
left a comment
There was a problem hiding this comment.
MM06 / live G1-131 — REVIEW-ONLY supplemental exact-head review for 22ca5ecd0f74da2312b5863917b0826b762c421d.
VERDICT: AMEND — 0 approval, 0 merge authority, 0 scientific PASS.
[P1] The ordinary/portfolio baseline adapter trusts a caller-supplied AttemptStatus.NO_ANSWER even when the terminal response contains nonempty model-visible bytes. That lets a nonempty Lean candidate be recorded as a terminal nonanswer and bypass the response/verifier semantics frozen for the baseline arms.
-
Frozen
goal1/CONFIRMATORY_BASELINES.jsondefines the response artifact asLEAN_TACTIC_BODY, fixesempty_response = TERMINAL_NONANSWER, and binds output cost to the exact terminal response bytes. Thus the nonanswer condition is an exact response-byte condition: the empty response is the terminal nonanswer case. -
Exact-head
src/supernova_goal1/execution/baselines.py::ModelAttemptObservation.__post_init__()enforces nonempty bytes forANSWERED, but forNO_ANSWERit checks only thaterror is None; it does not requireresponse_utf8 == b""._safe_model_call()then preserves the caller's status and response bytes. -
_execute_baseline_attempt()constructs the signed terminal response artifact from those exact nonempty bytes but copies the caller-selectedNO_ANSWERstatus intoAttemptResult. Because verifier invocation is selected solely byobservation.status is AttemptStatus.ANSWERED, that nonempty response is not sent to the verifier and receives no verifier receipt. -
AttemptResultindependently permits the same state: it authenticates response-artifact identity and status/error shape but does not require aNO_ANSWERartifact to have zero visible bytes. Therefore the inconsistent pair can survive into the signed completion rather than failing at a later type boundary.
Deterministic counterexample: have the model port return ModelAttemptObservation(dispatch_id, b"norm_num\n", AttemptStatus.NO_ANSWER). Construction succeeds; the response artifact records the nonempty tactic bytes; the completion records NO_ANSWER; and the verifier branch is skipped. Under the frozen baseline response contract, those nonempty bytes are not the defined empty terminal-nonanswer case and must be classified/adjudicated from the exact response bytes instead of caller-selected status.
This is distinct from the already-recorded finding that a true empty NO_ANSWER currently skips the frozen required verifier event. Here the defect is response-kind integrity: a nonempty model return can be mislabeled as nonanswer before that separate verifier-cardinality question is reached.
Required closure before scientific baseline execution: derive baseline terminal response kind from the exact returned bytes under the authenticated frozen response contract; require NO_ANSWER iff the exact visible response is empty; reject/normalize no caller-selected status that conflicts with the bytes; preserve exact bytes and cost; and add deterministic real-adapter regressions proving nonempty bytes + NO_ANSWER cannot produce a nonanswer completion or suppress the applicable verifier/admissibility path. Add the analogous negative case for a caller-selected ERROR carrying nonempty provider-returned bytes unless an authenticated typed failure rule explicitly permits that representation.
Current hosted failures remain blocking independently: ordinary CI and the hermetic executor path are red while verifier-only qualification is isolated green evidence. No failed, missing, or isolated-green evidence is promoted to approval.
MM06 authors no implementation edit, implementation push, merge, approval, or scientific adjudication.
Kitahl
left a comment
There was a problem hiding this comment.
MM06 / live G1-131 — REVIEW-ONLY supplemental exact-head review for 22ca5ecd0f74da2312b5863917b0826b762c421d.
VERDICT: AMEND — 0 approval, 0 merge authority, 0 scientific PASS.
[P1] The production verifier replay guard is process-local, while the durable duplicate check happens only after a new two-phase verifier execution. Restarting the host can therefore run the same verifier slot again before SQLite rejects the duplicate evidence row, violating the frozen one-verifier-event cardinality and leaving the extra invocation outside the persisted cost/evidence set.
-
Frozen authority requires exact verifier cardinality:
CONFIRMATORY_BASELINES.jsonrequires onelean_final_verifierevent per ordinary/portfolio attempt and 16 verifier calls per arm;CONFIRMATORY_PRODUCT_CONTROLS.jsonrequires one bound verifier event for each product/final response and exactly 16 final verifier calls for multi-fidelity. -
Exact-head
src/supernova_goal1/production_verifier.py::ProductionVerifierPortkeeps replay state only in the in-memory_bindings_by_dispatchdict._verify_resolved_subject()rejects a dispatch only if that dict already contains it. A freshProductionVerifierPortafter process restart starts with an empty dict, passes that check for an already-persisted dispatch, and callsself.supervisor.run_and_record(...). -
VerifierSupervisor.run_and_record()immediately enters_run_two_phase_and_record(...), which launches the hostile elaborator/checker path and only afterward reaches_persist()/VerifierEvidenceStore.append(...). The durable store does haveUNIQUE(run_id, actual_dispatch_id)andUNIQUE(run_spec_id, problem_id, arm_id, attempt_id), but those constraints are enforced at the final SQLINSERT, after the new verifier execution and signing work have already occurred. -
Deterministic restart counterexample from the public state boundaries: persist one legitimate verification for dispatch D; discard/reconstruct the Python port while reusing the same durable evidence database; call
verify(D, same_candidate)again. The fresh in-memory guard does not know D. A second two-phase verifier run occurs, then the append fails on the existing durable row. The database still exposes only the original row even though two real verifier executions occurred. A crash after sandbox execution but before append has the same cardinality problem on retry without even a pre-existing row.
Consequence: durable evidence replay rejection is too late to enforce the frozen runtime/cost event count. An extra real verifier invocation can occur without a second admissible evidence row, so exact verifier-call cardinality and complete observable cost cannot be established from the persisted store alone.
Required closure before scientific execution: create a durable pre-verifier one-shot claim keyed to the exact run/problem/arm/attempt/dispatch before any sandbox launch, with crash-safe RESERVED -> COMPLETE reconciliation. If a completed slot already exists, reconstruct/read the authenticated result or reject before any new container invocation. Add a deterministic restart regression that recreates ProductionVerifierPort over the same SQLite store and proves a second call for the same dispatch causes 0 additional verifier sandbox executions, plus crash-boundary tests around the durable reservation.
Current hosted failures remain blocking independently (test and build-test-publish are red; verifier-only qualification is green). MM06 authors no implementation edit, implementation push, merge, approval, or scientific adjudication.
Kitahl
left a comment
There was a problem hiding this comment.
MM06 / live G1-131 — REVIEW-ONLY supplemental exact-head review for 22ca5ecd0f74da2312b5863917b0826b762c421d.
VERDICT: AMEND — 0 approval, 0 merge authority, 0 scientific PASS.
[P0] The new product-chain and multi-fidelity controllers can mark scientific model slots complete from caller-supplied response bytes without any bound request issuance, dispatch, model call, or authenticated terminal completion. Their complete / model_phase_complete properties therefore do not prove the frozen 16 model calls occurred.
-
Frozen
goal1/CONFIRMATORY_PRODUCT_CONTROLS.jsonfixes attempts0..15, requiresREGISTER_ALL_16_SLOTS_AND_THE_EXACT_PREDECESSOR_GRAPH_BEFORE_FIRST_MODEL_CALL, and requiresEXECUTE_AND_RECONCILE_ALL_16_EVEN_AFTER_A_VERIFIED_PASS. A response/completion is evidence for a scientific slot only after the corresponding model call and authenticated completion boundary exist. -
ProductChainControllerdoes not bindsubmit_response()torender_request()or to anyDispatchEntry/ frozen request / model-call receipt / authenticated terminal completion.submit_response(attempt, visible_utf8)checks onlyattempt == len(_records)plus response syntax. In particular, exactNO_ANSWERappends aProductAttemptRecordimmediately.completeis onlylen(_records) == 16 and _pending is None. -
Deterministic zero-call counterexample: instantiate the controller and call
submit_response(0, exact_NO_ANSWER), then 1 ... 15 the same way, without ever callingrender_request()and without any dispatch/model callback. All 16 calls are accepted as local records andcompletebecomes true although 0 model calls / 0 registered requests / 0 authenticated completions occurred. -
MultiFidelityControllerhas the same missing issuance boundary.submit_response()may be called directly for the next index; it appends aMultiFidelityAttemptRecordfrom caller bytes without provingrender_request()was issued or any model dispatch/completion exists.model_phase_completeis onlylen(_records) == 16, andfreeze_promotions()therefore accepts a caller-synthesized 16-record sequence as “all 16 model calls complete.”
This is distinct from the previously reported predecessor-completion and BLOCKED-response retry findings. Those assume a response exists for a slot and ask whether successor/retry semantics are correct. This defect is earlier: the controller never proves that a returned response belongs to an actually issued/registered model call at all.
Required closure before these controllers can authorize scientific state: issue a one-shot request/dispatch capability for each exact frozen slot; make submit_response consume an authenticated terminal model completion bound to that exact run/problem/arm/attempt/request/dispatch rather than arbitrary bytes; refuse response submission before issuance or after capability reuse; and derive complete / model_phase_complete only from all 16 authenticated model completions. Add deterministic negative regressions proving 16 direct submit_response calls with no request issuance/dispatch leave the arm incomplete and authorize 0 successor/verifier/scientific completion.
Current hosted evidence remains blocking independently: #96 test and build-test-publish are FAILURE while verifier-only qualification is SUCCESS. MM06 authors no implementation edit, implementation push, merge, approval, or scientific adjudication.
Kitahl
left a comment
There was a problem hiding this comment.
MM06 / live G1-131 — REVIEW-ONLY supplemental exact-head review for 22ca5ecd0f74da2312b5863917b0826b762c421d.
VERDICT: AMEND — 0 approval, 0 merge authority, 0 scientific PASS.
[P0] An exact discriminator-only PRODUCT_CANDIDATE is classified as a product, then crashes before the frozen required product-verifier/parser event. The slot is left unconsumed and can be retried.
-
Frozen
goal1/CONFIRMATORY_PRODUCT_CONTROLS.jsonclassifies product-chain responses from the exact first two visible discriminator lines, requires product syntax/name admission separately, sets parse/name-policy failure toPRODUCT_NOT_ADMISSIBLE, and requiresONE_IDENTICAL_BOUND_LEAN_VERIFIER_EVENT_PER_PRODUCT_CANDIDATE. Forproduct_only, that verifier result is quarantined from admission; the event itself is still mandatory. -
Exact-head
confirmatory_io.classify_product_response()accepts any response beginning withPRODUCT_PREFIX. For the exact bytesPRODUCT_PREFIXand nothing after it,body = b"", but the function still returnsConfirmatoryResponseKind.PRODUCT_CANDIDATEwithproduct_parser_source_utf8=b"". -
ProductChainController.submit_response()then callsbuild_verification_subject()before setting_pending.VerificationSubject.__post_init__()rejects an empty non-nullproduct_parser_sourcewithTypeError("product_parser_source must be non-empty exact bytes"). Therefore the exact discriminator-only product response never becomes a pending verifier slot, never receives the mandatory product verifier event/parser-admission result, and never becomes a terminalPRODUCT_NOT_ADMISSIBLErecord. -
Deterministic public-API counterexample:
render_request(0)→submit_response(0, PRODUCT_PREFIX)raises →render_request(0)succeeds again because_recordsis still empty and_pendingis stillNone. This is distinct from the previously recorded generic BLOCKED-response retry finding: the frozen first-two-line classifier makes these bytes aPRODUCT_CANDIDATE; the defect is that an invalid/empty declaration body bypasses the required product verifier/parser event instead of becoming a charged, non-admissible product attempt.
Required closure before scientific product-chain execution: represent discriminator-valid but parser-invalid product candidates all the way through the bound product-verifier event; record signed parser admission false (including the empty-declaration case), preserve the verifier event/cost, consume the model slot exactly once, and forbid re-render/retry. Add a deterministic regression using exactly PRODUCT_PREFIX with no body: expected outcome is one consumed attempt, one bound product-verifier event, syntax_admissible=false, product_admitted=false, and no second render/dispatch for that slot.
Current hosted failures remain blocking independently; no failed, missing, or isolated-green evidence is promoted to approval. MM06 authors no implementation edit, implementation push, merge, approval, or scientific adjudication.
Kitahl
left a comment
There was a problem hiding this comment.
MM06 / live G1-131 — REVIEW-ONLY supplemental exact-head review for 22ca5ecd0f74da2312b5863917b0826b762c421d.
VERDICT: AMEND — 0 approval, 0 merge authority, 0 scientific PASS.
[P1] The new exact response classifiers have a command-grammar hole: Lean #... commands such as #check are not rejected by _FORBIDDEN_FINAL_TOKENS, so protocol-invalid non-tactic commands can survive classification and reach the verifier.
-
Frozen
goal1/CONFIRMATORY_BASELINES.jsondefines the response artifact asLEAN_TACTIC_BODY; its exact common prompt explicitly says to return only the tactic body and forbids commands. The product-chain final-answer surface likewise requires only the final tactic body under the frozen response contract. -
Exact-head
src/supernova_goal1/confirmatory_io.pyenforces final-response syntax with_FORBIDDEN_FINAL_TOKENS, but that regex enumerates imports/declarations/namespaces/options/sorry/admitand does not include Lean command forms beginning with#.classify_baseline_response()and the FINAL branch ofclassify_product_response()both rely on this regex. -
Lean's language reference defines
#check termas a top-levelcommand. The production verifier does not independently enforce the frozen tactic-body grammar:VerifierSupervisor._run_two_phase_and_record()formssolution_source = source + candidate,runtime/goal1_verifier/entrypoint.py::compile_module()compiles that full module, and export/checking is target-scoped to the configured theorem names/allowed axioms rather than proving that no extra informational command was present. -
Deterministic classifier counterexamples are therefore:
- product final:
FINAL_PREFIX + b"norm_num\n#check Nat\n"; - multi-fidelity:
b"-- MULTI_FIDELITY_SELF_SCORE=9999\nnorm_num\n#check Nat\n".
Both evade the current forbidden-token regex. The product form is classifiedFINAL_ANSWER; the multi-fidelity form remains a verifier candidate. The extra#checkis not a tactic-body response even though the classifier does not reject it.
- product final:
-
Current
tests/test_confirmatory_io.pycovers normal PRODUCT/FINAL/NO_ANSWER forms and score parsing but has no negative#check/#eval/other command regression.
This is distinct from the already-recorded old baseline-adapter bypass: this finding is in the newer byte-derived classifier/controller path itself, including ProductChainController and MultiFidelityController users.
Required closure before these classifiers authorize scientific state: parse/enforce the exact allowed tactic-body grammar rather than a partial denylist, or otherwise prove the entire candidate contains only the permitted tactic body/comment surface; reject top-level #check, #eval, #print, and any other non-permitted command before verifier/scientific disposition; add deterministic negative regressions for product-final and multi-fidelity responses carrying a valid proof followed by such a command. Preserve exact returned bytes/cost when blocked.
Current red/missing evidence remains blocking independently. MM06 authors no implementation edit, implementation push, merge, approval, or scientific adjudication.
Implements the two-phase hostile-Lean verifier boundary, host-observed signed evidence, immutable runtime identity, hermetic Kimina executor, and the preregistered exactly-two-attempt non-credit validation demo.
Local focused verification:
Immutable verifier publication run
33498934176passed real benign VALID and hostile containment. Image:ghcr.io/kitahl/supernova-goal1-verifier@sha256:dc7e5754612925b8d0de1482fc87ba13b39370c7965f8286912587f030efddc5.PR workflow run
33501251455passed executor build, networkless preflight, locked benchmark preparation, and immutable-verifier qualification, then stopped before any model call because signed synthetic prose rejection took7217 ms, above the prospectively frozen5000 mswall gate. Evidence artifact:9797961730; ZIP SHA-256f87cd45bce31f451bb6b439352802edd7f3731ac9ba444412a9e043a4a3f1753.Scientific attempts remain zero. Do not merge until the non-scientific timing-gate decision documented in
reports/STATUS.mdis resolved and CI passes.