Decoder accuracy: close the gap to SC-004 ≥85% target (currently ~53%)

[bashandbone]

Context

Phase 4 PR-6 (T057 decoder accuracy harness) landed the SC-004 measurement gate as crates/engine/tests/decoder_accuracy.rs::resolution_rate_at_0_85. The test is #[ignore]-marked because the decoder's current empirical accuracy is well below the spec's 85% target. The complementary resolution_rate_does_not_regress is always-on at a 50% floor and prevents the accuracy from getting worse.

This issue tracks closing the gap so the #[ignore] can be removed and SC-004 lands as a load-bearing gate.

Current state (2026-04-25 capture, branch 004-phase4-pr6-bench-accuracy-gates)

Per-class breakdown:
  GarbledDelimiter: 51/51  (100.0%)  ✅
  MissingDelimiter:  0/17  (  0.0%)  ❌
  Reordering:       41/41  (100.0%)  ✅
  SupersededToken:   2/3   ( 66.7%)  ⚠️
  Typo:             26/130 ( 20.0%)  ❌
  WrongCase:        18/18  (100.0%)  ✅
  Aggregate:       138/260 ( 53.1%)

To reach 85% aggregate (221/260), the decoder needs to recover roughly:

• +83 fixtures if the gain comes purely from Typo (104/130 → 80% Typo-class accuracy)
• +17 fixtures if MissingDelimiter is fully recovered (17/17 → 100%) plus +66 from Typo
• Or a mix across both classes — the per-class table above shows where the headroom is.

Specific gaps surfaced by the harness

The first five unresolved samples from the Typo class (representative, not exhaustive — full list reproducible by running the gate):

1. "TOP SECRET//SI/UK//NOFORN" → expected "TOP SECRET//SI/TK//NOFORN". Decoder returned Unambiguous(TopSecret) but did NOT correct the SCI sub-compartment typo (UK → TK). The fuzzy matcher's per-token pass appears not to cover SCI sub-compartment positions.
2. "SECRET//USAR-..." → expected "SECRET//SAR-...". Decoder produced 3 SCI controls instead of recognizing the multi-word SAR program identifier (USAR- typo prefix).
3. "TPP SECRET//SI//NOFORN" → expected "TOP SECRET//SI//NOFORN". Decoder lost the classification entirely (cls=None); TPP did not fuzzy-match TOP.
4. "SECRET//SAR-BP-J1 2J54-..." → expected "SECRET//SAR-BP-J12 J54-...". Intra-SAR-token typo (whitespace shift inside a multi-word SAR program identifier).
5. "SECRET//SAR-...//NOFORON" → expected "...//NOFORN". Returned zero-candidate. NOFORON is edit-distance-1 from NOFORN (insertion) but the fuzzy matcher rejected it — likely the per-token MIN_FUZZY_LEN gate or insertion handling.

Likely fix areas (decoder)

Based on the failure patterns above, the candidate work breakdown:

• Edit-distance-1 insertions for short tokens — NOFORON zero-candidate is the cleanest case. Probably a MIN_FUZZY_LEN edge or an insertion-handling gap in marque-core::fuzzy::FuzzyVocabMatcher.
• SCI sub-compartment fuzzy correction — extend the fuzzy pass beyond the bare control system into compartments / sub-compartments per CAPCO-2016 §A.6.
• SAR program-identifier fuzzy correction — multi-word SAR identifiers (CAPCO-2016 §H.5) need their own correction pass; the current per-token matcher splits them.
• MissingDelimiter class — 0/17 means the canonicalizer doesn't reconstruct missing // separators in canonical positions. Probably a missing transform in generate_candidate_bytes.
• Classification-token typos beyond edit-distance-1 — TPP SECRET is edit-distance-1 from TOP SECRET, but the matcher returned cls=None. Worth confirming whether TPP is being normalized differently from T0P-style typos.

Acceptance

• cargo test -p marque-engine --test decoder_accuracy --features decoder-harness -- --ignored exits 0.
• #[ignore] is removed from resolution_rate_at_0_85.
• The regression-floor constant AGGREGATE_FLOOR_REGRESSION in the same file is ratcheted up alongside the decoder improvements, so a future regression below the new measured rate also fails CI.

Constitution / spec references

• Spec SC-004 (specs/004-constraints-decoder-vocab/spec.md line 149): "Of a mangled-marking fixture of at least 200 labeled cases, at least 85 percent are resolved to the expected canonical marking when the probabilistic recognizer's aggregate confidence threshold is set to 0.85 or higher."
• Constitution Principle VIII (Authoritative Source Fidelity): any new fuzzy-correction transform that touches CAPCO syntax must cite the relevant §A–H passage in crates/capco/docs/CAPCO-2016.md.

Out of scope

• Lowering SC-004 below 85% — the spec target stands.
• Removing fixtures from tests/fixtures/mangled/ to inflate the rate — the SC-004 floor of ≥200 cases is enforced by the harness's MIN_FIXTURE_COUNT constant.

[github-actions]

🤖 Hi @bashandbone, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

[github-actions]

🤖 I'm sorry @bashandbone, but I was unable to process your request. Please see the logs for more details.

[bashandbone]

Root-cause analysis from diagnostic harness (PR #136)

Ran a per-fixture pipeline trace on hand-picked unresolved samples from each mangling class. Diagnostic test added in #136 — cargo test -p marque-engine --features decoder-harness --test decoder_diagnostic -- --ignored --nocapture to reproduce. Four distinct root causes identified, ordered by corpus impact:

1. correction_vocab is missing all long-form dissem controls — dominant cause

marque_ism::CapcoTokenSet::correction_vocab() returns ALL_CVE_TOKENS, which contains the abbreviations (NF, OC, PR) plus a few single-form tokens (RELIDO, FISA, FOUO). It does not contain NOFORN, ORCON, PROPIN, EXDIS, NODIS, EYESONLY, etc.

The strict parser handles long forms via parse_dissem_full_form → marking_forms::banner_to_portion (crates/core/src/parser.rs:1042), but the fuzzy matcher only sees correction_vocab, so any long-form dissem typo has no canonical target.

Direct probes from the diagnostic:

"NOFORON" → None — closest 5: [(4, "FOUO"), (5, "NF"), (6, "BUR"), (6, "DSEN"), (6, "FISA")]
"NOFRON"  → None — closest 5: [(4, "FOUO"), (4, "NF"), (5, "BUR"), (5, "DSEN"), (5, "FISA")]
"ORCON"   → None — closest 5: [(3, "OC"),   (4, "C"),  (4, "DSEN"), (4, "FOUO"), (4, "HCS")]

NOFORON is distance 1 from NOFORN — should be a clean correction. But NOFORN isn't in vocab; closest match is FOUO at distance 4, well above MAX_EDIT_DISTANCE = 2. Matcher correctly returns None, candidate falls through as TokenKind::Unknown, decoder step 3a discards (crates/engine/src/decoder.rs:205-212).

Estimated corpus impact: most of Typo class (104/130 unresolved). Closing this should jump aggregate accuracy from ~53% well into the 70%+ range.

Fix sketch: extend correction_vocab() (or build a parallel correction_vocab_with_long_forms()) to include the banner long forms from marque_ism::marking_forms. The fuzzy matcher then either (a) emits the canonical short form directly, or (b) leaves the long form in place and lets parse_dissem_full_form translate at strict-parse time. Either works — (b) is cheaper and preserves the source's chosen abbreviation level for downstream rules.

2. MissingDelimiter class is a complete shape gap (0/17)

generate_candidate_bytes (decoder.rs:473) only normalizes existing delimiters; there is no transform that inserts a missing //. So:

[MissingDelimiter / NOFORN EXDIS]
  observed: "SECRET//NOFORN EXDIS"
  expected: "SECRET//NOFORN//EXDIS"
  fuzzy result: "SECRET//NOFORN EXDIS"  (no changes)
  strict-parse(canonical): OK — token_kinds={Classification=1, Separator=1, Unknown=1}
    Unknown spans: ["NOFORN EXDIS"]
    → REJECTED by decoder step 3a
  decoder verdict: Ambiguous (candidates=0)

Same shape on TOP SECRET//SI/TK NOFORN (Unknown="TK NOFORN") and TOP SECRET HCS-P INTEL OPS//ORCON/NOFORN (no // between classification and SCI).

Fix sketch: add a try_insert_delimiter candidate generator that walks token boundaries and tries // insertion at points where two adjacent tokens belong to different categories (classification→SCI, SCI→dissem, dissem→dissem). Bounded by K=8 so it doesn't combinatorially explode. Pairs with a category_of_token helper that already exists in try_canonical_reorder.

3. SAR-prefix typos unreachable through fuzzy

[Typo / USAR]
  observed: "SECRET//USAR-BP-J12 J54-K15/CD-YYY 456 689/XR-XRA RB//NOFORN"
  expected: "SECRET//SAR-BP-J12 J54-K15/CD-YYY 456 689/XR-XRA RB//NOFORN"
  fuzzy result: ... USAR-BP-J12 ...  (no changes)

scan_token (decoder.rs:723) keeps internal hyphens, so USAR-BP-J12 is one mega-token. Distance to anything in vocab is ≥6. Even with better tokenization, SAR is not in ALL_CVE_TOKENS either — SAR programs are agency-extensible by design (per CLAUDE.md: "the ODNI public CVEnumISMSAR.xml is empty").

Fix sketch: a targeted try_strip_prefix_chars transform for SAR-shape compounds (drop 1-2 leading non-vocab chars before the first -), or extend tokenization to split on the first - for SAR-shape compounds and let fuzzy see the prefix in isolation. Smaller wedge — corpus has only 3 USAR-shape fixtures.

4. Short-token typos below the fuzzy floor (intentional)

"TPP" → None — closest 5: [(2, "PR"), (2, "TK"), (2, "TS"), (3, "BUR"), (3, "C")]

TPP (length 3) at distance 2 to PR/TK/TS scores correction_confidence(2, 3) = 0.30, below MIN_USEFUL_CONFIDENCE = 0.45. Matcher correctly rejects.

UK→TK is even harder: UK is 2 chars (below MIN_FUZZY_LEN = 3) AND UK is a valid country trigraph, so is_trigraph() fast-paths return None before fuzzy even runs.

The TPP case has a possible second angle: TOP SECRET is in vocab as a single 10-char entry (with space), but fuzzy_correct_tokens scans on alnum boundaries so it never compares "TPP SECRET" as a unit against "TOP SECRET". Multi-word fuzzy is bigger surgery than is justified here.

Fix sketch: leave the matcher's floors alone — they're correctly tuned to suppress noise. These ~6 fixtures are an acceptable accuracy cost to keep false positives out.

Estimated path to SC-004 ≥85%

Fix	Class affected	Expected resolved gain	Cumulative
Long-form dissem in vocab (#1)	Typo	+60–90	~75–87%
Missing // insertion (#2)	MissingDelimiter	+12–17	~80–93%
SAR-prefix strip (#3)	Typo	+2–3	tiny

Fix #1 alone likely closes the SC-004 gap by itself; #2 is independent and small; #3 is incremental.

Reproduction

git fetch origin && git checkout decoder-diagnostic-issue-133
cargo test -p marque-engine --features decoder-harness \
  --test decoder_diagnostic -- --ignored --nocapture

Diagnostic stays in the repo (#[ignore]-d, feature-gated) so follow-up PRs to close #133 can re-run the trace after each candidate fix and confirm where in the pipeline the change took effect.

Submitted by: Claude noreply@anthropic.com

[bashandbone]

I wanted to provide some overall guidance here based on @claude 's post above (my name, claude wrote it).

1. I think the short path on typos is probably too conservative. Importantly, trigraphs are three characters -- UK is not a valid country code (it's GBR). We could reduce false positives based on position.

   I'm thinking something like a heuristic that:
   a) considers where the token is in the marking
   b) if it's at the very beginning, it's likely a classification (and this only applies to portion markings) -- this would be a scenario like (RS//REL TO USA, FVEY). We know from position (and key proximity) that RS is, with high confidence, really TS.
   c) The safest form would be if the first token is malformed/unknown, you just try to map (pseudo code):
   if [AaWwEeZzs] -> S
   if [Vvc] -> C
   if [TtRrYyHhGg][AaWwEeZzSs] -> TS
   if len(2) and [Ff][AaWwEeZzSs] -> TS
   if len(1) and [Ff] -> C
   if len(1) and [Xx] -> recommend "did you mean S"? (S is more common and also better to default to the higher level)
   else:
   if len(1):
   did you mean to write S or C?
   elif len(2):
   did you mean to write TS?
   if len 3:
   see if it's a valid country code (fix it to be a non-u.s. classification) otherwise see if it fuzzy matches on and suggest those.
   (note I avoided some keys that were more ambiguous, and used narrower strategies for them)

   Any of the above fix paths would be a fix and warn pattern ('we fixed this to TS from YS, if that's not what you meant to write, then please correct it')

2. We could similarly improve confidence by looking at position as an indicator. For example: (TS//SI/JCS-P//OC//NC) - 'JCS-P' -> 'HCS-P' with high confidence given that it is otherwise correct.

3. Behavior -- to the extent possible (noting that subcompartments and SAR entries can get ambiguous very quicky), we ultimately want marque to be delimiter agnostic. It should be able to distinguish that S/NF/oc) -> (S//NF/OC). Or that (TS//REL TO USA FVEY DEU FRA) should be (TS//REL TO USA, DEU, FVEY, FRA). So we need to figure out the delimiter issue and come up with a way to accomplish that goal with respect to delimiters.