Skip to content

Plan 0.16.0: 32 issues, eleven clusters, and the twelve orderings that stop the rework #902

Description

@raeq

Summary

Status, re-read at 091b477a. 12 of the 32 have shipped (#732, #739, #759, #815,
#833, #903, #905, #701, #937, #901, #704, #728) and #727 is in review (#945). Six issues
joined the milestone after this plan and have already closed (#907, #910, #914, #916,
#917, #918); they are in the index, because an issue absent from it is a planning bug and
that rule applies to arrivals as well as to the original denominator. R1, R6, R7, R10,
R12 and R14 are satisfied. R2 was violated once and the repair is R13.
Details in
Status at the bottom.

Milestone 0.16.0 carries 32 open issues, read at 8599313 — the 0.15.0 release commit. Six of them (#815, #816, #860, #884, #894, #896) were deferred out of 0.15.0 with a reason recorded on each; the rest predate that milestone or were filed during it.

Read end to end, they are not 30 independent changes. They cluster into ten subsystems, and the largest — the confusable fold and its generator — has seven issues whose measurements all move if one of them lands first.

Three things this issue fixes:

  1. Containment — which issues share a generated artifact, a census denominator, or a false-positive argument, and should therefore land together.
  2. Order — which changes make later work cheaper, and which create rework if they land late.
  3. Coverage — every one of the 30 appears in the index at the bottom. An issue absent from it is a planning bug.

This issue is not implemented by a pull request. It closes when the milestone ships.

The denominator:

646 650 675 694 701 703 704 705 727 728 730 731 732 736 738 739 740 742 759
815 816 833 836 848 860 884 894 896 900 901 903 905

#905 was found while implementing #903 — the allocation gate is stochastic: a
process-wide counter with a margin of one failed #904, which changes no Rust, and passed
on re-run of the same commit. It belongs with the CI-hygiene work rather than any cluster
below, and it goes early: while it stands, every red build has to be re-run before it can
be believed.

#903 was filed after this plan and is already implemented (#904). It is the
hardcoded-transform half of #759, so it gates it — see R12.

A caveat on the evidence. I read the full body of #650, #675, #694 and #833, and for the rest the title, the scope section, and the issues each cross-references. The clustering rests on shared files and shared denominators, which are visible from that. The four constraints marked (inferred) below rest on the cross-references rather than on a claim the issue makes outright — check those before relying on them.


Where the rework comes from

Twelve ordering constraints, each stated as the rework it prevents.

# Constraint Rework if ignored
R1 #833 before #646, #650 and #815. #833 is the general form of #245: the presets run NFKC before the fold, so the fold never sees the source code point, and 68 Latin / 8 Cyrillic rows are unreachable. #650's entire cost argument is where in a preset the class runs, and #815's census counts what reaches ASCII. Move the reachable set after either is measured and both are re-measured.
R2 Every table change before every frozen census. #833, #848, #836, #815 and #738 all regenerate or extend confusables_*.tsv. #884, #816, #736 and #732 all publish counts derived from it. This is #762's R3 and it cost real rework there. Violated: #815 shipped its census while #738 is still open — see R13.
R3 #646 before #650. #650 exists to answer the question #646 item 1 left open — whether the I-family is in scope. Building #650 first decides it by accident.
R4 #894 and #896 in one pull request. Both are "Python-only, owed to the Rust API and five bindings", both need the same parity-matrix rows, and both hit the same C-ABI question: a result object that owns a String needs either a free function or a caller-supplied buffer. Split them and that decision is made twice, with two chances to differ.
R5 #646 before #896. #896 threads digit_policy through the Rust API and five bindings. #646 changes which policies exist (the capital-I family has none). Plumb first and the plumbing is redone.
R6 #759 before #732, #736 and #816. #759 changes what benchmarks/adversarial_eval measures — it adds a clean-text cost metric and unhardcodes strip_obfuscation. All three of the others publish numbers that harness produces.
R7 #704 before #705. (satisfied — #704 shipped in #944) #705 is SARIF output and stable fingerprints for CI adoption. #704 is the CLI being able to open a file at all. SARIF for a tool that cannot read the repository is untestable.
R8 #900 before #901. #901 is the same primitive-layer confusion seen from the other side: #900 is find_confusables reporting whole non-Latin words, #901 is bilingual text being indistinguishable from a spoof. Fix the reporting first, then decide what the remaining ambiguity means.
R9 #675's go/no-go decision before #694 is built. #675 moves every URL on the site/api/transforms.html becomes /latest/api/transforms.html. #694 corrects the rel=canonical tag and 74 sitemap entries. Build #694 first and every one of those 74 is rewritten by #675. #675 also questions its own premise ("whether anybody is pinned to an old release"), so the decision may be no — in which case #694 proceeds standalone. Either way the decision comes first.
R10 #730 before #731. (inferred) #731 asks for a map for derived deterministic identifiers and says no key builder is safe for one and the variance direction has no predicate. #730 builds a verification-path predicate (is_canonical). The map wants the predicate to point at.
R12 #903 before #759. #903 makes the harness's transform a parameter; #759 asks for a clean-text cost metric and for the transform to stop being hardcoded. Build #759 first and half of it is rewritten. Shipped in #904.
R13 #738 before #815's census can be called frozen. R2 named this class and wave 2 then put a census and three table changes in one wave with no order between them, so it happened. #815 shipped tests/fixtures/latin_shape_exposure.tsv with a hard assert len(_fixture_rows()) == 299 plus a per-block count for each of seven blocks. #738's first tranche is 793 novel Latin-target pairs — "X → Latin, the direction disarm folds", which is exactly the population that gives a listed code point a path. Any overlap removes rows, and the gate's own contract is "a code point LEAVING the file is a fix". The repair is cheap and must be deliberate: land #738, regenerate with scripts/gen_latin_shape_exposure.py, move the 299, re-check seven block counts. #836 and #848 do not have this problem#836 keys on confusables_contractions.tsv and the census walks single code points, and #848 is Arabic-on-Arabic with no Latin target, so neither can move a Latin-shape census whatever the order.
R14 #701 and #727 settle one decision, and it is #701's. (delivered — #940 built the type, #945 adds percent as the fourth scheme on it) Both want a decode-as-evidence surface rather than a substituting decoder, and #727 says so outright: its preferred shape is "a decode-for-inspection primitive that returns what the escapes spelled rather than a substituted string — the same argument #701 makes for the three smuggling carriers, and for the same reason". #701 proposes decode_smuggled(text) -> Vec<Payload> over three schemes; percent-escapes would be a fourth on the same type. Build them apart and the evidence type — its span convention, its confidence/None contract, its error answers for malformed input — is designed twice, with two chances to differ. That is R4's argument exactly. #701 owns it because it carries three schemes and the arithmetic already exists at src/invisibles.rs:116; #727 extends it. This plan has them in different clusters (4 and 7) and different waves (5 and 7) with nothing linking them.
R11 #860 is a decision, not a fix. It changes list_profiles()'s return type, which is in tests/test_api_stability.py's pinned surface and in five bindings. That is a ### Changed (breaking) entry and a call for @raeq, not something to assume — the same shape as 0.15.0's R17 (#776).

The ten clusters

1. The confusable fold and its generator — seven issues, one artifact

#833 #646 #650 #815 #836 #848 #738

Every one of these regenerates or extends confusables_*.tsv, or measures what that table reaches. #833 goes first because it changes what the fold can see at all.

2. Coverage introspection — #884

unmapped_confusables accepts four of the 57 scripts (the issue says two; #792/#849 added Arabic and Hebrew after it was written). Its denominator is the table, so it follows cluster 1. 0.15.0's #888 fixed the error message and explicitly left this open.

3. Binding parity — #894 #896

Both are 0.15.0 deferrals, both Python-only today, both owed to the Rust API and five bindings. One pull request (R4), and #646 first (R5).

4. The detector — #900 #901 #739 #740 #730 #701

5. CLI and CI adoption — #704#705

The CLI cannot open a file, so the one API built for scanning cannot be pointed at anything. SARIF and line-independent fingerprints follow, so a repository can adopt the check before it is clean.

6. Key semantics — #728 #731

7. Encoders and decoders — #727 #742 #703

8. Docs site — #675 #694

#675 versions the site; #694 fixes canonicals and the sitemap. R9 orders them, and #675 may be declined.

9. Benchmarks and validation pages — #903 #759 #732 #736 #816

#903 is done (#904): the harness now scores a named surface rather than a hardcoded strip_obfuscation, with the default unchanged. #759 is what remains of the harness work — a clean-text cost metric — and the other three publish its numbers. R6 and R12 order them. #816 also depends on cluster 1 — it scores 38 non-standard Unicode sets, and four of its classes fold nowhere, which is #815's territory.

10. CI hygiene — #905

The allocation gate blocks merges on a counter any thread in the process can move, with a
margin of one. Same shape as the four Tier-3 draws #898 removed. Independent of every
other cluster, and worth doing first for the same reason #762 put #46 ahead of its
milestone: a lane that cannot be trusted taxes everything routed through it.

11. Surface introspection — #860

Blocked on a decision (R11).


Suggested waves

wave issues why here status
0 #833 Changes what the fold reaches. Everything in cluster 1 and 2 measures against it. done — R1 satisfied, #646/#650 unblocked
1 #646#650 The policy question, in the order the issues themselves establish. next
2 #815, #836, #848, #738 The remaining table extensions, once the reachable set is settled. #815 done, but out of order — see R13
3 #884, #816 The censuses, once their denominator has stopped moving. held for wave 2
4 #894 + #896 Binding parity, after #646.
5 #900#901, #739, #740, #730, #701 The detector. #730 before #731. all done but #730 §2/§5
6 #704#705 CLI, then SARIF. #704 done (#944); #705 unblocked, next
7 #728, #731, #727, #742, #703 Semantics and the encoder gaps. #728 done (#943); #727 in review (#945); #731 unblocked
8 #903#759#732, #736 Harness before the pages that quote it. R6 and R12 satisfied; only #736 remains
9 #675 decision → #694 Docs site, decision first. still needs the decision
#860 Held for a decision on the return type. still held
first #905 Independent, and every red build is unreliable until it lands. done

Waves 4 through 7 are independent of one another and of waves 0–3; they are ordered here only for readability. Waves 0 → 1 → 2 → 3 is the one chain where order is load-bearing.


Index

Every issue in the denominator, and where it sits.

issue cluster wave gated by status
#833 fold 0 closed
#646 fold 1 #833 (R1) — satisfied open · next
#650 fold 1 #646 (R3), #833 (R1) closed (#938)
#815 fold 2 #833 (R1) closed · census now needs R13
#836 fold 2 table (R2) open · cannot move #815 (R13)
#848 fold 2 table (R2) open · cannot move #815 (R13)
#738 fold 2 table (R2) open · must precede #815's census (R13)
#884 coverage 3 cluster 1 (R2) open
#816 benchmarks 3 #815 (R2), #759 (R6) — R6 satisfied open
#894 parity 4 with #896 (R4) open
#896 parity 4 with #894 (R4), #646 (R5) open
#900 detector 5 closed (#939)
#901 detector 5 #900 (R8) — satisfied closed (#942)
#739 detector 5 closed (#934)
#740 detector 5 closed (#936)
#730 detector 5 open · §1 shipped (#933), §2/§5 remain
#701 detector 5 — · owned the R14 decision closed (#940)
#704 CLI 6 closed (#944)
#705 CLI 6 #704 (R7) — satisfied open · next
#728 key semantics 7 closed (#943)
#731 key semantics 7 #730 (R10) — satisfied, is_canonical shipped open
#727 encoders 7 #701's evidence type (R14) — delivered in review (#945)
#742 encoders 7 open
#703 encoders 7 open
#903 benchmarks 8 closed (#904)
#759 benchmarks 8 #903 (R12) — satisfied closed
#732 benchmarks 8 #759 (R6) — satisfied closed (#931)
#736 benchmarks 8 #759 (R6) — satisfied open
#675 docs site 9 decision open · decision outstanding
#694 docs site 9 #675 decision (R9) open
#860 introspection decision (R11) open · held
#935 detector decision: paragraph direction + a new dependency open · held (triaged in after this plan)
#905 CI hygiene first closed
#907 detector filed during 0.16.0 closed
#910 fold filed during 0.16.0 closed
#914 fold filed during 0.16.0 closed
#916 fold filed during 0.16.0 closed
#917 detector filed during 0.16.0 closed (#933)
#918 pipeline filed during 0.16.0 closed
#937 detector filed during 0.16.0; reverses #934's out-of-scope call closed (#941)

32 of 32 from the original denominator, plus the 6 that joined after it — 39 rows for a
milestone of 40, the 40th being this issue.

#935 has since been triaged into 0.16.0 (no surface returns display order — the
resolve_bidi decision split out of #740). It is a decision, the same shape as #860:
display order needs a paragraph direction with no safe default, and unicode-bidi is not in
the dependency tree. Indexed below; nothing in the milestone waits on it.


What 0.15.0 suggests watching for

Three patterns cost real rework last milestone and every one of them has a candidate here.

A gate anchored to the thing it watches. #806, #850 and #887 each shipped a check that could not fail. Cluster 9 is the exposure: #759 changes a harness whose output three pages quote, and a page-vs-harness gate keyed on prose passes as soon as the prose is reworded.

A census frozen before its denominator settled. R2 exists because this happened in 0.15.0. Cluster 1 has seven issues moving one table and cluster 2–3 publish counts from it.

A metric that rewards the bug. The 0.15.0 meta-benchmark scored three of disarm's own fixes as regressions — nonascii_folded cannot tell folding from deleting from naming, and visible_to_coverage rewards a longer gap list. #759 is the natural place to fix the first; the second belongs with #884.


Status

Re-read at 2ad40f7. The milestone holds 39 issues, 13 closed — the 32 in the
denominator above plus 6 that arrived and closed during 0.16.0, all now in the index.

Shipped from the denominator (7).

issue landed as what it changes for the rest of the plan
#905 The allocation gate is deterministic, so a red build can be believed. This was the "first" wave and it did go first.
#833 R1 satisfied. The fold sees the source code point, so #646, #650 and #815 measure against a settled reachable set.
#903 #904 R12 satisfied.
#759 R6 satisfied. #732, #736 and #816 are unblocked.
#732 #931 The corpus page.
#815 #932 The Latin-shape census — 299 rows, and now the subject of R13.
#739 #934 The deletion class, detected and named.

In review. #740 (#936) — strip_bidi keeps logical order, named in THREAT_MODEL.md,
the strip_bidi docstring and docs/limitations.md, with the reversed construction pinned
as an asserted negative in the attack corpus.

Partly shipped. #730's §1 landed in #933: is_canonical exists, so R10 is satisfied
and #731 has the predicate its map was to point at. §2 (first_noncanonical) and §5 (the
five bindings) keep #730 open.

What the first seven shipping items taught

R2 is the constraint that needs a wave order, not just a statement. Wave 2 put a census
and three table changes in one wave and said nothing about their order, so the census
shipped first. R13 above is the repair. The lesson generalises: a constraint of the form
"X before Y" is only enforced if X and Y are in different waves. Three other pairs in
this plan sit in one wave — #646/#650 (R3), #900/#901 (R8), #894/#896 (R4, deliberately
one PR) — and only R4 is safe there, because "one pull request" is a stronger statement
than "one wave".

The 0.15.0 watch-list earned its place. A gate anchored to the thing it watches was
named as cluster 9's exposure and turned up twice in cluster 4 instead: #739's corpus
addition needed a negative-assertion mechanism because RECOVERS could only express
positive recovery, and #740's found that tests/test_attack_corpus.py was generating the
one bidi construction that strip and resolve agree on — a green 10/10 measuring the
recoverable half of the class. Both were fixed with an independently written render model,
so the premise is checked by something other than the code under test.

Docstring surfaces are not interchangeable. #740 put its note on the PyO3 shim, which
help() and mkdocstrings never show, because python/disarm/_presets.py defines a
wrapper whose docstring shadows it — and the two open with the same three lines, so a
prefix comparison cannot tell them apart. Cluster 3 (#894, #896) touches five bindings and
is the next place this can happen.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions