You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Status, re-read at 091b477a. 12 of the 32 have shipped (#732, #739, #759, #815, #833, #903, #905, #701, #937, #901, #704, #728) and #727 is in review (#945). Six issues
joined the milestone after this plan and have already closed (#907, #910, #914, #916, #917, #918); they are in the index, because an issue absent from it is a planning bug and
that rule applies to arrivals as well as to the original denominator. R1, R6, R7, R10,
R12 and R14 are satisfied. R2 was violated once and the repair is R13. Details in Status at the bottom.
Milestone 0.16.0 carries 32 open issues, read at 8599313 — the 0.15.0 release commit. Six of them (#815, #816, #860, #884, #894, #896) were deferred out of 0.15.0 with a reason recorded on each; the rest predate that milestone or were filed during it.
Read end to end, they are not 30 independent changes. They cluster into ten subsystems, and the largest — the confusable fold and its generator — has seven issues whose measurements all move if one of them lands first.
Three things this issue fixes:
Containment — which issues share a generated artifact, a census denominator, or a false-positive argument, and should therefore land together.
Order — which changes make later work cheaper, and which create rework if they land late.
Coverage — every one of the 30 appears in the index at the bottom. An issue absent from it is a planning bug.
This issue is not implemented by a pull request. It closes when the milestone ships.
#905 was found while implementing #903 — the allocation gate is stochastic: a
process-wide counter with a margin of one failed #904, which changes no Rust, and passed
on re-run of the same commit. It belongs with the CI-hygiene work rather than any cluster
below, and it goes early: while it stands, every red build has to be re-run before it can
be believed.
#903 was filed after this plan and is already implemented (#904). It is the hardcoded-transform half of #759, so it gates it — see R12.
A caveat on the evidence. I read the full body of #650, #675, #694 and #833, and for the rest the title, the scope section, and the issues each cross-references. The clustering rests on shared files and shared denominators, which are visible from that. The four constraints marked (inferred) below rest on the cross-references rather than on a claim the issue makes outright — check those before relying on them.
Where the rework comes from
Twelve ordering constraints, each stated as the rework it prevents.
#833 is the general form of #245: the presets run NFKC before the fold, so the fold never sees the source code point, and 68 Latin / 8 Cyrillic rows are unreachable. #650's entire cost argument is where in a preset the class runs, and #815's census counts what reaches ASCII. Move the reachable set after either is measured and both are re-measured.
R2
Every table change before every frozen census.
#833, #848, #836, #815 and #738 all regenerate or extend confusables_*.tsv. #884, #816, #736 and #732 all publish counts derived from it. This is #762's R3 and it cost real rework there. Violated: #815 shipped its census while #738 is still open — see R13.
Both are "Python-only, owed to the Rust API and five bindings", both need the same parity-matrix rows, and both hit the same C-ABI question: a result object that owns a String needs either a free function or a caller-supplied buffer. Split them and that decision is made twice, with two chances to differ.
#896 threads digit_policy through the Rust API and five bindings. #646 changes which policies exist (the capital-I family has none). Plumb first and the plumbing is redone.
#759 changes what benchmarks/adversarial_eval measures — it adds a clean-text cost metric and unhardcodes strip_obfuscation. All three of the others publish numbers that harness produces.
#705 is SARIF output and stable fingerprints for CI adoption. #704 is the CLI being able to open a file at all. SARIF for a tool that cannot read the repository is untestable.
#901 is the same primitive-layer confusion seen from the other side: #900 is find_confusables reporting whole non-Latin words, #901 is bilingual text being indistinguishable from a spoof. Fix the reporting first, then decide what the remaining ambiguity means.
#675 moves every URL on the site — /api/transforms.html becomes /latest/api/transforms.html. #694 corrects the rel=canonical tag and 74 sitemap entries. Build #694 first and every one of those 74 is rewritten by #675. #675 also questions its own premise ("whether anybody is pinned to an old release"), so the decision may be no — in which case #694 proceeds standalone. Either way the decision comes first.
#731 asks for a map for derived deterministic identifiers and says no key builder is safe for one and the variance direction has no predicate. #730 builds a verification-path predicate (is_canonical). The map wants the predicate to point at.
#903 makes the harness's transform a parameter; #759 asks for a clean-text cost metric and for the transform to stop being hardcoded. Build #759 first and half of it is rewritten. Shipped in #904.
R2 named this class and wave 2 then put a census and three table changes in one wave with no order between them, so it happened. #815 shipped tests/fixtures/latin_shape_exposure.tsv with a hard assert len(_fixture_rows()) == 299 plus a per-block count for each of seven blocks. #738's first tranche is 793 novel Latin-target pairs — "X → Latin, the direction disarm folds", which is exactly the population that gives a listed code point a path. Any overlap removes rows, and the gate's own contract is "a code point LEAVING the file is a fix". The repair is cheap and must be deliberate: land #738, regenerate with scripts/gen_latin_shape_exposure.py, move the 299, re-check seven block counts. #836 and #848 do not have this problem — #836 keys on confusables_contractions.tsv and the census walks single code points, and #848 is Arabic-on-Arabic with no Latin target, so neither can move a Latin-shape census whatever the order.
R14
#701 and #727 settle one decision, and it is #701's.(delivered — #940 built the type, #945 adds percent as the fourth scheme on it)
Both want a decode-as-evidence surface rather than a substituting decoder, and #727 says so outright: its preferred shape is "a decode-for-inspection primitive that returns what the escapes spelled rather than a substituted string — the same argument #701 makes for the three smuggling carriers, and for the same reason". #701 proposes decode_smuggled(text) -> Vec<Payload> over three schemes; percent-escapes would be a fourth on the same type. Build them apart and the evidence type — its span convention, its confidence/None contract, its error answers for malformed input — is designed twice, with two chances to differ. That is R4's argument exactly. #701 owns it because it carries three schemes and the arithmetic already exists at src/invisibles.rs:116; #727 extends it. This plan has them in different clusters (4 and 7) and different waves (5 and 7) with nothing linking them.
It changes list_profiles()'s return type, which is in tests/test_api_stability.py's pinned surface and in five bindings. That is a ### Changed (breaking) entry and a call for @raeq, not something to assume — the same shape as 0.15.0's R17 (#776).
The ten clusters
1. The confusable fold and its generator — seven issues, one artifact
Every one of these regenerates or extends confusables_*.tsv, or measures what that table reaches. #833 goes first because it changes what the fold can see at all.
unmapped_confusables accepts four of the 57 scripts (the issue says two; #792/#849 added Arabic and Hebrew after it was written). Its denominator is the table, so it follows cluster 1. 0.15.0's #888 fixed the error message and explicitly left this open.
The CLI cannot open a file, so the one API built for scanning cannot be pointed at anything. SARIF and line-independent fingerprints follow, so a repository can adopt the check before it is clean.
#903 is done (#904): the harness now scores a named surface rather than a hardcoded strip_obfuscation, with the default unchanged. #759 is what remains of the harness work — a clean-text cost metric — and the other three publish its numbers. R6 and R12 order them. #816 also depends on cluster 1 — it scores 38 non-standard Unicode sets, and four of its classes fold nowhere, which is #815's territory.
The allocation gate blocks merges on a counter any thread in the process can move, with a
margin of one. Same shape as the four Tier-3 draws #898 removed. Independent of every
other cluster, and worth doing first for the same reason #762 put #46 ahead of its
milestone: a lane that cannot be trusted taxes everything routed through it.
Independent, and every red build is unreliable until it lands.
done
Waves 4 through 7 are independent of one another and of waves 0–3; they are ordered here only for readability. Waves 0 → 1 → 2 → 3 is the one chain where order is load-bearing.
Index
Every issue in the denominator, and where it sits.
32 of 32 from the original denominator, plus the 6 that joined after it — 39 rows for a
milestone of 40, the 40th being this issue.
#935 has since been triaged into 0.16.0 (no surface returns display order — the resolve_bidi decision split out of #740). It is a decision, the same shape as #860:
display order needs a paragraph direction with no safe default, and unicode-bidi is not in
the dependency tree. Indexed below; nothing in the milestone waits on it.
What 0.15.0 suggests watching for
Three patterns cost real rework last milestone and every one of them has a candidate here.
A gate anchored to the thing it watches.#806, #850 and #887 each shipped a check that could not fail. Cluster 9 is the exposure: #759 changes a harness whose output three pages quote, and a page-vs-harness gate keyed on prose passes as soon as the prose is reworded.
A census frozen before its denominator settled. R2 exists because this happened in 0.15.0. Cluster 1 has seven issues moving one table and cluster 2–3 publish counts from it.
A metric that rewards the bug. The 0.15.0 meta-benchmark scored three of disarm's own fixes as regressions — nonascii_folded cannot tell folding from deleting from naming, and visible_to_coverage rewards a longer gap list. #759 is the natural place to fix the first; the second belongs with #884.
Status
Re-read at 2ad40f7. The milestone holds 39 issues, 13 closed — the 32 in the
denominator above plus 6 that arrived and closed during 0.16.0, all now in the index.
In review.#740 (#936) — strip_bidi keeps logical order, named in THREAT_MODEL.md,
the strip_bidi docstring and docs/limitations.md, with the reversed construction pinned
as an asserted negative in the attack corpus.
Partly shipped.#730's §1 landed in #933: is_canonical exists, so R10 is satisfied
and #731 has the predicate its map was to point at. §2 (first_noncanonical) and §5 (the
five bindings) keep #730 open.
What the first seven shipping items taught
R2 is the constraint that needs a wave order, not just a statement. Wave 2 put a census
and three table changes in one wave and said nothing about their order, so the census
shipped first. R13 above is the repair. The lesson generalises: a constraint of the form
"X before Y" is only enforced if X and Y are in different waves. Three other pairs in
this plan sit in one wave — #646/#650 (R3), #900/#901 (R8), #894/#896 (R4, deliberately
one PR) — and only R4 is safe there, because "one pull request" is a stronger statement
than "one wave".
The 0.15.0 watch-list earned its place.A gate anchored to the thing it watches was
named as cluster 9's exposure and turned up twice in cluster 4 instead: #739's corpus
addition needed a negative-assertion mechanism because RECOVERS could only express
positive recovery, and #740's found that tests/test_attack_corpus.py was generating the
one bidi construction that strip and resolve agree on — a green 10/10 measuring the
recoverable half of the class. Both were fixed with an independently written render model,
so the premise is checked by something other than the code under test.
Docstring surfaces are not interchangeable.#740 put its note on the PyO3 shim, which help() and mkdocstrings never show, because python/disarm/_presets.py defines a
wrapper whose docstring shadows it — and the two open with the same three lines, so a
prefix comparison cannot tell them apart. Cluster 3 (#894, #896) touches five bindings and
is the next place this can happen.
Summary
Milestone 0.16.0 carries 32 open issues, read at
8599313— the 0.15.0 release commit. Six of them (#815, #816, #860, #884, #894, #896) were deferred out of 0.15.0 with a reason recorded on each; the rest predate that milestone or were filed during it.Read end to end, they are not 30 independent changes. They cluster into ten subsystems, and the largest — the confusable fold and its generator — has seven issues whose measurements all move if one of them lands first.
Three things this issue fixes:
This issue is not implemented by a pull request. It closes when the milestone ships.
The denominator:
#905 was found while implementing #903 — the allocation gate is stochastic: a
process-wide counter with a margin of one failed #904, which changes no Rust, and passed
on re-run of the same commit. It belongs with the CI-hygiene work rather than any cluster
below, and it goes early: while it stands, every red build has to be re-run before it can
be believed.
#903 was filed after this plan and is already implemented (#904). It is the
hardcoded-transform half of #759, so it gates it — see R12.
A caveat on the evidence. I read the full body of #650, #675, #694 and #833, and for the rest the title, the scope section, and the issues each cross-references. The clustering rests on shared files and shared denominators, which are visible from that. The four constraints marked (inferred) below rest on the cross-references rather than on a claim the issue makes outright — check those before relying on them.
Where the rework comes from
Twelve ordering constraints, each stated as the rework it prevents.
confusables_*.tsv. #884, #816, #736 and #732 all publish counts derived from it. This is #762's R3 and it cost real rework there. Violated: #815 shipped its census while #738 is still open — see R13.Stringneeds either a free function or a caller-supplied buffer. Split them and that decision is made twice, with two chances to differ.digit_policythrough the Rust API and five bindings. #646 changes which policies exist (the capital-I family has none). Plumb first and the plumbing is redone.benchmarks/adversarial_evalmeasures — it adds a clean-text cost metric and unhardcodesstrip_obfuscation. All three of the others publish numbers that harness produces.find_confusablesreporting whole non-Latin words, #901 is bilingual text being indistinguishable from a spoof. Fix the reporting first, then decide what the remaining ambiguity means./api/transforms.htmlbecomes/latest/api/transforms.html. #694 corrects therel=canonicaltag and 74 sitemap entries. Build #694 first and every one of those 74 is rewritten by #675. #675 also questions its own premise ("whether anybody is pinned to an old release"), so the decision may be no — in which case #694 proceeds standalone. Either way the decision comes first.is_canonical). The map wants the predicate to point at.tests/fixtures/latin_shape_exposure.tsvwith a hardassert len(_fixture_rows()) == 299plus a per-block count for each of seven blocks. #738's first tranche is 793 novel Latin-target pairs — "X → Latin, the direction disarm folds", which is exactly the population that gives a listed code point a path. Any overlap removes rows, and the gate's own contract is "a code point LEAVING the file is a fix". The repair is cheap and must be deliberate: land #738, regenerate withscripts/gen_latin_shape_exposure.py, move the 299, re-check seven block counts. #836 and #848 do not have this problem — #836 keys onconfusables_contractions.tsvand the census walks single code points, and #848 is Arabic-on-Arabic with no Latin target, so neither can move a Latin-shape census whatever the order.decode_smuggled(text) -> Vec<Payload>over three schemes; percent-escapes would be a fourth on the same type. Build them apart and the evidence type — its span convention, its confidence/None contract, its error answers for malformed input — is designed twice, with two chances to differ. That is R4's argument exactly. #701 owns it because it carries three schemes and the arithmetic already exists atsrc/invisibles.rs:116; #727 extends it. This plan has them in different clusters (4 and 7) and different waves (5 and 7) with nothing linking them.list_profiles()'s return type, which is intests/test_api_stability.py's pinned surface and in five bindings. That is a### Changed (breaking)entry and a call for @raeq, not something to assume — the same shape as 0.15.0's R17 (#776).The ten clusters
1. The confusable fold and its generator — seven issues, one artifact
#833 #646 #650 #815 #836 #848 #738
Every one of these regenerates or extends
confusables_*.tsv, or measures what that table reaches. #833 goes first because it changes what the fold can see at all.llm_guardrail("ϲecure")returns"oecure"#833 — NFKC before the fold; 68 Latin / 8 Cyrillic rows unreachable,llm_guardrail("ϲecure")→"σecure". The general form of Confusable coverage: 60 single-codepoint TR39 sources not neutralized by the full pipeline (XMR v2 per-source probe, 0.6.3) #245, which closed with one hand-written override.digit_policyreaches one function; the capital-I family has no policy. The I/l/1 and O/0 equivalence: cheap for letters, ruinous for identifiers, and step order decides which you get #650 answers No profile can express digit_policy: the eight pipelines pick a side of the trade for you #646's open question and turns on step order.Three numbers are in circulation and all three were true when written: the title says
408, this plan said 385, and the fixture holds 299, because fix: single-letter Latin small capitals fold to their letter (#815) #915, feat: the 54 negative enclosed letters fold to their letter (#815) #920 and fix: the detector went silent exactly when the disguise was complete (#815) #929
gave the difference a path in the interim. Quote 299, or regenerate.
2. Coverage introspection — #884
unmapped_confusablesaccepts four of the 57 scripts (the issue says two; #792/#849 added Arabic and Hebrew after it was written). Its denominator is the table, so it follows cluster 1. 0.15.0's #888 fixed the error message and explicitly left this open.3. Binding parity — #894 #896
Both are 0.15.0 deferrals, both Python-only today, both owed to the Rust API and five bindings. One pull request (R4), and #646 first (R5).
4. The detector — #900 #901 #739 #740 #730 #701
find_confusablesreports whole non-Latin words as Latin confusables #900 → Bilingual text is indistinguishable from a spoof at the primitive layer; onlyhas_anomaliestells them apart #901 — whole non-Latin words reported as Latin confusables, then bilingual text vs a spoof.paypalcanonicalizes topXaXyXpXaXlX, and the corpus that names the taxonomy has no generator for it #739 — the deletion class survives every preset:pXaXyXpXaXlXrenderspaypal.<RLO>lapyap<PDF>renderspaypaland canonicalizes tolapyap— and nothing in the repo names the distinction #740 —strip_bidikeeps logical order, so<RLO>lapyap<PDF>renderspaypaland canonicalizes tolapyap. The deletion class survives every preset and profile: a string that renderspaypalcanonicalizes topXaXyXpXaXlX, and the corpus that names the taxonomy has no generator for it #739 and strip_bidi keeps the logical order, so<RLO>lapyap<PDF>renderspaypaland canonicalizes tolapyap— and nothing in the repo names the distinction #740 are the same argument shape — a class that survives everything and is named nowhere — and share a page.has_anomaliescalls 143,804 code points clean thatcanonicalizerewrites, and there is nois_canonical#730 — nois_canonical:has_anomaliescalls 143,804 code points clean thatcanonicalizerewrites.5. CLI and CI adoption — #704 → #705
The CLI cannot open a file, so the one API built for scanning cannot be pointed at anything. SARIF and line-independent fingerprints follow, so a repository can adopt the check before it is clean.
6. Key semantics — #728 #731
"", so absence and a stripped value share one key —sanitize_filenameis the only surface that reserves a sentinel #728 — every preset maps non-empty input to"", so absence and a stripped value share one key;sanitize_filenameis the only surface that reserves a sentinel.7. Encoders and decoders — #727 #742 #703
percent_encodeand no decoder: every detector reports clean on a percent-encoded value #727 —percent_encodeships with no decoder, so every detector reports clean on a percent-encoded value.canonicalizeblocks it turns on a TR39 ASCII row: 11 of 26 delimiters survive #742 — a chat-template delimiter in user text has no encoder; 11 of 26 survive, and whethercanonicalizeblocks one turns on a TR39 ASCII row.canonicalizeandtransliteratein nearly complementary halves, and neither covers it #703 — the dash family is folded bycanonicalizeandtransliteratein nearly complementary halves, and neither covers it.8. Docs site — #675 #694
#675 versions the site; #694 fixes canonicals and the sitemap. R9 orders them, and #675 may be declined.
9. Benchmarks and validation pages — #903 #759 #732 #736 #816
#903 is done (#904): the harness now scores a named surface rather than a hardcoded
strip_obfuscation, with the default unchanged. #759 is what remains of the harness work — a clean-text cost metric — and the other three publish its numbers. R6 and R12 order them. #816 also depends on cluster 1 — it scores 38 non-standard Unicode sets, and four of its classes fold nowhere, which is #815's territory.10. CI hygiene — #905
The allocation gate blocks merges on a counter any thread in the process can move, with a
margin of one. Same shape as the four Tier-3 draws #898 removed. Independent of every
other cluster, and worth doing first for the same reason #762 put #46 ahead of its
milestone: a lane that cannot be trusted taxes everything routed through it.
11. Surface introspection — #860
Blocked on a decision (R11).
Suggested waves
#833#815, #836, #848, #738#900→#901,#739,#740, #730,#701#704→ #705#728, #731, #727, #742, #703#903→#759→#732, #736#905Waves 4 through 7 are independent of one another and of waves 0–3; they are ordered here only for readability. Waves 0 → 1 → 2 → 3 is the one chain where order is load-bearing.
Index
Every issue in the denominator, and where it sits.
is_canonicalshipped32 of 32 from the original denominator, plus the 6 that joined after it — 39 rows for a
milestone of 40, the 40th being this issue.
#935 has since been triaged into 0.16.0 (no surface returns display order — the
resolve_bididecision split out of #740). It is a decision, the same shape as #860:display order needs a paragraph direction with no safe default, and
unicode-bidiis not inthe dependency tree. Indexed below; nothing in the milestone waits on it.
What 0.15.0 suggests watching for
Three patterns cost real rework last milestone and every one of them has a candidate here.
A gate anchored to the thing it watches. #806, #850 and #887 each shipped a check that could not fail. Cluster 9 is the exposure: #759 changes a harness whose output three pages quote, and a page-vs-harness gate keyed on prose passes as soon as the prose is reworded.
A census frozen before its denominator settled. R2 exists because this happened in 0.15.0. Cluster 1 has seven issues moving one table and cluster 2–3 publish counts from it.
A metric that rewards the bug. The 0.15.0 meta-benchmark scored three of disarm's own fixes as regressions —
nonascii_foldedcannot tell folding from deleting from naming, andvisible_to_coveragerewards a longer gap list. #759 is the natural place to fix the first; the second belongs with #884.Status
Re-read at
2ad40f7. The milestone holds 39 issues, 13 closed — the 32 in thedenominator above plus 6 that arrived and closed during 0.16.0, all now in the index.
Shipped from the denominator (7).
In review. #740 (#936) —
strip_bidikeeps logical order, named inTHREAT_MODEL.md,the
strip_bididocstring anddocs/limitations.md, with the reversed construction pinnedas an asserted negative in the attack corpus.
Partly shipped. #730's §1 landed in #933:
is_canonicalexists, so R10 is satisfiedand #731 has the predicate its map was to point at. §2 (
first_noncanonical) and §5 (thefive bindings) keep #730 open.
What the first seven shipping items taught
R2 is the constraint that needs a wave order, not just a statement. Wave 2 put a census
and three table changes in one wave and said nothing about their order, so the census
shipped first. R13 above is the repair. The lesson generalises: a constraint of the form
"X before Y" is only enforced if X and Y are in different waves. Three other pairs in
this plan sit in one wave — #646/#650 (R3), #900/#901 (R8), #894/#896 (R4, deliberately
one PR) — and only R4 is safe there, because "one pull request" is a stronger statement
than "one wave".
The 0.15.0 watch-list earned its place. A gate anchored to the thing it watches was
named as cluster 9's exposure and turned up twice in cluster 4 instead: #739's corpus
addition needed a negative-assertion mechanism because
RECOVERScould only expresspositive recovery, and #740's found that
tests/test_attack_corpus.pywas generating theone bidi construction that strip and resolve agree on — a green 10/10 measuring the
recoverable half of the class. Both were fixed with an independently written render model,
so the premise is checked by something other than the code under test.
Docstring surfaces are not interchangeable. #740 put its note on the PyO3 shim, which
help()and mkdocstrings never show, becausepython/disarm/_presets.pydefines awrapper whose docstring shadows it — and the two open with the same three lines, so a
prefix comparison cannot tell them apart. Cluster 3 (#894, #896) touches five bindings and
is the next place this can happen.