Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions docs/adr/003-shared-oxford-spelling-base.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,25 @@ Refreshes validate content before atomic replacement, preserve a valid cache
when its authority is not newer, and allow explicit offline reuse. Generated
configuration remains tracked so review and CI can detect drift.

Treat source identity as part of cache validity. Conditional validators, stale
fallback, and HTTP `304 Not Modified` reuse require metadata for the exact
requested local path or HTTPS URL. Refresh decisions use bounded structured
logging fields and never expose source or repository paths.

Split the implementation by policy boundary while retaining
`typos_rollout.py` as the stable facade. Dedicated modules own regular
expression policy, cache persistence, HTTP refresh, deterministic rendering,
phrase checking, and harvesting. Every source module remains below 400 lines.

Validate ignore expressions before repository scanning. Reject malformed
expressions, backreferences, and compounded repetition, including Python's
`{,n}` form, while accepting repetitions separated by ordinary atoms. Exercise
the complete brace-quantifier grammar and safe separators with Hypothesis.

Phrase checking and harvesting skip only non-UTF-8 tracked content. Other file
read failures propagate after a bounded structured diagnostic, preventing a
partial scan from being reported as successful.

## Consequences

**Positives:**
Expand All @@ -48,12 +67,21 @@ configuration remains tracked so review and CI can detect drift.
test make the generated boundary reviewable.
- Exact phrase corrections remain shared and enforceable despite Typos token
boundaries.
- Source-scoped cache decisions prevent validators or stale content crossing
authority boundaries.
- Bounded regex validation limits backtracking risk before repository scans.
- Module ownership and generated property tests make the policy easier to
review without weakening the facade contract.
- Structured diagnostics explain refresh and read decisions without disclosing
unbounded source or path values.

**Costs and trade-offs:**

- Consumers carry a small generator and tracked generated configuration.
- Consumers run one additional tracked-text pass for the small curated phrase
table.
- Non-UTF-8 tracked files are intentionally omitted, while all other read
failures now fail the spelling operation.
- A fresh consumer needs its shared source once before offline generation.
- Curators must inspect harvested context because suffix matches alone include
genuine `-ise` words and identifiers that are not Oxford stems.
38 changes: 37 additions & 1 deletion docs/developers-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -305,18 +305,54 @@ reporting a path, line, column and canonical replacement. The companion
unavailable, a valid existing cache remains usable with `--offline`; generation
fails rather than silently inventing an empty base when no cache exists.

Freshness metadata is source-scoped. Local modification times, HTTP validators,
stale-cache fallback, and `304 Not Modified` reuse apply only when the saved
source identity exactly matches the requested authority. A missing or different
identity forces refresh or propagates the authority failure. Standard-library
logging records these decisions with bounded `operation`, `source_kind`,
`error_class` and `decision` fields. Never add an authority URL, repository
path, response body or exception message to these records.

Refresh callers bind the metadata path, offline policy and optional test opener
in an immutable `RefreshOptions` value. The helper owns the private local and
remote request records that coordinate freshness and persistence; consumers
should compose the public options value rather than reuse those infrastructure
details.

The `typos_rollout.py` facade preserves the public CLI and import surface.
Sibling modules own one policy boundary each:

- `typos_rollout_policy.py` validates schemas, local exceptions, and bounded
regular expressions.
- `typos_rollout_cache.py` owns cache records, validator metadata, and atomic
persistence.
- `typos_rollout_http.py` coordinates source-scoped local and HTTPS refreshes.
- `typos_rollout_render.py` expands Oxford stems and renders deterministic TOML.
- `typos_rollout_check.py` enforces curated exact phrase corrections.
- `typos_rollout_harvest.py` gathers contextual Oxford-form evidence.

Keep each source module below 400 lines and route new behaviour to its owning
boundary rather than expanding the facade. Regular expression validation
rejects malformed patterns, backreferences, and compounded repetition. The
scanner recognizes all Python brace forms, including `{n}`, `{n,}`, `{n,m}`
and `{,n}`. It permits repetitions separated by unquantified atoms. Example
regressions pin known hazards, while Hypothesis properties generate every brace
shape and varied safe separators.

Phrase checking and harvesting read only Git-tracked files. A
`UnicodeDecodeError` identifies non-UTF-8 content and is skipped with a bounded
informational record. Every `OSError`, including permission and disappearance
failures, is logged without a path and propagated so the gate fails closed.
Caplog tests assert structured record fields rather than rendered log text.

Run `make spelling` after dictionary or generator changes. The target generates
the committed config from the local authoritative base, checks exact phrase
policy, and runs the version of `typos` pinned by `TYPOS_VERSION`. The full
`make ci` sequence includes this gate. Tests assert byte-for-byte config drift,
TOML validity, cache freshness, offline recovery, exact phrase boundaries and
real-binary Oxford behaviour.
real-binary Oxford behaviour. Property tests exercise the regular expression
repetition grammar, and logging tests pin bounded diagnostics for source-scope
decisions and tracked-file read failures.

The initial shared stem set was curated on 10 July 2026 from both correct
Oxford forms and incorrect plain-British forms across the 96 non-empty,
Expand Down
15 changes: 15 additions & 0 deletions docs/users-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,14 @@ the shared base into ignored `.typos-oxendict-base.toml` and records freshness
metadata in `.typos-oxendict-base.json`. A valid cache supports offline runs;
the tracked `typos.toml` remains deterministic and reviewable.

Cache metadata is scoped to the exact authority that supplied it. A stale
cache or HTTP `304 Not Modified` response is accepted only when the metadata
names the requested source and the cached dictionary still validates. Switching
local paths or HTTPS URLs therefore forces a refresh without reusing another
authority's validators. Refresh decisions are available through standard
Python logging with bounded operation, source-kind, error-class and decision
fields; logs do not contain authority URLs or local paths.

The rollout CLI exposes the underlying operations:

```bash
Expand All @@ -80,6 +88,13 @@ policy. `harvest` emits JSON Lines evidence for Oxford `-ize` and plain-British
shared base; product names, quoted upstream terms, and deliberate fixtures
belong in a consumer's `typos.local.toml`.

Ignore expressions are validated before scanning. Malformed expressions,
backreferences, and nested or adjacent repetitions are rejected, including
Python's `{,n}` upper-bound form; separated bounded repetitions remain valid.
Phrase checking and harvesting skip tracked files that are not UTF-8. Other
tracked-file read failures stop the operation and emit a bounded diagnostic,
so an incomplete repository scan cannot appear successful.

## Common settings

### `RUST_ENTRYPOINT_PHASE`
Expand Down
Loading