vault-lint.py cannot see into a non-UTF-8 note, and reports it as clean
Problem
vault-lint.py reads every note as UTF-8 with errors="replace", in four places:
| Line |
Call |
What it reads |
| 115 |
p.read_text(encoding="utf-8", errors="replace") |
every note, for link checking |
| 217 |
src.open("r", encoding="utf-8", errors="replace", newline="") |
the --fix read |
| 276 |
moc.read_text(encoding="utf-8", errors="replace") |
MOC freshness |
| 288 |
manifest.read_text(encoding="utf-8", errors="replace") |
manifest coverage |
This is the same defect generate-backlinks.py carried until v0.2.10, and errors="replace" fails the same way errors="ignore" did. A UTF-16 document decoded as UTF-8 becomes NUL-interleaved text — #\x00 \x00T\x00i… — because NUL is a valid UTF-8 byte and is never replaced. Every [[wikilink]] in the file therefore fails to match.
The consequence is not a crash. It is a clean bill of health:
$ python vault-lint.py <vault-with-a-utf16-note> --check
vault-lint: 0 broken links, 0 orphans outside expected dirs — OK
The note in question contains a broken link. The linter reports zero because it cannot see any links at all. A tool whose entire job is to find broken links silently reports success on the one class of file where it is blind — and there is nothing in the output to suggest anything was skipped.
This matters more now than it did last week. As of v0.2.10 its sibling generate-backlinks.py does detect and report these files by name. A user who runs both tools gets a warning from one and silence from the other about the same file, which reads as a contradiction rather than a gap.
The latent half
Line 238 writes the --fix result back with encoding="utf-8" unconditionally. If a UTF-16 file ever reached that write, its NUL-interleaved decode would be written out as single-byte UTF-8 and the document would be destroyed.
This is currently unreachable. --fix only rewrites a note where it found a mechanically fixable broken link, and such a note parses to zero links — so the same bug that hides the file also keeps --fix away from it. The hazard is real but latent, and it is worth noting that fixing the read without fixing the write would make it reachable for the first time. Both belong in the same change.
Proposed solution
Reuse read_note_text() from generate-backlinks.py. vault-lint.py already loads that module — _load_backlinks_module() at line 47 — precisely so there is "one parsing truth"; today it only takes FENCE_RE off it. Taking the reader as well is the same decision applied one layer down.
_GB = _load_backlinks_module()
FENCE_RE = _GB.FENCE_RE
read_note_text = _GB.read_note_text # one reading truth, as well as one parsing truth
Then replace the four read sites. The --fix path needs more than a substitution, because it must also write back in the encoding it read rather than silently converting — read_note_text would need to report the encoding it chose, or the fix path needs to refuse non-UTF-8 files outright.
Refusing is probably right, and cheaper: the vault spec requires UTF-8, generate-backlinks.py now tells the user which files violate it, and --fix already refuses everything it cannot resolve unambiguously. "I will not rewrite a file whose encoding I would have to change" fits that existing posture exactly.
Design considerations
--fix must keep preserving line endings. v0.2.9 fixed a silent LF→CRLF conversion on Windows by using newline="" on both read and write. Any change to the read path has to keep that guarantee, and a byte-level reader interacts with it — this is the one place where care is needed rather than a mechanical swap.- Reporting should not be duplicated. If both tools print their own encoding report, a user running both sees it twice. Better for
vault-lint.py to surface the finding in its own report vocabulary (a link class, or a new counter) than to reuse report_encoding_issues() verbatim.
--check's exit code is the real deliverable. It gates CI and hooks. A vault containing an unreadable note should probably not exit 0, whatever the link count says.- Rejected: making
vault-lint.py self-sufficient. Duplicating the decode logic would give the two tools two definitions of "what a note says", which is the exact coupling _load_backlinks_module() was written to prevent.
Acceptance criteria
Related issues
Analysis
The defect was found while fixing the same bug in generate-backlinks.py for v0.2.10. Reproduction steps are recorded in tests/checklists/v0.2.10__Tool__generate-backlinks-encoding-detection.md, Section 5, which deliberately documents the wrong behaviour so it can be flipped to a verification when this is fixed.
vault-lint.pycannot see into a non-UTF-8 note, and reports it as cleanProblem
vault-lint.pyreads every note as UTF-8 witherrors="replace", in four places:p.read_text(encoding="utf-8", errors="replace")src.open("r", encoding="utf-8", errors="replace", newline="")--fixreadmoc.read_text(encoding="utf-8", errors="replace")manifest.read_text(encoding="utf-8", errors="replace")This is the same defect
generate-backlinks.pycarried until v0.2.10, anderrors="replace"fails the same wayerrors="ignore"did. A UTF-16 document decoded as UTF-8 becomes NUL-interleaved text —#\x00 \x00T\x00i…— because NUL is a valid UTF-8 byte and is never replaced. Every[[wikilink]]in the file therefore fails to match.The consequence is not a crash. It is a clean bill of health:
The note in question contains a broken link. The linter reports zero because it cannot see any links at all. A tool whose entire job is to find broken links silently reports success on the one class of file where it is blind — and there is nothing in the output to suggest anything was skipped.
This matters more now than it did last week. As of v0.2.10 its sibling
generate-backlinks.pydoes detect and report these files by name. A user who runs both tools gets a warning from one and silence from the other about the same file, which reads as a contradiction rather than a gap.The latent half
Line 238 writes the
--fixresult back withencoding="utf-8"unconditionally. If a UTF-16 file ever reached that write, its NUL-interleaved decode would be written out as single-byte UTF-8 and the document would be destroyed.This is currently unreachable.
--fixonly rewrites a note where it found a mechanically fixable broken link, and such a note parses to zero links — so the same bug that hides the file also keeps--fixaway from it. The hazard is real but latent, and it is worth noting that fixing the read without fixing the write would make it reachable for the first time. Both belong in the same change.Proposed solution
Reuse
read_note_text()fromgenerate-backlinks.py.vault-lint.pyalready loads that module —_load_backlinks_module()at line 47 — precisely so there is "one parsing truth"; today it only takesFENCE_REoff it. Taking the reader as well is the same decision applied one layer down.Then replace the four read sites. The
--fixpath needs more than a substitution, because it must also write back in the encoding it read rather than silently converting —read_note_textwould need to report the encoding it chose, or the fix path needs to refuse non-UTF-8 files outright.Refusing is probably right, and cheaper: the vault spec requires UTF-8,
generate-backlinks.pynow tells the user which files violate it, and--fixalready refuses everything it cannot resolve unambiguously. "I will not rewrite a file whose encoding I would have to change" fits that existing posture exactly.Design considerations
--fixmust keep preserving line endings. v0.2.9 fixed a silent LF→CRLF conversion on Windows by usingnewline=""on both read and write. Any change to the read path has to keep that guarantee, and a byte-level reader interacts with it — this is the one place where care is needed rather than a mechanical swap.vault-lint.pyto surface the finding in its own report vocabulary (a link class, or a new counter) than to reusereport_encoding_issues()verbatim.--check's exit code is the real deliverable. It gates CI and hooks. A vault containing an unreadable note should probably not exit 0, whatever the link count says.vault-lint.pyself-sufficient. Duplicating the decode logic would give the two tools two definitions of "what a note says", which is the exact coupling_load_backlinks_module()was written to prevent.Acceptance criteria
vault-lint.pydecodes notes viagenerate-backlinks.py'sread_note_text()rather than assuming UTF-8--check--fixrefuses to rewrite a note that is not UTF-8, and says why--fixstill preserves line endings exactly on the files it does rewrite (v0.2.9 regression guard)--fixrefusaltests/checklists/v0.2.10__Tool__generate-backlinks-encoding-detection.mdSection 5 is updated from "records the gap" to "verifies the fix"Related issues
pytest tests/only -- project-embedded suites never gate a push, and the hook reports success anyway #10 —pre-pushrunspytest tests/only; the new tests need to actually gate a push to be worth havingAnalysis
The defect was found while fixing the same bug in
generate-backlinks.pyfor v0.2.10. Reproduction steps are recorded intests/checklists/v0.2.10__Tool__generate-backlinks-encoding-detection.md, Section 5, which deliberately documents the wrong behaviour so it can be flipped to a verification when this is fixed.