M1: the catalog model and sync

[tamnd]

The catalog model and sync. No model calls.

The gate for everything downstream is byte fidelity. A writer that reformats is a writer that produces 548 files of churn on the first run and hides every real change afterwards.

Checklist

• catalog.py: read and write, segment_id, entry identity on (msgctxt, msgid), header writing
• Byte-fidelity test: read all 548 files, write them back unchanged, git diff empty
• memory.py: the store, the four source values, precedence
• tm rebuild from provenance comments, once M6 writes the comments there is something to rebuild from
• sync.py: pin to tamnd/python-docs-vi branch 3.15 at da475ff, manifests/upstream.yaml, counts written off the files
• sync --human: load the 1 435 human msgstr values into the memory as source=human
• sync diff report: added and orphaned, written to reports/sync.md
• stale.py: the three staleness causes, upstream text change, glossary bump, prompt change
• Tests over hand-written fixtures, property test on segment_id stability

Exit

pydocvi sync writes a pin recording 548 files, 87 008 entries and 1 711 382 English words. Round-trip of all 548 files is byte-identical. The memory holds 1 435 human segments. The real numbers go in a comment on this issue.

[tamnd]

Merged in #18.

The numbers

Measured against tamnd/python-docs-vi at da475ff9dbbd2eaaefcd7338ca2648b6b7f0c61b, and every one of them counted off the files rather than read from anywhere:

	expected	measured
files	548	548
entries	87 008	87 008
English words	1 711 382	1 711 382
msgid characters	12 526 506	12 526 506
human translations	1 435	1 435

Round-trip: 548 of 548 files byte-identical, in 1.2 seconds. The exit criterion is met.

Two corrections to the plan

polib cannot be used. The checklist said to read and write with it. All 60 files in the sample I tried came back different: it reorders the header keys and it breaks a line before the space where this corpus breaks after it. catalog.py handles the format directly instead, which is affordable because the corpus is uniform: no #: references, no msgctxt, no plurals, no obsolete entries, three escape sequences in total.

No wrapping rule reproduces the corpus, so unchanged fields keep their own lines. This is a Transifex export. Breaks land after spaces but also mid-token after a hyphen or a slash, and the effective width shifts depending on which. Brute-forced over all 175 112 fields: a space-only greedy fill at 79 misses 9.00 %, and the best textwrap variant, width 77 with hyphen breaking, reproduces 86.31 % exactly, with nothing doing better. So an entry carries the physical lines it was read from and only a field we actually rewrote goes through our own wrapper. Byte fidelity then holds by construction rather than by approximation, which is the property that has to survive the next ten milestones.

SyncDiff also lost two of the four categories the checklist asked for. removed was the same set as orphaned under a second name. changed cannot exist: the segment id hashes the string, so an edited msgid is one addition plus one orphan. That is the useful reading anyway, because the stored translation is of the old English and the new string wants re-queueing.

One bug worth recording

The property test on render_field failed on CI and passed locally. A carriage return, a vertical tab or a form feed in a value was written into the file as itself, and textwrap replaces all five whitespace control characters with a space when it rewraps, so the character was silently lost. The corpus never contains one and no fixture covered it. It passed locally because the hypothesis database had already settled on examples that avoided it, which is a good argument for the CI leg running against a cold database.

Fixed by escaping the full gettext set rather than only the three the corpus uses, with one table driving both directions so they cannot drift.

Left open

tm rebuild from provenance comments. Provenance currently lives as fields on a segment in the memory, not in catalog comments, so there is nothing to rebuild from. It belongs with the writer in M6.

State

ruff format, ruff check, mypy --strict and 94 tests at 96.36 % coverage, green on Linux, macOS and Windows on 3.14 and on the 3.15 prerelease leg. The corpus tests carry the corpus marker and skip when the checkout is absent, so CI stays honest on a runner that has no corpus while the gate still runs here.

Next: M2, segmentation and batching, tracked in #8.