Emit byte-safe JSON for non-UTF-8 paths and content (lone surrogates break strict parsers)

[wkentaro]

Context

Surfaced during review of #11 / PR #32 (decode git output with surrogateescape). That fix is correct and tested; this is a separate downstream gap in the --json output, not a regression in #32.

Problem

With surrogateescape decoding, a non-UTF-8 byte in a path or diff line becomes a lone surrogate in the Python str (e.g. 0xe9 -> \udce9). The --json commands serialize these with json.dumps(..., ensure_ascii=True) (the default), which emits the lone surrogate as a \udcXX escape:

$ git-hunk list --json
[{"file": "pass\udce9.txt", ...}]

This is a lone-surrogate JSON string, which downstream consumers handle inconsistently:

• Python's own json.loads round-trips it.
• jq (1.8.x) parses it but replaces it with U+FFFD — the original byte is silently lost (lossy).
• Strict parsers (Go encoding/json, Rust serde_json, older jq) reject the document outright.

So list --json on a repo containing a non-UTF-8 filename or diff content either corrupts the value or breaks the consumer. Affected sinks: the three json.dumps(...) calls in git_hunk/_cli.py (list --json, and the per-command JSON outputs).

Proposal

Pick a byte-safe representation for non-UTF-8 text in JSON output (maintainer's call):

1. Replace undecodable bytes with U+FFFD before serializing (valid, but lossy).
2. Emit the raw bytes in a separate, explicitly-encoded field (e.g. base64 / percent-escaped), keeping the human field best-effort.
3. Document that --json assumes UTF-8 paths and fail loudly with a clear error otherwise.

Acceptance criteria

• list --json on a repo with a non-UTF-8 path or diff line produces output that strict JSON parsers (jq, Go, Rust) accept.
• The chosen representation is covered by a test that asserts the output parses cleanly.

Scope notes

• Distinct from Document and version the --json output schema #23 (document and version the --json schema): that is about the schema's shape and versioning; this is about byte-safety of the encoding. The two can be addressed together but are not the same gap.
• Distinct from Decode git output with surrogateescape to avoid UnicodeDecodeError crash #11 / fix: handle non-UTF-8 bytes in git output without crashing #32 (the decode fix itself), which is already merged/correct.

[wkentaro]

This was generated by AI during triage.

Agent Brief

Category: bug
Summary: --json output emits lone surrogates for non-UTF-8 bytes; adopt ripgrep's {"text" | "bytes"} discriminated-union encoding so the output is strict-parser-safe and lossless.

Current behavior:
Git output is decoded with surrogateescape, so a non-UTF-8 byte in a path or diff line becomes a lone surrogate in the Python str (0xe9 → \udce9). The --json path serializes with json.dumps(..., ensure_ascii=True) (the default), which emits the surrogate as a literal \udce9 escape:

$ git-hunk list --json
... "diff": "@@ -1,2 +1,2 @@\n first line\n-l\udce9gacy content\n+..."

Confirmed by reproduction on a repo with a 0xe9 byte in a diff line:

• Python's json.loads round-trips it (the existing non_utf8 e2e test passes because it parses with Python).
• jq 1.8.1 parses but silently replaces it with U+FFFD — the original byte is lost (lossy corruption).
• Strict parsers (Go encoding/json, Rust serde_json) reject the document.

So list --json on a repo containing a non-UTF-8 path or diff content either corrupts the value or breaks the consumer. The human/Rich render path is already protected (it backslash-escapes lone surrogates before writing); only the JSON sinks are exposed.

Desired behavior:
Adopt the encoding ripgrep ships for the identical problem (a CLI streaming JSON over paths and line contents that may not be UTF-8). Every field that carries arbitrary, git/source-derived bytes is emitted as a discriminated union, exactly one key:

• bytes are valid UTF-8 → {"text": "<string>"}
• bytes are not valid UTF-8 → {"bytes": "<standard base64 of the raw bytes>"}

Use the always-union form (option 2a): the field is always an object, even in the common all-UTF-8 case — never sometimes-a-string. This gives every consumer a single code path and matches ripgrep's design exactly. The raw bytes are recovered by re-encoding the original str with surrogateescape and, when that is not valid UTF-8, base64-encoding it.

The arbitrary-data fields in the per-hunk record are file, header, context_before, and diff. The internally-generated fields id (hex hash), status (enum), additions, and deletions (ints) are never non-UTF-8 and stay plain — matching how ripgrep wraps path/lines but leaves line_number/offsets plain.

This is a breaking shape change to the --json contract, so bump the schema version from 1 to 2.

Key interfaces:

• The per-hunk serialization (currently Hunk.to_dict()) — the four arbitrary-data fields change from plain strings to {"text"|"bytes": …} objects. Factor the per-value choice into a single helper (e.g. an encode_arbitrary(s: str) -> dict[str, str]) so the rule lives in one place and is reused by any future JSON sink (show --json, show --json with structured per-line hunk body #56).
• The JSON schema-version constant (currently JSON_SCHEMA_VERSION / the schema_version field in the list --json envelope) — increment to 2.
• The --json envelope from list keeps its {"schema_version", "hunks": [...]} shape; only the per-hunk field values change.
• The encoding must round-trip: text for any value that re-encodes to valid UTF-8; bytes = standard base64 (with padding) of s.encode(errors="surrogateescape") otherwise.

Acceptance criteria:

• list --json on a repo with a non-UTF-8 path or diff line produces a document that strict JSON parsers accept — assert this with a parser that rejects lone surrogates (e.g. re-encode the output to UTF-8, or json.loads then json.dumps(..., ensure_ascii=False).encode("utf-8") must not raise), not just Python's lenient json.loads.
• For non-UTF-8 content, the affected field is {"bytes": "<base64>"} and base64decode of it equals the original raw bytes (lossless round-trip).
• For all-UTF-8 content, the affected field is {"text": "<string>"} (always-union: the field is an object, not a bare string).
• id, status, additions, deletions remain plain (string/int), unwrapped.
• JSON_SCHEMA_VERSION is 2 and the envelope reflects it.
• The SKILL.md schema documentation (the list --json envelope description in git_hunk/skills/core/SKILL.md) is updated to describe schema_version: 2 and the text/bytes union, so agent consumers know the shape.
• Existing --json tests and any test helper that reads h["file"] / h["id"] from list --json are updated to the new field shape and pass.

Out of scope:

• Parse paths git always quotes (tab, newline, backslash, quote in filename) #30 (git-quoted paths with tab/newline/backslash/quote). That is an upstream parse gap — such paths are dropped in path extraction before they ever reach JSON serialization — so the encoding union does not address it. Independent issue.
• show --json with structured per-line hunk body #56 (show --json). show has no --json output today; do not add one here. The shared encode_arbitrary helper should be written so show --json with structured per-line hunk body #56 can reuse it, but adding the command is separate.
• Document and version the --json output schema #23 (documenting/versioning the --json schema as a whole). Bump the version number here; the broader schema-doc/versioning work is its own issue.
• Changing the human-readable (non---json) output in any way.
• Choosing percent-escaping or U+FFFD-replacement instead — the representation is decided: text/bytes union with base64.

[wkentaro]

This was generated by AI during triage.

Update: superseded by ADR 0001 (schema v2)

This issue is now part of the converged --json schema v2 — see docs/adr/0001-json-v2-hunk-schema.md (PR #66). The byte-safe encoding decision is locked there (ADR §7); implement against the ADR, not the standalone brief above. Deltas from that brief:

• Decision confirmed: ripgrep-style {text|bytes} discriminated union, always-union (2a), standard base64 for the bytes case. Recover raw bytes via s.encode(errors="surrogateescape").
• Field list changed. The v1 diff blob is removed from JSON (ADR §6), so it is no longer a union target. The union now applies to: file, context_before (now {text|bytes} | null), and lines[].content (the per-line body from show --json with structured per-line hunk body #56).
• Dependency: the lines[].content half requires show --json with structured per-line hunk body #56's lines[] to exist. The file / context_before half is independent.

Coordination: part of schema v2; lands with #28/#40/#50/#56 as one coordinated change (shared schema_version 1→2 bump, shared to_dict/parse_diff rewrite). Do not open a competing PR that edits to_dict in parallel.