Decode multibyte UTF-8 paths in quoted diff headers (_unquote_c_path mojibakes per-byte octal escapes)

[wkentaro]

This was generated by AI during PR processing.

Context

Surfaced while finalizing PR #68 (parse git's quoted, C-escaped diff headers, issue #30). PR #68 is correct for its scope — the ASCII chars git quotes regardless of core.quotePath (tab, newline, backslash, double-quote). This is a separate, out-of-scope gap on the decode path.

Problem

_unquote_c_path in git_hunk/_hunk.py decodes each octal escape one byte at a time:

chars.append(chr(int(path[i + 1 : i + 4], 8)))

With the default core.quotePath=true, git octal-escapes every byte >= 0x80, so a multibyte UTF-8 path is emitted as a sequence of per-byte octal escapes. Example: café.txt → header "a/caf\303\251.txt". The current code produces chr(0o303) + chr(0o251) = é (U+00C3 U+00A9), i.e. the UTF-8 bytes mis-decoded as Latin-1 — mojibake that does not match the real filename. The correct decode collects the escaped bytes and decodes them as a unit (UTF-8, ideally with the same surrogateescape strategy used elsewhere per #11/#32).

Repro (default git config):

$ git init && printf 'a\nb\n' > café.txt && git add . && git commit -m init
$ printf 'a\nB\n' > café.txt
$ git-hunk list --unstaged --json   # "file" comes back as "café.txt", not "café.txt"

Impact

list reports the wrong path for any non-ASCII filename, and subsequent stage/unstage/discard keyed off that path target the wrong (nonexistent) file.

Scope notes

• Distinct from Parse paths git always quotes (tab, newline, backslash, quote in filename) #30 / PR fix: parse git's quoted, C-escaped diff headers #68: that targets the always-quoted ASCII chars; this is the multibyte-octal axis triggered by core.quotePath.
• Distinct from Emit byte-safe JSON for non-UTF-8 paths and content (lone surrogates break strict parsers) #44: that is the emit side (byte-safe JSON output); this is the decode side in _unquote_c_path. They share the non-ASCII-path theme but are different code paths.

[wkentaro]

This was generated by AI during triage.

Triage: confirmed bug, but blocked on an unmerged PR — parking for a maintainer call.

Confirmed

Reproduced the per-byte mojibake. Git octal-escapes each byte of a multibyte UTF-8 name separately (café.txt → caf\303\251.txt with core.quotePath=true), and decoding each escape via chr(int(..., 8)) yields one Latin-1 code point per byte:

Input:    'caf\303\251.txt'
Got:      'café.txt'    # chr(0xC3) + chr(0xA9)
Expected: 'café.txt'

The fix is to accumulate contiguous octal-escaped bytes into a bytearray and decode the whole run as bytes(buf).decode("utf-8", errors="surrogateescape") — consistent with how run_git already handles git output.

Why this is parked, not ready-for-agent

_unquote_c_path does not exist on main — it is introduced by the still-open #68 (fix/30-quoted-paths). An AFK implementation agent works against main, where there is nothing to fix, so this can't be cleanly delegated yet. It's a sequencing call for you:

• Fold the fix into fix: parse git's quoted, C-escaped diff headers #68 before it merges (cleanest — the defect is in code that PR adds), or
• Merge fix: parse git's quoted, C-escaped diff headers #68 first, then flip this to ready-for-agent as a follow-on.

Relationship to neighbors

• Parse paths git always quotes (tab, newline, backslash, quote in filename) #30 / fix: parse git's quoted, C-escaped diff headers #68: this is a sub-bug of the fix fix: parse git's quoted, C-escaped diff headers #68 implements for Parse paths git always quotes (tab, newline, backslash, quote in filename) #30. Not a duplicate.
• Emit byte-safe JSON for non-UTF-8 paths and content (lone surrogates break strict parsers) #44: distinct — Emit byte-safe JSON for non-UTF-8 paths and content (lone surrogates break strict parsers) #44 is the JSON serialization (encode) side; this is the path decode side.

Draft brief (for whoever picks it up)

Root cause: git_hunk/_hunk.py, the octal-escape branch of _unquote_c_path (in #68's version), decodes one byte at a time via chr().

Fix: buffer contiguous \NNN bytes; flush via UTF-8 decode with surrogateescape when a non-octal char or end-of-string is reached.

Acceptance criteria (pytest):

1. _unquote_c_path('caf\\303\\251.txt') == 'café.txt' — multibyte UTF-8
2. _unquote_c_path('\\303\\251.txt') == 'é.txt' — UTF-8 at start
3. _unquote_c_path('od\\ttab.txt') == 'od\ttab.txt' — ASCII C-escape unaffected
4. _unquote_c_path('caf\\303\\251\\ttab.txt') == 'café\ttab.txt' — mixed
5. E2e (skip on Windows): a file named café\ttab.txt (non-ASCII and a tab, so it routes through the quoted-header path) round-trips through list --json / stage / unstage / discard with h["file"] == 'café\ttab.txt'.

Files: git_hunk/_hunk.py; tests in tests/unit/_hunk/extract_file_path_test.py and tests/e2e/quoted_paths_test.py.

[wkentaro]

This was generated by AI during triage.

Unblocked: #68 merged, flipping to ready-for-agent.

_unquote_c_path now lives on main (git_hunk/_hunk.py), and the per-byte mojibake is confirmed there — _unquote_c_path('caf\303\251.txt') returns 'café.txt', not 'café.txt'. The draft brief from triage holds; promoting it to the canonical agent brief below.

Agent Brief

Root cause: In git_hunk/_hunk.py, the octal-escape branch of _unquote_c_path decodes one byte at a time:

chars.append(chr(int(path[i + 1 : i + 4], 8)))

With core.quotePath=true (the default), git octal-escapes every byte >= 0x80, so a multibyte UTF-8 name arrives as a run of per-byte \NNN escapes. Decoding each via chr() yields one Latin-1 code point per byte (mojibake) instead of the real character.

Fix: Buffer contiguous \NNN octal escapes into a bytearray, and flush the run via bytes(buf).decode("utf-8", errors="surrogateescape") when the next char is non-octal or the string ends. surrogateescape matches how run_git already handles git output. C-escapes (\t, \n, etc.) and literal chars stay as they are.

Acceptance criteria (pytest):

1. _unquote_c_path('caf\\303\\251.txt') == 'café.txt' — multibyte UTF-8
2. _unquote_c_path('\\303\\251.txt') == 'é.txt' — UTF-8 at start
3. _unquote_c_path('od\\ttab.txt') == 'od\ttab.txt' — ASCII C-escape unaffected
4. _unquote_c_path('caf\\303\\251\\ttab.txt') == 'café\ttab.txt' — mixed octal run + C-escape
5. E2e (skip on Windows): a file named café\ttab.txt (non-ASCII and a tab, so it routes through the quoted-header path) round-trips through list --json / stage / unstage / discard with h["file"] == 'café\ttab.txt'.

Files:

• git_hunk/_hunk.py — _unquote_c_path
• tests/unit/_hunk/extract_file_path_test.py — criteria 1–4
• tests/e2e/quoted_paths_test.py — criterion 5

Out of scope: the encode/JSON-output side (#44) is a separate code path; don't touch it here.

[wkentaro]

This was generated by AI during implementation.

Blocked: the stated impact does not reproduce — _unquote_c_path's octal-multibyte branch is unreachable through the CLI.

While implementing, I found the fix targets a code path that git-hunk never feeds octal-escaped multibyte input.

Root cause of the discrepancy

run_git invokes git with -c core.quotePath=false (git_hunk/_git.py:13). git emits per-byte octal \NNN escapes for multibyte UTF-8 paths only when core.quotePath=true. With quotePath=false, multibyte bytes are emitted raw:

• plain café.txt → diff --git a/café.txt b/café.txt (unquoted; never reaches _unquote_c_path)
• café\ttab.txt → diff --git "a/café\ttab.txt" ... (quoted for the tab, but café stays raw UTF-8; only \t is C-escaped)

So _unquote_c_path only ever sees the always-quoted ASCII C-escapes (tab/newline/backslash/quote) plus single-byte control-char octals — never the multibyte octal runs the bug is about. The function's own comment at _hunk.py:116 already notes those ASCII chars are quoted "regardless of core.quotePath".

Empirical check — the issue's exact repro on current main, no fix applied:

$ printf 'a\nB\n' > café.txt
$ git-hunk list --unstaged --json | grep file
      "file": "café.txt",      # correct, not "café.txt"

The mojibake reproduces only when calling _unquote_c_path('caf\303\251.txt') directly with an octal string — input git-hunk never produces.

Consequence for the brief

• Acceptance criteria 1-4 (direct _unquote_c_path unit tests) are satisfiable; the fix makes them pass.
• Acceptance criterion 5 (e2e round-trip of café\ttab.txt) cannot be a genuine regression test: it passes on main without any fix, because café arrives raw, not octal-escaped.

Needs a maintainer decision (why I am demoting to ready-for-human rather than shipping):

1. Keep _unquote_c_path's octal-multibyte handling as defensive code, accepting unit-test-only coverage with no reachable e2e, or
2. Treat the octal-multibyte branch as effectively dead given quotePath=false and simplify/drop it, or
3. Revisit whether quotePath should ever be true.

Unassigning and demoting to ready-for-human.