Skip to content

gateway, pycode: split the fsencode boundary; report the byte-exact co_filename from every reader - #1018

Merged
youknowone merged 5 commits into
mainfrom
import
Aug 4, 2026
Merged

youknowone merged 5 commits into
mainfrom
import

Conversation

@youknowone

@youknowone youknowone commented Aug 4, 2026 •

Copy link
Copy Markdown
Owner

Follow-up to #1001, closing the two items its review left open.

1. Split the fsencode boundary into the converter and the raw encode

Upstream has two distinct things here and the difference is load-bearing:

  • baseobjspace.py:1970-1983 space.fsencode_w — the argument converter. It warns
    path should be %s, not %s as a DeprecationWarning for a readable buffer
    that is not bytes, and ends in bytesbuf0_w (:2005-2018), which rejects an
    embedded null with ValueError: embedded null byte.
  • baseobjspace.py:1962 space.fsencode — the plain unicode→bytes filesystem
    encode, with neither. interp_socket.py:157 takes this one deliberately:
    "Not using space.fsencode_w since Linux allows embedded NULs."

pyre collapsed both into one encoder, so it had neither the warning nor the null
check anywhere, and could not express the socket carve-out at all.

fsencode_bytes_w now follows fsencode_w: it resolves a readable buffer before
__fspath__, warns for a buffer that is not bytes, names the types in both
TypeErrors (interp_posix.py:3043 and :3053), and applies the null check.
gateway::fsencode is the raw form, and the AF_UNIX address packer takes it.

open() drops its own hand-rolled str / bytes-like / __fspath__ resolution for
the converter, the way interp_fileio.py:134-137 _open_fd reaches
interp_posix.py:64 space.fsencode_w through dispatch_filename.

The four resulting messages match CPython 3.14 character for character:

TypeError: expected G.__fspath__() to return str or bytes, not int
TypeError: expected str, bytes or os.PathLike object, not object
ValueError: embedded null byte          # open("a\0b")
ValueError: embedded null byte          # os.system("a\0b")

2. Report the byte-exact co_filename from every reader

PyCode owns the raw path (pycode.py:135 self.co_filename = filename) and the
co_filename getter decodes it with objspace.py:438 newfilename. Every other
reader still went to CodeObject::source_path — the lossy UTF-8 spelling, where
a path byte with no UTF-8 form has already become U+FFFD. So a module imported
from x\udcff.py showed x<?>.py in its traceback, its warnings and its frame
repr while code.co_filename showed the correct name.

w_code_filename_obj realizes the filename the way w_code_name_obj realizes
co_name. It deliberately does not cache on the PyCode: the wrapper is
prebuilt-immortal and the decode hands back a movable managed string, which is
exactly the combination the filename_bytes field doc rules out.

Routed through it: the warning location, a coroutine's cr_origin summary, the
rendered traceback, and pyframe.py:849-853 descr_repr.

The last two were the half that looked blocked, because a Rust String cannot
hold the lone surrogate:

  • the traceback writer is std::io::Write, so it keys its
    StackSummary.format dedup on the filename bytes and emits them with
    write_all. read_source_line keeps the lossy spelling, which simply fails to
    resolve — error.py:150 linecache.getline renders nothing then too.
  • descr_repr returns a Wtf8Buf, and py_repr_wtf8 takes it directly instead
    of folding through py_repr's String. Upstream interpolates the filename
    ("%s" % code.co_filename) rather than its repr, so the surrogate reaches the
    result unescaped; CPython spells the same field with %R. Both keep the byte.

Verification

All four readers now report the exact name (x\udcff.py), and the traceback's
bytes are 78 ff 2e 70 79:

frame_repr     EXACT "<frame at 0x…, file 'x\udcff.py', line 2, code cap>"
warn_filename  EXACT 'x\udcff.py'
coro_origin    EXACT 'x\udcff.py'

New parity test traceback_filename_surrogateescape.py, plus assertions added to
os_boundary_surrogateescape.py for the null boundary and for the AF_UNIX
abstract-namespace carve-out. Both green on CPython 3.14 and on pyre.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes
    • Preserved non-standard filesystem filenames, including surrogate-escaped characters, in tracebacks, warnings, frame displays, and coroutine diagnostics.
    • Improved filesystem path handling, including clearer validation and rejection of embedded NUL bytes.
    • Added support for Unix abstract socket names and more accurate socket address handling.
  • Tests
    • Added coverage for path validation, abstract sockets, and surrogate-escaped filenames across error and diagnostic output.

@coderabbitai

coderabbitai Bot commented Aug 4, 2026 •

Copy link
Copy Markdown

Review Change Stack

Walkthrough

The interpreter now preserves filesystem bytes and surrogate-escaped filenames across path conversion, traceback and frame diagnostics, warnings, coroutine origins, and AF_UNIX socket addresses. Parity tests cover embedded NUL rejection, abstract sockets, and diagnostic output.

Changes

Filesystem surrogateescape preservation

Layer / File(s) Summary
Path encoding and validation
pyre/pyre-interpreter/src/gateway.rs, pyre/pyre-interpreter/src/builtins.rs, pyre/pyre-interpreter/src/typedef.rs, pyre/extra_tests/parity_tests/os_boundary_surrogateescape.py
Shared path conversion validates path-like results, preserves filesystem bytes, rejects embedded NUL bytes, and supports WTF-8 decoding.
Code filename and diagnostic representation
pyre/pyre-interpreter/src/pycode.rs, pyre/pyre-interpreter/src/error.rs, pyre/pyre-interpreter/src/pyframe.rs, pyre/pyre-interpreter/src/display.rs, pyre/pyre-interpreter/src/module/_warnings/mod.rs, pyre/pyre-interpreter/src/typedef.rs, pyre/extra_tests/parity_tests/traceback_filename_surrogateescape.py
Code filenames remain available as preserved bytes and objects across traceback formatting, source loading, frame representations, warnings, coroutine origins, and subprocess output.
AF_UNIX address packing and unpacking
pyre/pyre-interpreter/src/module/_socket/interp_socket.rs, pyre/extra_tests/parity_tests/os_boundary_surrogateescape.py
AF_UNIX handling supports abstract names, uses kernel-reported address lengths, and decodes regular filesystem paths from bytes.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant PythonSocket
  participant AF_UNIXPacking
  participant Kernel
  participant AF_UNIXUnpacking
  PythonSocket->>AF_UNIXPacking: provide filesystem or abstract address
  AF_UNIXPacking->>Kernel: bind or send sockaddr with calculated length
  Kernel-->>AF_UNIXUnpacking: return sockaddr and address length
  AF_UNIXUnpacking-->>PythonSocket: preserve abstract bytes or decode filesystem path
Loading

Possibly related PRs

  • youknowone/pyre#721: Both changes touch shared representation code, including builtins.rs, display.rs, and typedef.rs.
  • youknowone/pyre#887: Both changes modify frame filename handling in pyframe.rs.
  • youknowone/pyre#1001: Both changes extend filesystem-byte and surrogateescape handling across interpreter, socket, and parity-test paths.

Suggested reviewers: lifthrasiir

Poem

A rabbit guards each filename byte,
Keeps NULs from paths where they cannot wait.
Frames speak names without replacement,
Sockets preserve address representation.
“WTF-8 stays whole!” the rabbit says.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly identifies the filesystem-encoding split and byte-exact co_filename reporting, which are the PR's main changes.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch import

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Aug 4, 2026 •

Copy link
Copy Markdown

🤖 Codex parity review

Static analysis of this diff vs the local RPython/PyPy sources (commit 71298bc).
Updated: 2026-08-04T06:25:30.895Z

Files in the reviewed diff
pyre/extra_tests/parity_tests/os_boundary_surrogateescape.py
pyre/extra_tests/parity_tests/traceback_filename_surrogateescape.py
pyre/pyre-interpreter/src/builtins.rs
pyre/pyre-interpreter/src/display.rs
pyre/pyre-interpreter/src/error.rs
pyre/pyre-interpreter/src/gateway.rs
pyre/pyre-interpreter/src/module/_socket/interp_socket.rs
pyre/pyre-interpreter/src/module/_warnings/mod.rs
pyre/pyre-interpreter/src/pycode.rs
pyre/pyre-interpreter/src/pyframe.rs
pyre/pyre-interpreter/src/typedef.rs

1. Regressions to PyPy parity introduced by this patch

None.

2. Other mismatches introduced by this patch

None.

3. Pre-existing mismatches (already present before this patch)

  • pyre/pyre-interpreter/src/gateway.rs:1540 ↔ pypy/interpreter/baseobjspace.py:1970-1983 — fsencode_w_with_kind converts filesystem bytes through String::from_utf8_lossy, replacing invalid bytes with U+FFFD; PyPy retains the original byte string. This behavior was already present on upstream/main.

  • pyre/pyre-interpreter/src/gateway.rs:1572 ↔ pypy/interpreter/baseobjspace.py:1971-1983 — Pyre recognizes only bytes | bytearray via is_bytes_like; PyPy accepts any full read-only buffer exporter for fsencode_w. This limitation predates the patch.

  • pyre/pyre-interpreter/src/module/_socket/interp_socket.rs:2218 ↔ rpython/rlib/rsocket.py:409-420 — regular AF_UNIX addresses include the trailing terminator in addrlen; RPython sets addrlen to offsetof(sun_path) + len(path) and stores the terminator outside that length. The former implementation had the same extra byte.

  • pyre/pyre-interpreter/src/pycode.rs:363 ↔ pypy/interpreter/pycode.py:135-136 — Pyre recreates co_filename on each read, whereas PyPy retains w_filename and thus preserves object identity. The prior co_filename getter already recreated the object; this patch centralizes that behavior.

4. Structural adaptations

  • pyre/pyre-interpreter/src/pyframe.rs:3194 ↔ pypy/interpreter/pyframe.py:849-853 — the frame repr is assembled in Wtf8Buf, rather than an RPython string, because Rust String cannot represent lone surrogates. This preserves the surrogateescape spelling.

  • pyre/pyre-interpreter/src/error.rs:2961 ↔ pypy/interpreter/error.py:136-150 — source lookup uses an OS-native Path from raw bytes on Unix instead of invoking Python’s linecache; this is the Rust host/sandbox adaptation needed to retain non-UTF-8 filesystem names.

@youknowone

Copy link
Copy Markdown
Owner Author

Adjudicated the Codex parity review. Three findings fixed, one refuted, one deferred with the blocker named.

§1 — refuted

the new shared fsencode_bytes_w rejects NUL bytes … PyPy's pathname='fsencode' uses raw space.fsencode

unwrap_spec 'fsencode' does not reach space.fsencode. gateway.py:366 and
gateway.py:578 both expand visit_fsencode to space.fsencode_w(...), which
ends in bytesbuf0_w — so _imp._fix_co_filename(code, "a\0b") raises
ValueError: embedded null byte upstream too, exactly as it now does here.

CPython does diverge: it accepts the name and stores 'a\x00b'. That is a
CPython-vs-PyPy difference, and this repo follows PyPy.

§3 — two fixed, one deferred

  • __fspath__ returning bytearray — fixed. interp_posix.py:3049-3051
    narrows the result to str or bytes; a readable buffer is not a path. The
    message now matches upstream and CPython character for character:
    expected B.__fspath__() to return str or bytes, not bytearray.

  • AF_UNIX abstract namespace — fixed, and it was the larger of the two.
    unpack_inet_addr had no address length at all, so it could not compute
    rsocket.py:432's maxlength and the NUL scan stopped at byte 0. It now
    takes the length, every one of the nine callers passes what the OS wrote back
    (slen, msg_namelen, ai_addrlen), and interp_socket.py:45 returns the
    abstract name as bytes. The packing side gained the mirror fix:
    rsocket.py:409-410's baseofs + len(path), with the terminator counted only
    for a regular name, and the offset taken from sockaddr_un rather than from
    sa_family_t — not the same on every platform.

  • traceback source lookup — fixed. The lossy spelling reached the source
    provider, so a surrogate-named file could never render its offending line.
    The bytes are carried through and turned into a Path with
    OsStr::from_bytes.

  • memoryview as a path — deferred. baseobjspace.py:1971 accepts any
    BUF_FULL_RO exporter; this boundary has no buffer-protocol entry point to
    call, only the bytes | bytearray predicates. Closing it needs that seam
    first, which is more than this diff should carry.

§4 — documented, no change

Non-stable co_filename object identity is the deliberate adaptation the
filename_bytes field doc describes: the wrapper is prebuilt-immortal and the
decode hands back a movable managed string, so caching one on it would need
per-collection forwarding. It predates this diff — the previous getter also
built a fresh string per read — and the value is unchanged either way.

Verification

cargo fmt --check clean; full extra_tests dynasm run is 232/241 both before
and after, with the same nine pre-existing failures (NaN ordering ×3, ast
identity, dict.fromkeys.__qualname__, sys._getframemodulename, absent
_sqlite3, absent __pypy__.builders, the known unpickler desync). The LLBC
was re-extracted after the rebase, and the numbers are identical across that
too. The abstract-namespace branch is compiled out on macOS, so only Linux CI
executes it; the regular AF_UNIX bind/getsockname() round trip is verified
locally and matches CPython.

— commented by Claude

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pyre/extra_tests/parity_tests/traceback_filename_surrogateescape.py`:
- Around line 66-78: Strengthen the assertions after the subprocess invocation
in the traceback test: verify that child.returncode is nonzero, and require the
expected filename value from the _imp._fix_co_filename call to appear in stderr,
while retaining the surrogateescape decoding and U+FFFD check.

In `@pyre/pyre-interpreter/src/module/_socket/interp_socket.rs`:
- Around line 2205-2218: Update the AF_UNIX address handling around
abstract_name to recognize a leading NUL on both Linux and Android, then reject
any NUL occurring in all other paths before copying bytes into sun.sun_path.
Ensure Android abstract names use abstract sizing and addrlen without a
regular-path terminator, while preserving the existing capacity and error
behavior.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 41e4aefc-b739-4288-b70d-b91cd95cc77e

📥 Commits

Reviewing files that changed from the base of the PR and between 4d75883 and b8edf00.

📒 Files selected for processing (11)
  • pyre/extra_tests/parity_tests/os_boundary_surrogateescape.py
  • pyre/extra_tests/parity_tests/traceback_filename_surrogateescape.py
  • pyre/pyre-interpreter/src/builtins.rs
  • pyre/pyre-interpreter/src/display.rs
  • pyre/pyre-interpreter/src/error.rs
  • pyre/pyre-interpreter/src/gateway.rs
  • pyre/pyre-interpreter/src/module/_socket/interp_socket.rs
  • pyre/pyre-interpreter/src/module/_warnings/mod.rs
  • pyre/pyre-interpreter/src/pycode.rs
  • pyre/pyre-interpreter/src/pyframe.rs
  • pyre/pyre-interpreter/src/typedef.rs

Comment on lines +66 to +78
child = subprocess.run(
[
sys.executable,
"-c",
"import _imp\n"
"code = compile(\"raise ValueError('boom')\", 'orig.py', 'exec')\n"
"_imp._fix_co_filename(code, 'x\\udcff.py')\n"
"exec(code)\n",
],
capture_output=True,
)
stderr = child.stderr.decode(errors="surrogateescape")
assert "�" not in stderr, ascii(stderr)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Assert the child failure and the exact filename.

Lines 77-78 only reject U+FFFD. An empty stderr or a successful child process passes this test. Assert a nonzero return code and require FILENAME in stderr.

Proposed test fix
     child = subprocess.run(
         [
             sys.executable,
@@
         ],
         capture_output=True,
+        check=False,
     )
     stderr = child.stderr.decode(errors="surrogateescape")
+    assert child.returncode != 0
+    assert FILENAME in stderr, ascii(stderr)
     assert "�" not in stderr, ascii(stderr)
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
child = subprocess.run(
[
sys.executable,
"-c",
"import _imp\n"
"code = compile(\"raise ValueError('boom')\", 'orig.py', 'exec')\n"
"_imp._fix_co_filename(code, 'x\\udcff.py')\n"
"exec(code)\n",
],
capture_output=True,
)
stderr = child.stderr.decode(errors="surrogateescape")
assert "�" not in stderr, ascii(stderr)
child = subprocess.run(
[
sys.executable,
"-c",
"import _imp\n"
"code = compile(\"raise ValueError('boom')\", 'orig.py', 'exec')\n"
"_imp._fix_co_filename(code, 'x\\udcff.py')\n"
"exec(code)\n",
],
capture_output=True,
check=False,
)
stderr = child.stderr.decode(errors="surrogateescape")
assert child.returncode != 0
assert FILENAME in stderr, ascii(stderr)
assert "�" not in stderr, ascii(stderr)
🧰 Tools
🪛 Ruff (0.16.0)

[warning] 66-66: subprocess.run without explicit check argument

Add explicit check=False

(PLW1510)


[warning] 70-73: Unparenthesized implicit string concatenation in collection

Did you forget a comma?

(ISC004)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pyre/extra_tests/parity_tests/traceback_filename_surrogateescape.py` around
lines 66 - 78, Strengthen the assertions after the subprocess invocation in the
traceback test: verify that child.returncode is nonzero, and require the
expected filename value from the _imp._fix_co_filename call to appear in stderr,
while retaining the surrogateescape decoding and U+FFFD check.

Source: Linters/SAST tools

Comment on lines +2205 to +2218
let abstract_name = cfg!(target_os = "linux") && path_bytes_vec.first() == Some(&0);
// `rsocket.py:412-420 UNIXAddress.__init__`: an abstract name may fill
// `sun_path` exactly, a regular one has to leave room for its
// terminator.
let capacity = sun.sun_path.len() - usize::from(!abstract_name);
if path_bytes_vec.len() > capacity {
return Err(crate::PyError::os_error("AF_UNIX path too long"));
}
for (i, &b) in path_bytes_vec.iter().enumerate() {
sun.sun_path[i] = b as libc::c_char;
}
return Ok((
storage,
(core::mem::size_of::<libc::sa_family_t>() + path_bytes_vec.len() + 1)
as libc::socklen_t,
));
// `rsocket.py:409-410 self.setdata(sun, baseofs + len(path))`: the
// terminator is counted only for a regular name that has one.
let addrlen = SUN_PATH_OFFSET + path_bytes_vec.len() + usize::from(!abstract_name);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Reject NULs in regular AF_UNIX names.

A non-abstract path such as b"a\0b" passes this branch and reaches libc. The OS then treats the NUL as the pathname boundary instead of raising the required ValueError.

Treat a leading NUL as abstract on both Linux and Android. Reject any NUL in every other path. This also prevents Android abstract names from receiving a regular-path terminator.

Proposed fix
-        let abstract_name = cfg!(target_os = "linux") && path_bytes_vec.first() == Some(&0);
+        let abstract_name =
+            cfg!(any(target_os = "linux", target_os = "android"))
+                && path_bytes_vec.first() == Some(&0);
+        if !abstract_name && path_bytes_vec.contains(&0) {
+            return Err(crate::PyError::value_error("embedded null byte"));
+        }

As per coding guidelines, “Preserve interpreter semantics exactly in the generated JIT.”

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
let abstract_name = cfg!(target_os = "linux") && path_bytes_vec.first() == Some(&0);
// `rsocket.py:412-420 UNIXAddress.__init__`: an abstract name may fill
// `sun_path` exactly, a regular one has to leave room for its
// terminator.
let capacity = sun.sun_path.len() - usize::from(!abstract_name);
if path_bytes_vec.len() > capacity {
return Err(crate::PyError::os_error("AF_UNIX path too long"));
}
for (i, &b) in path_bytes_vec.iter().enumerate() {
sun.sun_path[i] = b as libc::c_char;
}
return Ok((
storage,
(core::mem::size_of::<libc::sa_family_t>() + path_bytes_vec.len() + 1)
as libc::socklen_t,
));
// `rsocket.py:409-410 self.setdata(sun, baseofs + len(path))`: the
// terminator is counted only for a regular name that has one.
let addrlen = SUN_PATH_OFFSET + path_bytes_vec.len() + usize::from(!abstract_name);
let abstract_name =
cfg!(any(target_os = "linux", target_os = "android"))
&& path_bytes_vec.first() == Some(&0);
if !abstract_name && path_bytes_vec.contains(&0) {
return Err(crate::PyError::value_error("embedded null byte"));
}
// `rsocket.py:412-420 UNIXAddress.__init__`: an abstract name may fill
// `sun_path` exactly, a regular one has to leave room for its
// terminator.
let capacity = sun.sun_path.len() - usize::from(!abstract_name);
if path_bytes_vec.len() > capacity {
return Err(crate::PyError::os_error("AF_UNIX path too long"));
}
for (i, &b) in path_bytes_vec.iter().enumerate() {
sun.sun_path[i] = b as libc::c_char;
}
// `rsocket.py:409-410 self.setdata(sun, baseofs + len(path))`: the
// terminator is counted only for a regular name that has one.
let addrlen = SUN_PATH_OFFSET + path_bytes_vec.len() + usize::from(!abstract_name);
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pyre/pyre-interpreter/src/module/_socket/interp_socket.rs` around lines 2205
- 2218, Update the AF_UNIX address handling around abstract_name to recognize a
leading NUL on both Linux and Android, then reject any NUL occurring in all
other paths before copying bytes into sun.sun_path. Ensure Android abstract
names use abstract sizing and addrlen without a regular-path terminator, while
preserving the existing capacity and error behavior.

Source: Coding guidelines

…ncode

`fsencode_bytes_w` now follows `baseobjspace.py:1970-1983 fsencode_w`: it
resolves a readable buffer before `__fspath__`, warns `path should be string,
bytes, or os.PathLike, not <type>` as a `DeprecationWarning` for a buffer that
is not `bytes`, names the types in both `TypeError`s (`interp_posix.py:3043`
and `:3053`), and rejects an embedded null with `ValueError: embedded null
byte` (`baseobjspace.py:2005-2018 bytesbuf0_w`).

`gateway::fsencode` is `baseobjspace.py:1962 space.fsencode` with none of
that. The AF_UNIX address packer takes it, per `interp_socket.py:157`'s note
that Linux allows embedded NULs in an abstract-namespace path.

`open()` drops its own str / bytes-like / `__fspath__` resolution for the
converter, the way `interp_fileio.py:134-137 _open_fd` reaches
`interp_posix.py:64 space.fsencode_w` through `dispatch_filename`.

Assisted-by: Claude
…ce_path

`w_code_filename_obj` realizes `pycode.py:135 self.co_filename` the way
`w_code_name_obj` realizes `co_name`, decoding the owned filesystem bytes with
`objspace.py:438 newfilename`. It does not cache the result on the `PyCode`:
the wrapper is prebuilt-immortal and the decode hands back a movable managed
string. `code_get_field`'s `co_filename` arm now goes through it.

The consumers that previously read `CodeObject::source_path` — the lossy UTF-8
spelling, where a path byte with no UTF-8 form has already become U+FFFD — take
the exact name instead: the warning location, a coroutine's `cr_origin`
summary, the rendered traceback and `pyframe.py:849-853 descr_repr`.

The traceback writer keys its `StackSummary.format` dedup on the filename bytes
and emits them with `write_all`; `read_source_line` keeps the lossy spelling,
which simply fails to resolve, as `error.py:150 linecache.getline` does when
the source cannot be read. `descr_repr` returns a `Wtf8Buf`, since a Rust
`String` cannot hold the lone surrogate, and `py_repr_wtf8` takes it directly
rather than through `py_repr`'s `String`.

Assisted-by: Claude
`interp_posix.py:3049-3051 fspath` narrows the result to those two; a
`bytearray` exports a readable buffer but is not a path, so the resolution now
reports `expected <type>.__fspath__() to return str or bytes, not bytearray`
instead of taking its bytes.

Assisted-by: Claude
… bytes

`read_source_line` took the lossy UTF-8 spelling, so a path byte with no UTF-8
form reached the source provider as U+FFFD and named a file that does not
exist. `error.py:150 linecache.getline` is handed `co_filename` itself, whose
surrogateescape spelling reopens the byte-exact path, so the bytes are carried
through and turned into a `Path` with `OsStr::from_bytes`. Platforms with no
bytes spelling of a path keep the lossy form, where such a name does not
resolve either way.

Assisted-by: Claude
… name

An AF_UNIX name is not always NUL-terminated: on Linux a path whose first byte
is `\0` names the abstract namespace, where the name runs to the end of the
address, embedded NULs included, and its length is only derivable from the
address length.

`unpack_inet_addr` now takes that length and follows `rsocket.py:430-443
UNIXAddress.get_path`: `maxlength` is the address length past
`offsetof(sockaddr_un, sun_path)`, an abstract name takes all of it, and a
regular name is the NUL scan bounded by it. `interp_socket.py:41-47` returns
the abstract name as `bytes` and the regular one through the filesystem
decoding. Previously the scan stopped at byte 0, so `getsockname()` on an
abstract socket answered `''`.

Each of the nine callers passes the length the OS wrote back — `slen` for
`accept`/`recvfrom`/`getsockname`/`getpeername`, `msg_namelen` for the two
`recvmsg` forms, `ai_addrlen` for `getaddrinfo`.

`pack_inet_addr` reports `rsocket.py:409-410`'s `baseofs + len(path)`, counting
the terminator only for a regular name that has one, and takes the offset from
`sockaddr_un` rather than from `sa_family_t`, which is not the same on every
platform. An abstract name may fill `sun_path` exactly
(`rsocket.py:412-420`).

Assisted-by: Claude

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

https://github.com/youknowone/pyre/blob/71298bcfe33a52d1dbc6b8cd1d4c7e37e91bf310/pyre-interpreter/src/error.rs#L2749
P2 Badge Preserve the filename when writing through sys.stderr

When this traceback is routed through a live Python text stream—for example, by calling sys.__excepthook__ with sys.stderr = io.StringIO()—writing the raw filename byte here loses the surrogate: builtins.rs:3940 converts the rendered buffer with String::from_utf8_lossy, so x\udcff.py becomes x�.py before stderr.write; write_unraisable_to_sys_stderr has the same conversion at error.rs:1424. Render the filename as WTF-8/text through these paths (as upstream's text formatting does), rather than inserting filesystem bytes into a buffer later decoded as UTF-8.

AGENTS.md reference: AGENTS.md:L194-L196

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
pyre/pyre-interpreter/src/pycode.rs (1)

2018-2063: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Free filename_bytes in pycode_destructor.

set_filename_bytes stores the custom filename in filename_bytes via Box::into_raw, so pycode_destructor should drop that Box to keep custom filename allocations from leaking when the GC reclaims the PyCode wrapper.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pyre/pyre-interpreter/src/pycode.rs` around lines 2018 - 2063, Update
pycode_destructor to release the allocation held by code.filename_bytes,
mirroring the existing Box::from_raw cleanup for globals_caches, mapdict_caches,
and co_consts_w. Guard the cleanup for a null pointer and reset the field
afterward.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pyre/extra_tests/parity_tests/traceback_filename_surrogateescape.py`:
- Around line 42-43: Update the assertions after constructing frame_repr to
first verify that it contains the expected FILENAME value, then retain the
assertion rejecting U+FFFD. Ensure the test fails when the frame reports orig.py
instead of the expected filename.

---

Outside diff comments:
In `@pyre/pyre-interpreter/src/pycode.rs`:
- Around line 2018-2063: Update pycode_destructor to release the allocation held
by code.filename_bytes, mirroring the existing Box::from_raw cleanup for
globals_caches, mapdict_caches, and co_consts_w. Guard the cleanup for a null
pointer and reset the field afterward.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 770cab40-d7b7-4627-a5ae-627b4e62377a

📥 Commits

Reviewing files that changed from the base of the PR and between b8edf00 and 71298bc.

📒 Files selected for processing (11)
  • pyre/extra_tests/parity_tests/os_boundary_surrogateescape.py
  • pyre/extra_tests/parity_tests/traceback_filename_surrogateescape.py
  • pyre/pyre-interpreter/src/builtins.rs
  • pyre/pyre-interpreter/src/display.rs
  • pyre/pyre-interpreter/src/error.rs
  • pyre/pyre-interpreter/src/gateway.rs
  • pyre/pyre-interpreter/src/module/_socket/interp_socket.rs
  • pyre/pyre-interpreter/src/module/_warnings/mod.rs
  • pyre/pyre-interpreter/src/pycode.rs
  • pyre/pyre-interpreter/src/pyframe.rs
  • pyre/pyre-interpreter/src/typedef.rs

Comment on lines +42 to +43
frame_repr = repr(namespace["frame"])
assert "�" not in frame_repr, ascii(frame_repr)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Assert the expected frame filename.

Line 43 only rejects U+FFFD. A frame representation that still reports orig.py passes. Assert FILENAME before the replacement-character assertion.

Proposed fix
 frame_repr = repr(namespace["frame"])
+assert FILENAME in frame_repr, ascii(frame_repr)
 assert "�" not in frame_repr, ascii(frame_repr)
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
frame_repr = repr(namespace["frame"])
assert "�" not in frame_repr, ascii(frame_repr)
frame_repr = repr(namespace["frame"])
assert FILENAME in frame_repr, ascii(frame_repr)
assert "�" not in frame_repr, ascii(frame_repr)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pyre/extra_tests/parity_tests/traceback_filename_surrogateescape.py` around
lines 42 - 43, Update the assertions after constructing frame_repr to first
verify that it contains the expected FILENAME value, then retain the assertion
rejecting U+FFFD. Ensure the test fails when the frame reports orig.py instead
of the expected filename.

@youknowone

Copy link
Copy Markdown
Owner Author

Rebased onto origin/main and re-verified. The pyre/check.py reds on all three OSes are base-owned.

Attribution

main's own CI run at the same base sha (ff42f0e3317, #950) fails the same three jobs. The failure sets are identical, not merely similar:

main @ base this branch
ubuntu dynasm 8 / cranelift 16 / wasm 14 — 38 FAIL lines identical, same 38 lines
macOS dynasm 8 / cranelift 16 identical
windows dynasm 8 / cranelift 17, same 25-bench list character-identical

Diffing the two ubuntu FAIL lists leaves nothing on either side — same benches, same counters, same numbers (loops_compiled 407 -> 409, guard_failures 19702 -> 802, and so on).

That is what you would expect from #950: it moved PyCode wrappers from Box-immortal to managed stable-oldgen, which changes trace shape across the exception, gc and raise benches, and the recorded baselines predate it. Nothing in this branch touches the optimizer or adds a JIT-visible struct field.

I deliberately did not re-record any baseline here — that would absorb main's drift into this diff.

Rebase notes

Two conflicts, both blended rather than taken from one side:

  • pycode.rs — main reworded the wrapper-family doc ("Bootstrap/prebuilt"), we inserted w_code_filename_obj at the same point. Kept both. The function's doc justified not caching by calling PyCode prebuilt-immortal, which builtins: exception, iterator and function attribute surface parity #950 made stale, so it now states the reason that survives the model change: the filesystem decode returns a movable managed string, so a retained slot would need a managed wrapper's custom trace and eval::walk_raw_code_roots for the bootstrap family. Corrected the same stale claim in the filename_bytes field doc it cites.
  • error.rs — main switched the traceback source read to read_source_bytes + decode_source_bytes, since linecache opens with tokenize.open and the BOM/PEP 263 cookie decide the encoding. We had changed the same call to take a byte-derived Path. Kept main's tokenize-aware decode over our path, so a non-UTF-8 source name both resolves and decodes by its declared encoding.

The other three commits are byte-identical across the rebase (same patch-ids).

Verification after the rebase

LLBC re-extracted, release rebuilt, full extra_tests dynasm run: 234/241, up from 232 — builtin_compile.py and pickle_pypy_module.py now pass on the new base. The remaining seven are the same pre-existing failures as before (NaN ordering ×3, dict.fromkeys.__qualname__, sys._getframemodulename, absent _sqlite3, the known unpickler desync). No new failure.

— commented by Claude

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant