Skip to content

builtins/eval: de-leak the __pyre_kw__ kwargs marker ABI and propagate raising key __eq__ from LOAD_GLOBAL - #821

Merged
youknowone merged 2 commits into
mainfrom
issue171
Jul 27, 2026
Merged

builtins/eval: de-leak the __pyre_kw__ kwargs marker ABI and propagate raising key __eq__ from LOAD_GLOBAL#821
youknowone merged 2 commits into
mainfrom
issue171

Conversation

@youknowone

@youknowone youknowone commented Jul 27, 2026

Copy link
Copy Markdown
Owner

Summary

Two related changes to builtin keyword handling and LOAD_GLOBAL name resolution, motivated by a per-call immortal-allocation (RSS) leak and a dict-key __eq__ error-propagation gap.

builtins: de-leak and harden the __pyre_kw__ kwargs marker ABI

Builtin keyword calls pack a trailing marker dict keyed by "__pyre_kw__" holding the marker sentinel.

  • Borrow-based literal-key lookups. The fixed-name lookups (print end/sep/file/flush, kwarg_get, has_builtin_kwargs; __build_class__ __pyre_kw__/metaclass; type_methods format; _pickle import-map; math prod/nextafter __pyre_kw__/start/steps) used w_dict_lookup(dict, w_str_new(name)), allocating an immortal W_UnicodeObject + Wtf8Buf wrapper per lookup solely to key the dict. Routed through borrow-based w_dict_getitem_str, which hashes the &str directly. Both are the same Option-returning strategy dispatch, and kwargs dicts hold exact-str keys, so the swap is behavior-preserving.
  • Cached marker key. call_with_kwargs/pack_pyre_kwargs minted a fresh immortal w_str_new("__pyre_kw__") per keyworded call. Cached once behind a OnceLock (w_kw_marker_key). The marker dict is collectable and drops only the borrowed key pointer on collection.
  • Sentinel-gated detection. nextafter detected kwargs with a presence-only .is_some() check on the "__pyre_kw__" key, misclassifying a positional dict that merely carries such a string key. Gated on is_kw_marker_sentinel over the value, matching prod.

eval/runtime_ops: collapse the LOAD_GLOBAL globals lookup to finditem_str

load_global_value (interpreter) and jit_load_name_from_namespace (the JIT LOAD_GLOBAL extern) both split the globals lookup by hand — an is_dict fast path through the unchecked w_dict_getitem_str and a dict-subclass path through finditem_str. The unchecked probe returns a plain Option, so a raising key __eq__ during the bucket comparison (a stored non-string key that hash-collides with the looked-up name) was swallowed as a miss instead of propagating — and the outcome differed depending on whether the frame was traced. finditem_str now takes the borrowed-string shortcut for shortcut dicts itself and drains the dict-key error, so both cases route through finditem_str(w_globals, name), matching pyopcode.py:958-960. The hand-rolled w_dict_getitem_str fast path was a pre-shortcut workaround that is now redundant.

Verification

  • python pyre/check.py --backend dynasm (reference python3.14): 329/329 PASS.
  • Leak: len(s) × 2M, JIT off, PYRE_GC_INTERP=1 — RSS flat at ~47MB (200k == 2M); the wrapper allocation is gone and finditem_str's borrowed shortcut keeps the builtins/globals lookup allocation-free.
  • Correctness: a non-string key with a colliding hash and a raising __eq__, injected into builtins.__dict__ / globals() and hit via LOAD_GLOBAL, now propagates ValueError (matching python3.14) instead of being swallowed as a miss — verified on both the interpreter and JIT paths (JIT on/off × globals/builtins).
  • math.nextafter(1.0, 2.0, {"__pyre_kw__": 1}) now raises TypeError (matching python3.14) instead of being misread as keyword arguments.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes

    • Improved global and mapping lookups so dictionary subclasses correctly honor custom key behavior.
    • Preserved expected keyword argument handling for built-in functions, math operations, formatting, and class creation.
    • Ensured missing keys and lookup errors continue to produce the appropriate results.
  • Performance

    • Reduced unnecessary temporary string creation during repeated keyword and dictionary lookups.

Builtin keyword calls pack a trailing marker dict keyed by "__pyre_kw__"
holding the marker sentinel. Three changes to that handling:

- Route the fixed-name lookups (print end/sep/file/flush, kwarg_get,
  has_builtin_kwargs; __build_class__ __pyre_kw__/metaclass; type_methods
  format; _pickle import-map; math prod/nextafter __pyre_kw__/start/steps)
  through borrow-based w_dict_getitem_str instead of
  w_dict_lookup(w_str_new(name)); the latter allocated an immortal
  W_UnicodeObject + Wtf8Buf wrapper per lookup. Both are the same
  Option-returning strategy dispatch, and kwargs dicts hold exact-str keys,
  so the borrowed probe is behavior-preserving.

- Cache the constant "__pyre_kw__" key behind a OnceLock (w_kw_marker_key)
  instead of minting a fresh immortal w_str_new per keyworded call in
  call_with_kwargs and pack_pyre_kwargs. The marker dict is collectable and
  drops only the borrowed key pointer on collection.

- Gate nextafter's kwargs detection on is_kw_marker_sentinel over the value,
  matching prod; the presence-only .is_some() check misclassified a
  positional dict carrying a "__pyre_kw__" string key as keyword arguments.

Assisted-by: Claude
…_str

load_global_value (interpreter) and jit_load_name_from_namespace (the JIT
LOAD_GLOBAL extern) both split the globals lookup by hand: an is_dict fast
path through the unchecked w_dict_getitem_str and a dict-subclass path
through finditem_str. The unchecked probe returns a plain Option, so a
raising key __eq__ during the bucket comparison (a stored non-string key
that hash-collides with the looked-up name) was swallowed as a miss instead
of propagating — and the outcome differed depending on whether the frame was
traced. finditem_str now takes the borrowed-string shortcut for shortcut
dicts itself and drains the dict-key error, so route both cases through
finditem_str(w_globals, name), matching pyopcode.py:958-960.

Assisted-by: Claude
@coderabbitai

coderabbitai Bot commented Jul 27, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 2eaa6e15-9076-4710-9022-e0a16d308914

📥 Commits

Reviewing files that changed from the base of the PR and between 3add72a and 0001c7c.

📒 Files selected for processing (8)
  • pyre/pyre-interpreter/src/builtins.rs
  • pyre/pyre-interpreter/src/call.rs
  • pyre/pyre-interpreter/src/eval.rs
  • pyre/pyre-interpreter/src/module/_pickle/mod.rs
  • pyre/pyre-interpreter/src/module/math/interp_math.rs
  • pyre/pyre-interpreter/src/runtime_ops.rs
  • pyre/pyre-interpreter/src/type_methods.rs
  • pyre/pyre-object/src/kw_marker.rs

Walkthrough

The interpreter replaces temporary string-object dictionary lookups with direct string-key retrieval, centralizes the kwargs-marker key, and routes global lookups through unified mapping dispatch.

Changes

Dictionary lookup consolidation

Layer / File(s) Summary
Cached kwargs marker key
pyre/pyre-object/src/kw_marker.rs, pyre/pyre-interpreter/src/call.rs
Adds a cached __pyre_kw__ key helper and uses it when packing builtin keyword marker dictionaries.
Builtin kwargs consumers
pyre/pyre-interpreter/src/builtins.rs, pyre/pyre-interpreter/src/call.rs, pyre/pyre-interpreter/src/module/math/interp_math.rs
Updates marker detection and keyword extraction to use direct string-key lookup while preserving sentinel checks and argument handling.
Namespace lookup dispatch
pyre/pyre-interpreter/src/eval.rs, pyre/pyre-interpreter/src/runtime_ops.rs
Routes globals and JIT namespace lookups through finditem_str, including dict-subclass mappings.
Additional string-key lookups
pyre/pyre-interpreter/src/module/_pickle/mod.rs, pyre/pyre-interpreter/src/type_methods.rs
Uses direct string-key retrieval for pickle compatibility mappings and format-render field lookup.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

  • youknowone/pyre#789: Adds related finditem and finditem_str dictionary lookup fast paths used by the globals lookup changes.

Poem

I’m a bunny with a cached little key,
Hopping through kwargs efficiently.
Strings slip softly, lookups grow bright,
Markers stay tucked and globals stay right.
Carrot cheers for cleaner code—
Hop, hop, shipped along the road!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title is concise and accurately captures the two main changes: kwargs marker handling and LOAD_GLOBAL error propagation.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch issue171

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown

🤖 Codex parity review

Static analysis of this diff vs the local RPython/PyPy sources (commit 0001c7c).
Updated: 2026-07-27T04:04:40.898Z

Files in the reviewed diff
pyre/pyre-interpreter/src/builtins.rs
pyre/pyre-interpreter/src/call.rs
pyre/pyre-interpreter/src/eval.rs
pyre/pyre-interpreter/src/module/_pickle/mod.rs
pyre/pyre-interpreter/src/module/math/interp_math.rs
pyre/pyre-interpreter/src/runtime_ops.rs
pyre/pyre-interpreter/src/type_methods.rs
pyre/pyre-object/src/kw_marker.rs

1. Regressions to PyPy parity introduced by this patch

None.

2. Other mismatches introduced by this patch

None.

3. Pre-existing mismatches (already present before this patch)

None.

4. Structural adaptations

  • pyre/pyre-interpreter/src/call.rs:2059 ↔ pypy/interpreter/gateway.py:875 — Pyre’s flat builtin ABI carries keywords in a trailing sentinel-marked dict, while PyPy passes an Arguments object through BuiltinCode.funcrun_obj and parses it against a signature. The new cached marker key preserves the existing adaptation; it does not change Python-visible keyword semantics.

  • pyre/pyre-interpreter/src/module/math/interp_math.rs:1014 ↔ pypy/module/math/interp_math.py:766 — Pyre supports the newer math.nextafter(..., steps=...) surface, whereas the local PyPy source has the older two-argument implementation. This was already present before the patch and is a Python-version adaptation.

@youknowone
youknowone merged commit 948340d into main Jul 27, 2026
18 of 19 checks passed
@youknowone
youknowone deleted the issue171 branch July 27, 2026 06:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant