Skip to content

#696: a closure returned from a module runs against its own chunk - #698

Merged
Masterplanner25 merged 1 commit into
mainfrom
fix/696-returned-closure-chunk
Aug 31, 2026
Merged

Masterplanner25 merged 1 commit into
mainfrom
fix/696-returned-closure-chunk

Conversation

@Masterplanner25

Copy link
Copy Markdown
Owner

Closes #696. The mirror of #691, found by probing the neighbourhood of that fix on the day it merged.

Why the two could not share a fix

This is the part worth reading, and it is a correction to the assumption I filed #696 with.

Every context source #691's fix uses records something a call is still inside of — a _ClosureProxy wrapped for an argument, a live cross-module frame, a caller VM. A returned closure is called after all three are gone: the frame popped when the module function returned, the proxy was for an argument, and at top level there is no caller VM. All three come back empty, and the closure runs at its own address in whatever chunk happens to be loaded.

So finding a second instance of a shape is not the same as finding a second instance of the fix. Worth checking whether the mechanism you already wrote can even reach the new case before assuming symmetry — #476's "check the reverse direction" gets you to the bug, not to the repair.

What did generalise was the predicate. _is_foreign_closure detected the returned closure correctly, because #691's widening made it chunk-relative rather than _caller_vm-relative. Only the resolution was missing. A correct question with no available answer is a much better place to be than a question nobody asks, and that is an argument for widening a predicate past its immediate caller.

Symptoms

Same five-way spread as #691, for the same reason — the symptom is whatever sits at that address. One repro each, measured on the unfixed tree:

Symptom
Method calls are only supported on records returned closure, no upvalue
run_workflow(workflow) expects a workflow same, called inside a step body
Cannot add int and string returned closure carrying an upvalue
Stack underflow returned nested in a list
'NoneType' object is not subscriptable a counter closure inside a step
prints nothing at all a stateful counter at top level

No workflow and no coroutine needed. let f = m.plain_maker() in fn main() was enough, which makes a factory function — an ordinary thing for a module to export — unusable.

The fix resolves rather than marks

Marking a closure on the way out would mean a hook at each exit and a walk of returned lists, maps and records — the case #339 found the entry side had missed. Instead VM._foreign_closure_origin asks which of the modules this VM can reach owns the closure's FunctionInfo. A module holds its functions table for its whole life, so the answer survives every frame being gone, needs no mark, and covers containers for free.

Reachability, not a process-wide registry of every module ever loaded. A global would be the module-scope state shape behind #185 and #390, and it would let one tenant's VM resolve another's chunk. What a program can call, it can see — including transitively imported modules, reached through the ones it binds.

VM.module_ctx is now the single definition of a module's execution context, consulted by _try_enter_module_call on the way in and by the resolution above. Two hand-built copies of that tuple would be two answers to one question, which is how most rows of CLAUDE.md's recurring-shape table got there. test_a_modules_context_is_built_in_one_place asserts on the source so a second one fails the suite.

Tests

11 new cases in tests/test_cross_module_closure.py, each run both under the CLI and embedded: top level, inside a step body, an upvalue-carrying adder, a stateful counter, nested in a list and in a map, a closure from a transitively imported module (main binds outer, only outer binds inner), two modules whose anonymous functions share a name, interleaved closures from two modules, plus two controls — a caller-owned closure round-tripping unchanged, and std:retry still working.

Also two unit tests on the resolution itself: that an unreachable module is not resolvable, and that a cycle between two modules terminates.

Falsified: 11 tests go red against the unfixed tree, and both source assertions with them.

One of those source assertions could not have failed as first written — the token module_ctx is a substring of _capture_module_ctx, so it matched the unfixed code too. Tightened to self.module_ctx( and re-checked. That is the fourth in-session unfalsifiable assertion this project has recorded; the check is cheap and it keeps catching them.

Testing

  • 394 targeted tests green across the module, closure, coroutine, workflow, stdlib and opcode suites.
  • nodus_gate green: closed-issues 3/3, opcodes 29/29, contracts 6/6, invariants 4/4, static 139/139, shapes 0 new, versions 13/13.
  • ruff and mypy clean.

Throughput: module_ctx builds a tuple where _try_enter_module_call used to assign fields directly. On 40,000 cross-module calls in a tight loop and nothing else, best-of-five is 3.045s against 2.929s baseline and 3.066s on a re-run — about 4% on the minima, while the medians overlap and put the fixed tree lower. Worth it for one definition instead of two.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EdYjdCRLuedehwRCdi2hXb

The mirror of #691, found by probing the neighbourhood of that fix on the day
it merged. #691 handled closures going INTO a module; this is one coming out.

The two could not share a fix, and that is the part worth reading. Every
context source #691 uses records something a call is still INSIDE of -- a
_ClosureProxy wrapped for an argument, a live cross-module frame, a caller VM.
A returned closure is called after all three are gone: the frame popped when
the module function returned, the proxy was for an argument, and there is no
caller VM at top level. So all three come back empty and the closure runs at
its own address in whatever chunk happens to be loaded.

Same five-way symptom spread as #691, for the same reason -- the symptom is
whatever sits at that address. One repro each, measured on the unfixed tree:

  Method calls are only supported on records
  run_workflow(workflow) expects a workflow
  Cannot add int and string
  Stack underflow
  'NoneType' object is not subscriptable

and a stateful factory closure that printed nothing at all.

No workflow and no coroutine needed: `let f = m.plain_maker()` in `fn main()`
was enough, which makes a factory function -- an ordinary thing for a module to
export -- unusable.

WHAT ALREADY WORKED, and why it matters. `_is_foreign_closure` detected the
returned closure correctly, because #691's widening made it chunk-relative
rather than `_caller_vm`-relative. Only the resolution was missing. A correct
question with no available answer is a much better place to be than a question
nobody asks.

THE FIX RESOLVES RATHER THAN MARKS. Marking a closure on the way out would mean
a hook at each exit AND a walk of returned lists, maps and records -- the case
#339 found the entry side had missed. Instead `_foreign_closure_origin` asks
which of the modules this VM can REACH owns the closure's FunctionInfo: a module
holds its `functions` table for its whole life, so the answer survives every
frame being gone, needs no mark, and covers containers for free.

Reachability, not a process-wide registry of every module ever loaded. A global
would be the module-scope state shape behind #185 and #390, and it would let one
tenant's VM resolve another's chunk. What a program can call, it can see --
including transitively imported modules, reached through the ones it binds.

`VM.module_ctx` is now the single definition of a module's execution context,
consulted by `_try_enter_module_call` on the way in and by the resolution above.
Two hand-built copies of that tuple would be two answers to one question, which
is how most rows of CLAUDE.md's recurring-shape table got there.
`test_a_modules_context_is_built_in_one_place` asserts on the source so a second
one fails the suite.

Tests: 11 new cases in `tests/test_cross_module_closure.py`, each run both under
the CLI and embedded -- top level, inside a step body, an upvalue-carrying
adder, a stateful counter, nested in a list and in a map, a closure from a
transitively imported module, two modules whose anonymous functions share a
name, interleaved closures from two modules, and two controls (a caller-owned
closure round-tripping unchanged, and `std:retry` still working). Verified
against the unfixed tree: 11 tests go red, and both source assertions with them.

One of those source assertions could not have failed as first written: the token
`module_ctx` is a substring of `_capture_module_ctx`, so it matched the unfixed
code too. Tightened to `self.module_ctx(` and re-checked against the unfixed
tree. That is the fourth in-session unfalsifiable assertion this project has
recorded; the check is cheap and it keeps catching them.

Throughput: `module_ctx` builds a tuple where `_try_enter_module_call` used to
assign fields directly. On 40,000 cross-module calls in a tight loop and nothing
else, best-of-five is 3.045s against 2.929s baseline and 3.066s on a re-run --
about 4% on the minima, while the medians overlap and put the fixed tree lower.
Worth it for one definition instead of two.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EdYjdCRLuedehwRCdi2hXb
@Masterplanner25
Masterplanner25 merged commit 43710fe into main Aug 31, 2026
4 checks passed
@Masterplanner25
Masterplanner25 deleted the fix/696-returned-closure-chunk branch August 31, 2026 00:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BUG: a closure RETURNED from a module runs against the caller's chunk — silent truncation at top level

1 participant