Repository navigation
#696: a closure returned from a module runs against its own chunk - #698
Merged
Merged
Conversation
The mirror of #691, found by probing the neighbourhood of that fix on the day it merged. #691 handled closures going INTO a module; this is one coming out. The two could not share a fix, and that is the part worth reading. Every context source #691 uses records something a call is still INSIDE of -- a _ClosureProxy wrapped for an argument, a live cross-module frame, a caller VM. A returned closure is called after all three are gone: the frame popped when the module function returned, the proxy was for an argument, and there is no caller VM at top level. So all three come back empty and the closure runs at its own address in whatever chunk happens to be loaded. Same five-way symptom spread as #691, for the same reason -- the symptom is whatever sits at that address. One repro each, measured on the unfixed tree: Method calls are only supported on records run_workflow(workflow) expects a workflow Cannot add int and string Stack underflow 'NoneType' object is not subscriptable and a stateful factory closure that printed nothing at all. No workflow and no coroutine needed: `let f = m.plain_maker()` in `fn main()` was enough, which makes a factory function -- an ordinary thing for a module to export -- unusable. WHAT ALREADY WORKED, and why it matters. `_is_foreign_closure` detected the returned closure correctly, because #691's widening made it chunk-relative rather than `_caller_vm`-relative. Only the resolution was missing. A correct question with no available answer is a much better place to be than a question nobody asks. THE FIX RESOLVES RATHER THAN MARKS. Marking a closure on the way out would mean a hook at each exit AND a walk of returned lists, maps and records -- the case #339 found the entry side had missed. Instead `_foreign_closure_origin` asks which of the modules this VM can REACH owns the closure's FunctionInfo: a module holds its `functions` table for its whole life, so the answer survives every frame being gone, needs no mark, and covers containers for free. Reachability, not a process-wide registry of every module ever loaded. A global would be the module-scope state shape behind #185 and #390, and it would let one tenant's VM resolve another's chunk. What a program can call, it can see -- including transitively imported modules, reached through the ones it binds. `VM.module_ctx` is now the single definition of a module's execution context, consulted by `_try_enter_module_call` on the way in and by the resolution above. Two hand-built copies of that tuple would be two answers to one question, which is how most rows of CLAUDE.md's recurring-shape table got there. `test_a_modules_context_is_built_in_one_place` asserts on the source so a second one fails the suite. Tests: 11 new cases in `tests/test_cross_module_closure.py`, each run both under the CLI and embedded -- top level, inside a step body, an upvalue-carrying adder, a stateful counter, nested in a list and in a map, a closure from a transitively imported module, two modules whose anonymous functions share a name, interleaved closures from two modules, and two controls (a caller-owned closure round-tripping unchanged, and `std:retry` still working). Verified against the unfixed tree: 11 tests go red, and both source assertions with them. One of those source assertions could not have failed as first written: the token `module_ctx` is a substring of `_capture_module_ctx`, so it matched the unfixed code too. Tightened to `self.module_ctx(` and re-checked against the unfixed tree. That is the fourth in-session unfalsifiable assertion this project has recorded; the check is cheap and it keeps catching them. Throughput: `module_ctx` builds a tuple where `_try_enter_module_call` used to assign fields directly. On 40,000 cross-module calls in a tight loop and nothing else, best-of-five is 3.045s against 2.929s baseline and 3.066s on a re-run -- about 4% on the minima, while the medians overlap and put the fixed tree lower. Worth it for one definition instead of two. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EdYjdCRLuedehwRCdi2hXb
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #696. The mirror of #691, found by probing the neighbourhood of that fix on the day it merged.
Why the two could not share a fix
This is the part worth reading, and it is a correction to the assumption I filed #696 with.
Every context source #691's fix uses records something a call is still inside of — a
_ClosureProxywrapped for an argument, a live cross-module frame, a caller VM. A returned closure is called after all three are gone: the frame popped when the module function returned, the proxy was for an argument, and at top level there is no caller VM. All three come back empty, and the closure runs at its own address in whatever chunk happens to be loaded.So finding a second instance of a shape is not the same as finding a second instance of the fix. Worth checking whether the mechanism you already wrote can even reach the new case before assuming symmetry — #476's "check the reverse direction" gets you to the bug, not to the repair.
What did generalise was the predicate.
_is_foreign_closuredetected the returned closure correctly, because #691's widening made it chunk-relative rather than_caller_vm-relative. Only the resolution was missing. A correct question with no available answer is a much better place to be than a question nobody asks, and that is an argument for widening a predicate past its immediate caller.Symptoms
Same five-way spread as #691, for the same reason — the symptom is whatever sits at that address. One repro each, measured on the unfixed tree:
Method calls are only supported on recordsrun_workflow(workflow) expects a workflowCannot add int and stringStack underflow'NoneType' object is not subscriptableNo workflow and no coroutine needed.
let f = m.plain_maker()infn main()was enough, which makes a factory function — an ordinary thing for a module to export — unusable.The fix resolves rather than marks
Marking a closure on the way out would mean a hook at each exit and a walk of returned lists, maps and records — the case #339 found the entry side had missed. Instead
VM._foreign_closure_originasks which of the modules this VM can reach owns the closure'sFunctionInfo. A module holds itsfunctionstable for its whole life, so the answer survives every frame being gone, needs no mark, and covers containers for free.Reachability, not a process-wide registry of every module ever loaded. A global would be the module-scope state shape behind #185 and #390, and it would let one tenant's VM resolve another's chunk. What a program can call, it can see — including transitively imported modules, reached through the ones it binds.
VM.module_ctxis now the single definition of a module's execution context, consulted by_try_enter_module_callon the way in and by the resolution above. Two hand-built copies of that tuple would be two answers to one question, which is how most rows of CLAUDE.md's recurring-shape table got there.test_a_modules_context_is_built_in_one_placeasserts on the source so a second one fails the suite.Tests
11 new cases in
tests/test_cross_module_closure.py, each run both under the CLI and embedded: top level, inside a step body, an upvalue-carrying adder, a stateful counter, nested in a list and in a map, a closure from a transitively imported module (mainbindsouter, onlyouterbindsinner), two modules whose anonymous functions share a name, interleaved closures from two modules, plus two controls — a caller-owned closure round-tripping unchanged, andstd:retrystill working.Also two unit tests on the resolution itself: that an unreachable module is not resolvable, and that a cycle between two modules terminates.
Falsified: 11 tests go red against the unfixed tree, and both source assertions with them.
One of those source assertions could not have failed as first written — the token
module_ctxis a substring of_capture_module_ctx, so it matched the unfixed code too. Tightened toself.module_ctx(and re-checked. That is the fourth in-session unfalsifiable assertion this project has recorded; the check is cheap and it keeps catching them.Testing
nodus_gategreen: closed-issues 3/3, opcodes 29/29, contracts 6/6, invariants 4/4, static 139/139, shapes 0 new, versions 13/13.Throughput:
module_ctxbuilds a tuple where_try_enter_module_callused to assign fields directly. On 40,000 cross-module calls in a tight loop and nothing else, best-of-five is 3.045s against 2.929s baseline and 3.066s on a re-run — about 4% on the minima, while the medians overlap and put the fixed tree lower. Worth it for one definition instead of two.🤖 Generated with Claude Code
https://claude.ai/code/session_01EdYjdCRLuedehwRCdi2hXb