Skip to content

fix(server): dispatch pipeline lifecycle in topological order, including control sub-pipelines - #1669

Merged
asclearuc merged 4 commits into
developfrom
fix/1650-open-closing-close-order
Jul 29, 2026
Merged

asclearuc merged 4 commits into
developfrom
fix/1650-open-closing-close-order

Conversation

@asclearuc

@asclearuc asclearuc commented Jul 24, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Per-object lifecycle (open/closing/close) was dispatched in .pipe canvas /
    input[] order, not dependency order, so a join node could be flushed before an
    upstream branch emitted its flush-time data — silently dropping it (the admin
    chat.pipe incident: embeddings never reached the vector store).
  • Lifecycle is now bound flat onto each region's root in topological order:
    closing/close upstream-first (a node flushes only after ALL its upstream parents),
    open reversed. The pipe head owns the source region; each control/invoke node (e.g.
    tool_pipe) owns the sub-pipeline it reaches
    and drives it the same way, to any
    nesting depth. Orders are computed once by computeLifecycleOrder (reachability + Kahn
    topo sort) and partitioned by computeLifecycleRegions.
  • tool_pipe now drives closing() before close(), so buffer-and-flush nodes in its
    sub-pipeline emit before the tool reads its result. A diamond sub-pipeline returns the
    merged output of all branches instead of dropping one.
  • Wirings that cannot honour one-owner-per-node are rejected when the pipeline opens:
    a sub-pipeline shared with the main flow or a second start; a node reached by two
    control nodes; an invoke node that is itself data-fed.

Behaviour notes for reviewers:

  • Policy: a node is flushed and closed only after every node that feeds it, and
    opened before them. In the diamond A -> J, B -> J:

    • closing/close: A, B, then J — the join is last
    • open: J, then B, A — the join is first, so it is ready before anyone writes to it

    This matches today's chain semantics, so existing linear pipelines are unchanged.

  • Control/invoke sub-pipelines are no longer left on per-parent bindings — this is the
    scope change versus the first review. Their nodes move onto the invoke node's own lanes.

  • The three rejections above are new hard failures at beginEndpoint / client.use().
    Normal agent -> tool_pipe -> sub-pipeline wiring is unaffected; invoking the same
    tool from two agents stays valid.

  • Rejection messages name nodes by service title and component id, e.g.
    Control node "Pipeline Tool" (pipe_tool_1) reaches node "Prompt" (sub_mid) ....

  • An error in a native node's closing now aborts the remaining flush walk (Python nodes
    already did; unifies semantics).

  • Ec::PreventDefault from a lifecycle callback no longer prunes a subtree (no production
    node uses it there).

  • Trace apaevt_flow: lifecycle frames appear flat under the pipe head, and under each
    control node for its sub-pipeline (pairs stay balanced; no protocol change).

  • buildConnections now clears the controls table (stale rows could survive a retried
    beginEndpoint).

Type

fix

Testing

  • Tests added or updated
    • store::lifecycleOrder — 22 sections / 31 assertions (per-region ordering)
    • store::lifecycleRegions — 12 sections / 20 assertions (partition + all three
      ownership rejections, nested invoke chains, multi-start topologies)
    • nodes/test/test_lifecycle_order.py — end-to-end guard for the closing order: a
      diamond of buffer-and-flush nodes whose join is declared before its branches, so
      declaration-order dispatch would flush it early and drop a branch. No LLM or mock.
  • Tested locally — store::linkages (38) and PipelineConfig (42) green, no
    regressions. Runtime verified against a live engine with instrumented sub-pipeline
    nodes: each node receives open/closing/close exactly once per invocation and
    the join flushes last, both single-level and two-level nested.
  • ./builder test passes

Scope of the new E2E: it asserts the closing order. open/close order and the
tool_pipe region are not covered by CI (driving a tool needs an agent + LLM); those are
covered by the unit tests, the manual runtime verification above, and the rejection
examples in examples/incorrect/.

Docs & examples

  • docs/agents/ROCKETRIDE_PIPELINE_RULES.md — lifecycle guarantee extended to control
    sub-pipelines, plus the three wiring constraints
  • docs/agents/ROCKETRIDE_COMMON_MISTAKES.md — new entry for sharing a sub-pipeline
  • nodes/src/nodes/tool_pipe/README.md — exclusive-ownership rule
  • packages/server/docs/observability.md — how sub-pipeline lifecycle frames appear
  • examples/tool-pipe-diamond.pipe, examples/tool-pipe-nested.pipe, and
    examples/incorrect/ (five rejected wirings + README)

Checklist

  • Commit messages follow conventional commits
  • No secrets or credentials included
  • Wiki updated (if applicable)
  • Breaking changes documented (if applicable)

Linked Issue

Fixes #1650

Summary by CodeRabbit

  • New Features

    • Improved pipeline lifecycle handling for deterministic, dependency-aware execution.
    • Ensured merge and join components receive all upstream outputs before flushing.
    • Added validation for invalid sub-pipeline wiring, including shared or conflicting ownership.
    • Added support and examples for nested and diamond-shaped sub-pipelines.
  • Bug Fixes

    • Fixed buffered sub-pipeline results being read before flush completion.
    • Ensured sub-pipelines close reliably even when flushing encounters an error.
  • Documentation

    • Expanded lifecycle, observability, troubleshooting, and invalid-pipeline guidance.

@github-actions github-actions Bot added docs Documentation module:server C++ engine and server components labels Jul 24, 2026
@github-actions

Copy link
Copy Markdown
Contributor
🤖 Internal: Discord sync marker

Auto-managed by the Discord notification workflow. Stores the linked Discord message ID and forum thread ID. Do not edit or delete.

@coderabbitai

coderabbitai Bot commented Jul 24, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

The engine now computes dependency-ordered lifecycle regions, binds open/closing/close accordingly, validates control-node ownership, and flushes tool sub-pipelines before reading responses. Tests and examples cover joins, nested pipelines, invalid wiring, and lifecycle tracing.

Changes

Pipeline lifecycle ordering

Layer / File(s) Summary
Lifecycle order computation and validation
packages/server/engine-lib/engLib/store/headers/endpoint.hpp, packages/server/engine-lib/engLib/store/stack.cpp, packages/server/engine-lib/test/store/pipeline/lifecycle_order.cpp
Adds deterministic topological lifecycle ordering, head/control region partitioning, cycle detection, ownership validation, stored endpoint state, and unit coverage.
Lifecycle lane binding
packages/server/engine-lib/engLib/store/core/endpoint/endpoint.pipes.cpp
Binds region-owned lifecycle lanes in dependency order while retaining deduplicated parent binding for unowned targets.
Sub-pipeline lifecycle execution and examples
nodes/src/nodes/tool_pipe/*, examples/tool-pipe-*.pipe, examples/incorrect/*
Flushes tool sub-pipelines before response reads, guarantees close after flush, and adds valid nested/diamond plus rejected ownership and wiring examples.
Lifecycle and ownership contracts
docs/agents/*, packages/server/docs/observability.md, examples/incorrect/README.md
Documents lifecycle ordering, flush guarantees, trace-frame structure, and control-node sub-pipeline ownership rules.
Flush-order end-to-end validation
nodes/test/test_lifecycle_order.py
Verifies a diamond join receives both branches’ flush-time outputs.

Estimated code review effort: 4 (Complex) | ~45 minutes

Suggested reviewers: jmaionchi, rod-christensen, stepmikhaylov, kwit75

Sequence Diagram(s)

sequenceDiagram
  participant PipelineInput
  participant IServiceEndpoint
  participant LifecycleRegions
  participant ToolPipe
  participant ResponseLane
  PipelineInput->>IServiceEndpoint: buildConnections
  IServiceEndpoint->>LifecycleRegions: compute dependency-ordered regions
  LifecycleRegions-->>IServiceEndpoint: return head and control orders
  IServiceEndpoint->>ToolPipe: dispatch open, closing, and close
  ToolPipe-->>ResponseLane: write flushed response
  IServiceEndpoint->>ResponseLane: read response after flush
Loading
🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Linked Issues check ⚠️ Warning The PR fixes source lifecycle order, but also changes control-sub-pipeline lifecycle and ownership rules that #1650 says belong in follow-up scope. Limit #1650 to source-connected data-flow ordering and move control-sub-pipeline lifecycle, ownership, docs, and examples to a separate PR.
Out of Scope Changes check ⚠️ Warning Docs, example pipes, and control-sub-pipeline region logic extend beyond the source-connected lifecycle fix requested by #1650. Trim those additions or split them into a follow-up so this PR only changes the dependency-ordered lifecycle dispatch.
✅ Passed checks (3 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main change: lifecycle dispatch is made topological, including control sub-pipelines.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/1650-open-closing-close-order

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@asclearuc
asclearuc requested a review from dsapandora July 24, 2026 07:27

@joshuadarron joshuadarron left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overview

Solid fix for a real bug. Lifecycle (open/closing/close) was dispatched in .pipe declaration order via per-parent binder chains, so a join node could flush before an upstream branch emitted its flush-time data. This binds the source-reachable nodes' lifecycle flat onto the pipe head in topological order — closing/close upstream-first, open reversed — leaving control/invoke sub-pipelines on their per-parent bindings.

I verified the two claims the whole design rests on, and both hold:

  • open downstream-first matches existing semantics. IPythonInstanceBase::open calls Parent::open (→ binder.open → children) before invoking the Python node's own open(), so children already opened first under the old chain. Reversing lifecycleOrder preserves that.
  • closing/close upstream-first matches existing semantics. IPythonInstanceBase::closing/close run the Python method first, then Parent:: → binder → children. Parent-before-child, same as the new flat order.

Also checked:

  • tool_pipe is the only production consumer of self.instance.open()/close() (nodes/src/nodes/tool_pipe/IInstance.py:98,106). It is wired to agents through control/invoke, never as a data input target, so neither it nor its sub-pipeline nodes are source-reachable — the exclusion works, and its per-parent bindings survive. packages/ai/src/ai/modules/data/data_conn.py:504 is the pipe head, unaffected.
  • The Ec::PreventDefault-no-longer-prunes note is accurate: every preventDefault() call in nodes/ is in a data lane (writeText, writeImage, writeDocuments, renderObject), none in open/closing/close.
  • pipeStack[0] is always the pipe filter (buildPipeStack pushes it first) and instanceStack is 1:1 with pipeStack, so lifecycleOrder indices can't collide with the head. bindFilters has exactly one caller, so the rewritten doc comment ("called when we create the initial filter stack") is correct.
  • No double-binding: the skip is keyed on toIndex, so a head-managed target is bound exactly once, on the head.
  • computeLifecycleOrder is deterministic and allocation-safe — unordered_map element references survive rehash, so for (int to : adj[node]) is fine even though adj[node] may insert.

Docs are updated per the co-located rule, CI is green on all three platforms, and the reasoning for skipping the E2E (a naive diamond with a response join is a false-green) is sound.

Two things I'd like addressed before merge — both small.


Requested changes

1. Binder::getBound() is dead code

packages/server/engine-lib/engLib/store/core/binder.cpp:59 + headers/binder.hpp:178

The accessor is added to the public Binder API, documented as "Used by tests to assert lifecycle lane contents/order" — and has zero callers. The only new test (lifecycle_order.cpp) exercises the pure computeLifecycleOrder function and never touches a Binder.

This is public surface added for a consumer that doesn't exist, which .claude/rules/architecture.md explicitly rules out ("Every file and folder exists because something depends on it right now"). Either use it (see #2) or drop it.

2. Nothing tests the actual bind order

All 10 new tests cover the ordering algorithm. Nothing covers the part of bindFilters that consumes it, which is where a silent regression would live:

  • closing/close bound on filters[0] in forward order, open in reverse
  • head-managed targets skip the per-parent lifecycle bind
  • non-reachable (control/invoke) targets keep it

That last one is the tool_pipe contract. If someone later drops the headManaged guard, computeLifecycleOrder still passes all 10 tests and tool_pipe breaks. getBound() is exactly the hook needed to assert this — please add the test it was written for, so the accessor earns its place.


Non-blocking nits

  • Bounds-check asymmetry (endpoint.pipes.cpp:145-159): the closing/close loop validates toIndex against filters.size(); the reversed open loop doesn't. It's safe only because loop 1 validates every entry before loop 2 runs — an implicit coupling that breaks the moment someone reorders or early-exits. Validate the whole lifecycleOrder once up front (or in buildConnections, against pipeStack.size()) and drop the in-loop check.
  • toIndex >= filters.size() compares int to size_t. Consistent with the pre-existing checks in this file and builds are clean, so noting only for consistency if you touch it per the point above.
  • headManaged allocation: the comment at stack.cpp says the order is precomputed "so bindFilters stays allocation-free per pipe", but bindFilters now builds an unordered_set per pipe stack. lifecycleOrder is small — a linear std::find would beat the hash set here anyway. Either way, the comment overstates it.
  • Kahn loop is O(N²) (stack.cpp): the inner scan restarts from the head of nodeOrder after each emit. Irrelevant at pipeline sizes and it runs once per endpoint, so fine — just flagging it's quadratic by construction, not by accident. A rolling start index would make it linear-ish if you ever care.
  • if (inDegree[to] > 0) inDegree[to]--;: with deduped edges and seeded in-degrees this guard can't fire. It converts a would-be logic error into a silently wrong order. An assert (or nothing) says more.
  • edges duplicates the information already in adj; the in-degree pass could walk adj directly and drop the extra vector.

Question

The mixed reachability node included after reachable parent test locks in a behavior change worth confirming: a node fed by both a source-reachable parent and a control/invoke node becomes head-managed, so its invoke node's self.instance.open()/close() no longer reaches it.

Concretely — a response node shared between the main pipeline and a tool_pipe sub-branch would stop being opened/closed per tool invocation. The tool_pipe README says to cap each branch with its own response node, so this is probably out of contract, but it's a wiring a user can produce on the canvas. Is that the intended trade-off? If so, worth a line in ROCKETRIDE_PIPELINE_RULES.md next to the new lifecycle guarantee.

Also: the ./builder test checkbox is still unchecked in the description.

Pipeline open/closing/close were dispatched in `.pipe` canvas order, not
dependency order, so a join node could be flushed before an upstream branch
emitted its flush-time data — silently dropping it.

Bind the source-reachable nodes' lifecycle onto the pipe head in topological
order instead: closing/close upstream-first (a node flushes only after all of
its upstream parents), open reversed. Control/invoke sub-pipelines are left
untouched — they keep their per-parent binding so their invoke node (e.g.
`tool_pipe`) can still open/close them.

The order is computed once by `computeLifecycleOrder` (reachability from the
source, then a Kahn topological sort over the reachable set; a cycle in the
reachable region is rejected). Adds unit tests for the ordering and documents
the guarantee.

Closes #1650
Generalize the open/closing/close dependency-order guarantee to a control node's
inline sub-pipeline (e.g. tool_pipe): it is flushed and closed in dependency order
per invocation, to any nesting depth, so a diamond sub-pipeline returns the merged
output of all branches. tool_pipe now drives closing() then close().

Reject at pipeline open the wirings that break exclusive sub-pipeline ownership:
sharing a sub-pipeline node with the main flow, a second start, or another control
node; and a data-fed invoke node.

Adds unit tests and updates pipeline-rules/observability docs and examples.
Report each node as its service title plus component id, e.g.
"Pipeline Tool" (pipe_tool_1), instead of a pipeStack index, so a rejected
wiring points at what the author sees on the canvas.

Document the exclusive-ownership rule in the tool_pipe README and add a
common-mistakes entry covering the three rejections. Examples switch to
agent_crewai and lane-specific response nodes.
@asclearuc
asclearuc force-pushed the fix/1650-open-closing-close-order branch from 411ec05 to 8eb14ff Compare July 27, 2026 20:04
@github-actions github-actions Bot added the module:nodes Python pipeline nodes label Jul 27, 2026
Remove `Binder::getBound`: public API added with no caller.

Range-check the lifecycle indices once up front rather than inside one of the
bind loops, so every walk is covered whatever order they run in, and correct
the comment that claimed bindFilters was allocation-free per pipe.

Add an end-to-end test for the closing order: a diamond of buffer-and-flush
nodes whose join is declared before its branches, so dispatching in declaration
order would flush the join early and drop a branch's output.
@asclearuc

Copy link
Copy Markdown
Collaborator Author

Overview

Solid fix for a real bug. Lifecycle (open/closing/close) was dispatched in .pipe declaration order via per-parent binder chains, so a join node could flush before an upstream branch emitted its flush-time data. This binds the source-reachable nodes' lifecycle flat onto the pipe head in topological order — closing/close upstream-first, open reversed — leaving control/invoke sub-pipelines on their per-parent bindings.

I verified the two claims the whole design rests on, and both hold:

  • open downstream-first matches existing semantics. IPythonInstanceBase::open calls Parent::open (→ binder.open → children) before invoking the Python node's own open(), so children already opened first under the old chain. Reversing lifecycleOrder preserves that.
  • closing/close upstream-first matches existing semantics. IPythonInstanceBase::closing/close run the Python method first, then Parent:: → binder → children. Parent-before-child, same as the new flat order.

Also checked:

  • tool_pipe is the only production consumer of self.instance.open()/close() (nodes/src/nodes/tool_pipe/IInstance.py:98,106). It is wired to agents through control/invoke, never as a data input target, so neither it nor its sub-pipeline nodes are source-reachable — the exclusion works, and its per-parent bindings survive. packages/ai/src/ai/modules/data/data_conn.py:504 is the pipe head, unaffected.
  • The Ec::PreventDefault-no-longer-prunes note is accurate: every preventDefault() call in nodes/ is in a data lane (writeText, writeImage, writeDocuments, renderObject), none in open/closing/close.
  • pipeStack[0] is always the pipe filter (buildPipeStack pushes it first) and instanceStack is 1:1 with pipeStack, so lifecycleOrder indices can't collide with the head. bindFilters has exactly one caller, so the rewritten doc comment ("called when we create the initial filter stack") is correct.
  • No double-binding: the skip is keyed on toIndex, so a head-managed target is bound exactly once, on the head.
  • computeLifecycleOrder is deterministic and allocation-safe — unordered_map element references survive rehash, so for (int to : adj[node]) is fine even though adj[node] may insert.

Docs are updated per the co-located rule, CI is green on all three platforms, and the reasoning for skipping the E2E (a naive diamond with a response join is a false-green) is sound.

Two things I'd like addressed before merge — both small.

Requested changes

1. Binder::getBound() is dead code

packages/server/engine-lib/engLib/store/core/binder.cpp:59 + headers/binder.hpp:178

The accessor is added to the public Binder API, documented as "Used by tests to assert lifecycle lane contents/order" — and has zero callers. The only new test (lifecycle_order.cpp) exercises the pure computeLifecycleOrder function and never touches a Binder.

This is public surface added for a consumer that doesn't exist, which .claude/rules/architecture.md explicitly rules out ("Every file and folder exists because something depends on it right now"). Either use it (see #2) or drop it.

2. Nothing tests the actual bind order

All 10 new tests cover the ordering algorithm. Nothing covers the part of bindFilters that consumes it, which is where a silent regression would live:

  • closing/close bound on filters[0] in forward order, open in reverse
  • head-managed targets skip the per-parent lifecycle bind
  • non-reachable (control/invoke) targets keep it

That last one is the tool_pipe contract. If someone later drops the headManaged guard, computeLifecycleOrder still passes all 10 tests and tool_pipe breaks. getBound() is exactly the hook needed to assert this — please add the test it was written for, so the accessor earns its place.

Non-blocking nits

  • Bounds-check asymmetry (endpoint.pipes.cpp:145-159): the closing/close loop validates toIndex against filters.size(); the reversed open loop doesn't. It's safe only because loop 1 validates every entry before loop 2 runs — an implicit coupling that breaks the moment someone reorders or early-exits. Validate the whole lifecycleOrder once up front (or in buildConnections, against pipeStack.size()) and drop the in-loop check.
  • toIndex >= filters.size() compares int to size_t. Consistent with the pre-existing checks in this file and builds are clean, so noting only for consistency if you touch it per the point above.
  • headManaged allocation: the comment at stack.cpp says the order is precomputed "so bindFilters stays allocation-free per pipe", but bindFilters now builds an unordered_set per pipe stack. lifecycleOrder is small — a linear std::find would beat the hash set here anyway. Either way, the comment overstates it.
  • Kahn loop is O(N²) (stack.cpp): the inner scan restarts from the head of nodeOrder after each emit. Irrelevant at pipeline sizes and it runs once per endpoint, so fine — just flagging it's quadratic by construction, not by accident. A rolling start index would make it linear-ish if you ever care.
  • if (inDegree[to] > 0) inDegree[to]--;: with deduped edges and seeded in-degrees this guard can't fire. It converts a would-be logic error into a silently wrong order. An assert (or nothing) says more.
  • edges duplicates the information already in adj; the in-degree pass could walk adj directly and drop the extra vector.

Question

The mixed reachability node included after reachable parent test locks in a behavior change worth confirming: a node fed by both a source-reachable parent and a control/invoke node becomes head-managed, so its invoke node's self.instance.open()/close() no longer reaches it.

Concretely — a response node shared between the main pipeline and a tool_pipe sub-branch would stop being opened/closed per tool invocation. The tool_pipe README says to cap each branch with its own response node, so this is probably out of contract, but it's a wiring a user can produce on the canvas. Is that the intended trade-off? If so, worth a line in ROCKETRIDE_PIPELINE_RULES.md next to the new lifecycle guarantee.

Also: the ./builder test checkbox is still unchecked in the description.

Thanks for the thorough review. The two semantics checks you did (open
downstream-first, closing/close upstream-first) were exactly the right things
to verify.

Please note the PR grew after your review. You reviewed 411ec05d. Three
more commits are on the branch now:

  • de913be1 — extend the topological lifecycle to control sub-pipelines
  • 8eb14ffb — name nodes in the ownership error messages
  • 4f9f7e4d — the fixes below, plus a bind-order test

This invalidates one of your conclusions. You wrote that tool_pipe and its
sub-pipeline keep their per-parent bindings. That is no longer true: extending
the same ordering guarantee into a control node's sub-pipeline is what the
new commits do. A re-review with that in mind would be appreciated.

Requested changes

1. Binder::getBound() is dead code — removed

You were right, and it was worse than a leftover: it was added by this PR
(7c9c9da8) with no caller, for a test that was never written. Declaration and
definition are both deleted.

2. Nothing tests the actual bind order — test added

New file: nodes/test/test_lifecycle_order.py. It runs a real pipeline — a
diamond of prompt nodes, which accumulate on write and emit on closing:

webhook -> branch_a -\
webhook -> branch_b --> join -> response

The join and its response are declared before the branches on purpose. A
dispatch that follows declaration order flushes the join first — the exact
failure you described.

Each branch reaches the join twice: once forwarded when it is written, once when
it flushes. So the join's merged question must hold 4 entries. If the join
flushes early, the branch flushes land on a closed node and the count drops.

No LLM, agent or mock is involved, so it is deterministic and runs in ~7s.

Three limitations, stated plainly:

  • It covers the closing order only. open and close order produce no
    observable data here, so they are not asserted.
  • It does not cover a control node's sub-pipeline (the tool_pipe region).
    Driving that needs an agent, which needs an LLM — too unstable for CI. That
    path is covered by manual runs and by the rejection examples in
    examples/incorrect/.
  • I have not yet demonstrated the test failing against a deliberately broken
    build. The counts above come from measured behaviour, not from a red run.

Nits applied

  • Bounds-check asymmetry — every lifecycle index is now range-checked once,
    up front, instead of inside one of the bind loops. This matters more than when
    you flagged it: the bind helper now runs once per region.
  • int vs size_t — fixed as part of that change.
  • "allocation-free per pipe" comment — corrected. It now says the order is
    computed once per endpoint, so errors surface at beginEndpoint.

Nits not applied, with reasons

  • Kahn loop is O(N²) — agreed it is quadratic. It runs once per endpoint on
    graphs of a few dozen nodes. Restructuring a correct topological sort for no
    measurable gain seems like the wrong trade.
  • edges duplicates adj — same reasoning. The algorithm works and is well
    covered; the tidy-up carries more risk than value.
  • if (inDegree[to] > 0) inDegree[to]--; — kept. You are right that it
    cannot fire today. It costs nothing and prevents an int underflow if the
    dedup invariant ever changes.

Happy to revisit any of these three if you disagree.

@asclearuc
asclearuc requested a review from joshuadarron July 27, 2026 21:18
@asclearuc asclearuc changed the title fix(server): dispatch pipeline lifecycle in topological order fix(server): dispatch pipeline lifecycle in topological order, including control sub-pipelines Jul 27, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
packages/server/engine-lib/engLib/store/stack.cpp (1)

332-345: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Reject every data-fed control root with a sub-pipeline.

Lines 337-345 use ownerOf to infer whether the control root is data-fed, but ownerOf only records nodes already claimed by lifecycle regions. A sub-pipeline root fed by an unowned node is accepted, then receives a per-parent lifecycle binding in bindFilters while its descendants are also flat-bound from the invocation root, allowing duplicate lifecycle execution and dropped/duplicated flush output.

Track incoming data targets directly and reject region.first whenever it appears as a connection target.

The PR objective explicitly requires rejecting control roots that own a region and are also data-fed.

Proposed fix
+    std::unordered_set<int> dataFed;
+    for (const auto &conn : connections)
+        dataFed.insert(std::get<1>(conn));
+
     for (const auto &region : regions.control) {
-        auto it = ownerOf.find(region.first);
-        if (it != ownerOf.end())
+        if (dataFed.count(region.first))
             return APERR(Ec::InvalidParam, "Control node", nameOf(region.first),
-                         "drives a sub-pipeline and is also data-fed (owned by",
-                         nameOf(it->second),
-                         "); an invoked node with a sub-pipeline must not take a "
-                         "data input");
+                         "drives a sub-pipeline and is also data-fed; an "
+                         "invoked node with a sub-pipeline must not take a "
+                         "data input");
     }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/server/engine-lib/engLib/store/stack.cpp` around lines 332 - 345,
Update the control-root validation in the region-checking loop to use the
incoming data-connection targets rather than ownerOf, since unowned data-fed
nodes are currently missed. Track or query those targets directly and reject
every region.first that is a data connection target, while preserving the
existing InvalidParam error behavior and message context.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@packages/server/engine-lib/engLib/store/stack.cpp`:
- Around line 332-345: Update the control-root validation in the region-checking
loop to use the incoming data-connection targets rather than ownerOf, since
unowned data-fed nodes are currently missed. Track or query those targets
directly and reject every region.first that is a data connection target, while
preserving the existing InvalidParam error behavior and message context.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 34b3ff4e-5fa8-4ea3-bf6d-752a3be6d2e4

📥 Commits

Reviewing files that changed from the base of the PR and between 8eb14ff and 4f9f7e4.

📒 Files selected for processing (3)
  • nodes/test/test_lifecycle_order.py
  • packages/server/engine-lib/engLib/store/core/endpoint/endpoint.pipes.cpp
  • packages/server/engine-lib/engLib/store/stack.cpp

@dsapandora dsapandora left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

solid work — the flat-per-region binding is the right model, and I verified it end to end: Binder::callMethods dispatches methodMap[lane] in bind order, and IServiceFilterInstance::open/closing/close are pre-order, so with region-owned nodes no longer bound per-parent there's no double dispatch. The controls.clear() in buildConnections and the try/finally in tool_pipe are nice catches too.

@asclearuc
asclearuc merged commit 27d61e6 into develop Jul 29, 2026
35 checks passed
@asclearuc
asclearuc deleted the fix/1650-open-closing-close-order branch July 29, 2026 08:38
dk-rocketride added a commit that referenced this pull request Sep 23, 2026
* fix(server): bind Python instance.closing() to cb_closing (#1667)

`IServiceFilterInstance`'s `closing` method was bound to `cb_close` — the same
callback as `close` — so `self.instance.closing()` walked the `close` lane
(teardown) instead of the `closing`/flush lane. The correct callback,
`cb_closing`, was implemented but bound to nothing, i.e. dead code.

Bind `closing` to `cb_closing` so a node can drive flush on its downstream.
No behavior change today (no node calls `instance.closing()`), but it removes a
correctness landmine and unblocks control-node sub-pipeline flush (e.g.
`tool_pipe`).

Closes #1666

* feat(nodes): tool_filesystem pipeline-sink lanes (#1651)

* feat(nodes): tool_filesystem sink lanes + filename handling

Split the tool_filesystem node changes out of the feat/http-mcp branch so
they can land independently of the HTTP MCP module. Adds pipeline-sink
behavior (write pipeline output to the scoped FileStore) and filename
handling, with unit tests that stub the engine/ai layer.

No changes to the FileStore/account package — the node consumes the
existing FileStore API only.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(nodes): address CodeRabbit review on tool_filesystem sink

- IGlobal._sink_config: clamp urlExpiresIn lower bound too — non-positive
  values now fall back to the 3600s default instead of passing through.
- services.json: add minimum/maximum (1..3600) bounds to filesystem.urlExpiresIn.
- test_sink_naming: cover negative/zero urlExpiresIn -> default.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* refactor(nodes)!: rename tool_filesystem node to filesystem

The node is no longer just an agent tool — it now also acts as a pipeline
sink (input/output lanes). Drop the tool_ prefix and rename the node to
`filesystem`, which aligns with its existing title ("File System"),
file-system.svg icon, and filesystem.* config namespace (no config-key churn).

BREAKING CHANGE: the node protocol/identity changes from `tool_filesystem://`
to `filesystem://` and the module path from `nodes.tool_filesystem` to
`nodes.filesystem`. Pipelines referencing the old identity must update.

- git mv nodes/src/nodes/tool_filesystem -> nodes/src/nodes/filesystem
- git mv nodes/test/tool_filesystem     -> nodes/test/filesystem
- services.json: protocol + path updated; description now covers both the
  agent-tool and pipeline-sink roles.
- Internal refs, log prefixes, README, and test module paths updated.

Generated docs under packages/docs/static (gitignored) regenerate from the
node source. A few unrelated files carry stale prose mentions of the old
name (rocketride_db.py, test_tool_tavily.py, test_tool_args.py) — left
untouched here to keep this change node-scoped.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(nodes): address review feedback on tool_filesystem sink; revert rename

Reverts the tool_filesystem -> filesystem rename. Renaming the node changes the
provider identity stored in saved .pipe files, which needs the engine-side
migration hop the graph rename got. That is out of scope here, so the rename
will land as its own PR together with the hop. This PR is purely additive on
top of develop again, contained to nodes/.

Review fixes applied:

- Per-lane filename resolution. Each lane owns its rule instead of one
  precedence chain where the source extension always won: text/table are
  markdown so they store .md; documents keep the source extension (else .txt);
  media derive from mime, then source extension, then .bin. Mime->extension now
  handles compound subtypes (image/svg+xml -> .svg, not .svg+xml) via an
  override map plus a suffix-stripping fallback.
- Streamed media. image/audio/video chunks stream straight to the store via
  open_write/write_chunk/close_write, so memory is bounded regardless of file
  size. The handle opens lazily on the first non-empty chunk, so an empty
  stream writes no file, and a fresh BEGIN discards a half-written prior
  stream instead of leaking it into the next object's file.
- Deterministic, collision-free store keys (targetDir + <objectId>/ + name,
  with an index for multi-document objects). Removes the stat-probe loop, which
  also closes the non-atomic write race and the unbounded-loop concern.
- Emission correctness: DocMetadata(self, ...) so objectId/nodeId/permissionId/
  signature are inherited, a distinct chunkId per reference so vector stores
  keyed on objectId+chunkId no longer overwrite each other, and
  preventDefault() on every lane so downstream receives the emitted references
  rather than the originals as well.
- Path whitelist is validated before anything touches the store, so a rejected
  path no longer probes for existence.
- Config robustness: non-numeric urlExpiresIn falls back to the default instead
  of raising out of beginGlobal; endGlobal resets all three sink fields.
- classType reordered to ["store", "tool"] to match the codebase convention.
  Order is not behavioural: the engine keys a controller map by the classType
  string and all UI reads are membership tests, so nothing depends on position.
- Tests rewritten and de-rigged: named objects now exercise the real lane
  semantics (report.pdf through the text lane must store .md), plus streaming
  media, empty-stream, abort-on-new-BEGIN, chunkId, preventDefault, and the
  previously missing writeVideo case. 36 pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(nodes): capped collision probe, per-lane extensions, per-object reset for tool_filesystem sink

Review feedback from dsapandora and asclearuc on PR #1651:

- collision handling: keep clean names (output/report.txt), append _1, _2, ...
  only on real collision, bounded by MAX_COLLISION_SUFFIX (100) then raise;
  every candidate is whitelist-checked before any stat probe
- documents lane always stores .txt: page_content is parsed text, so the
  stored extension stays truthful about the bytes (same rule as text/table .md)
- _ext_from_mime rejects unmapped vnd.* subtypes instead of emitting junk
- open() per-object hook: aborts media streams the prior object left
  unfinished (closes stale handle, deletes partial file) and restarts chunkIds
- _run_async docstring names the sink lane handlers as a supported caller
- tests: 47 passing (blank/unmappable mime fallbacks, conflicting mime,
  emitUrl on emitted Doc metadata, open() reset, whitelist-before-probe)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(nodes): make tool_filesystem sink tests environment-proof

Under ./builder nodes:test the suite runs in the engine python with
pytest-xdist, where two things differ from the bare venv: other node
suites install bare rocketlib stubs (some with an empty IInstanceBase),
and the real pydantic ai.common.schema is importable. Whichever test
file ran first in a worker decided which base class our node inherited,
producing 12 AttributeError failures in CI (no preventDefault on the
foreign stub; no IEndpoint for the real DocMetadata).

- judge rocketlib.IInstanceBase by capability (has preventDefault), not
  presence, so foreign empty stubs are replaced while the real class and
  our own stub are kept
- wire concrete currentObject fields and IEndpoint.endpoint.jobConfig so
  the real pydantic DocMetadata constructs cleanly under the builder
- pin the PREVENT_DEFAULT sentinel at instance level so tests behave the
  same whether the stub base (returns) or the real base (raises) won

Verified: bare venv (47 passed), poisoned ordering with tool_mem0 first
(59 passed), engine python with real ai + client-python on the path
(47 passed).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(nodes): keep tool_filesystem importable under minimal rocketlib stubs

The Entry import added for the open() hook broke test_read_size_cap.py
at collection: its minimal rocketlib stub predates Entry. Entry is
annotation-only, so import it under TYPE_CHECKING (PEP 563 lazy
annotations make the runtime import unnecessary).

Also complete that stub with rocketlib.warning: importing the real
ai.common.utils fresh under the stub always needed it transitively
(tool_args), and only ever passed via module-cache ordering luck in the
xdist worker.

Verified: read_size_cap alone in a fresh engine python (19 passed), the
whole test/tool_filesystem dir under engine python with the real ai +
client-python on the path (66 passed), bare venv with poisoned stub
ordering (59 passed).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

* fix(llm_anthropic): disable extended thinking at the node as an interim stop-gap (#1681)

Enabling extended thinking makes Anthropic reasoning models return content as typed
blocks (thinking + text) that the agent / expectJson path cannot consume, which caused
the production incident (#1658). Gate thinking activation behind a node-level constant
instead of flipping capabilities.reasoning, since that flag is stamped by
tools/sync_models and re-stamps to true on the next sync. Flip the constant back to
re-enable once thinking is normalized behind the adapter (#1679). The flatten in #1659
stays as defense-in-depth for that point.

Fixes #1680

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(endpoints): unique stable per-pipe endpoint URLs ({host}/{type}/{project_id}/{source}) (#1557)

* fix(endpoints): unique stable per-pipe endpoint URLs

Advertised chat/dropper/webhook URLs now embed the task identity as
{host}/{type}/{project_id}/{source}; additive path routes accept data
POSTs and resolve the task from the path. Legacy routes and ?auth=
resolution unchanged.

Fixes rocketride-ai/rocketride-saas#262

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(task-http): only treat missing tasks as token-resolution fallback

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(task): guard identity placeholders for partially-initialized tasks

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Krish Garg <krishgarg@Krishs-MacBook-Pro.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

* feat(ai): every subprocess exposes /task/data via a shared web server (#912)

* test(ai): characterize lazy-target contract for modules/data

Add `test_lazy_target.py` (15 tests) pinning the contract that
the shared-subprocess-web-server work needs from `modules/data`:

- `initModule` tolerates `state.target` missing or `None` at load time.
- `DataServer._target` reads `state.target` lazily on each access
  (picks up writes made by source nodes after the module is loaded).
- `DataConn.__init__` accepts `target=None` without dereferencing;
  thread count falls back to the existing default (4).
- `DataConn._require_target()` returns the target when set, or raises
  a controlled `RuntimeError` (not the bare `AttributeError` from
  `None.putPipe()`) when no source has registered one.

These tests fail against the current production code. The next commit
makes them pass.

* refactor(ai): tolerate `state.target=None` in modules/data

Make `modules/data` safe to eager-load before any source node has
registered a target. Needed so the shared subprocess web server (in a
follow-up commit to `node.py`) can expose `/task/data` from the moment
the subprocess starts, regardless of pipeline shape.

Changes:

- `__init__.py`: `initModule` no longer reads `state.target` eagerly.
  `DataServer` now receives the `server` reference and reads the target
  lazily via property.
- `data_server.py`: `DataServer.__init__(server, **kwargs)` stores the
  server reference; new `_target` `@property` returns
  `getattr(server.app.state, 'target', None)` on each access, so source
  nodes that write `state.target` later are seen automatically.
- `data_conn.py`: `__init__` guards `target.taskConfig.get(...)` for
  `target=None` (defaults to 4 threads); new `_require_target()` helper
  raises a controlled `RuntimeError` instead of the bare
  `AttributeError` from `None.putPipe()`. The 3 deref sites
  (`_cleanup_pipe`, `open_sync` allocate, `open_sync` error path) now
  route through it.

Behavior for source-bearing pipelines (webhook, telegram) is unchanged:
they still set `state.target` before any data flows, so the lazy read
returns the same target and the `_require_target` guard passes.

Test coverage: 42/42 in `tests/ai/modules/data/` (27 pre-existing + 15
new from `test_lazy_target.py`); 370/370+2 skipped across
`tests/ai/modules/`. No regressions.

* feat(ai): shared web server in node.py for every subprocess

Bootstrap a FastAPI/Uvicorn WebServer in `node.py` whenever EaaS spawns
the subprocess with `--data_port=N` (which it always does, see
`task_engine.py:1558-1565`). The server runs on the existing
`server_loop` daemon thread alongside `processArguments`, so EaaS can
reach every subprocess on `ws://localhost:N/task/data` for DAP traffic
regardless of pipeline shape — including agentic / sourceless
pipelines that previously had no listener at all.

Changes:

- `_setup_shared_web_server()`: parses `--data_host`/`--data_port` from
  `sys.argv`; returns `(None, None)` if absent (legacy / test
  invocations). Otherwise constructs the WebServer, calls
  `.use('data')` to register `/task/data` (which carries
  `rrext_cprofile_*` DAP commands as of PR #803), schedules `serve()`
  on `server_loop` via `asyncio.run_coroutine_threadsafe`, and blocks
  on a `threading.Event` set by an `on_startup` lifespan callback so
  the listener is guaranteed up before `processArguments` runs. A 10s
  timeout (`_SHARED_SERVER_STARTUP_TIMEOUT_SECONDS`) safety-nets a
  stuck startup.
- `_teardown_shared_web_server()`: called from `run()`'s `finally`
  block. No-op when no server was constructed. Otherwise calls
  `server.stop()` (flips uvicorn's `should_exit` flag) and awaits the
  `serve()` future with a 5s budget. Exception-safe — failures here
  log via `debug` and never mask the original `processArguments`
  error.
- `run()`: wires the helpers in between `_start_event_loop()` and the
  `rocketlib` import; tears down in `finally` before
  `_stop_event_loop()`. Exposes `shared_web_server` as a module-level
  attribute (via `global`) so source nodes can discover it later
  through `from ai.node import shared_web_server`.

Tests (`tests/ai/test_node.py`, 14 cases): argv parsing, WebServer
construction, `.use('data')` call, `serve()` scheduling on
`server_loop`, the startup-event handshake (both the happy path and
the timeout-without-signal path), and the teardown contract including
exception swallowing.

Intermediate state: Phases 3 and 4 (webhook and telegram source-node
refactors) still pending; until then, webhook/telegram pipelines
launched via EaaS will fail with `EADDRINUSE` because the source
nodes' own `_run()` still tries to bind the same `data_port`. Agentic
pipelines work correctly and gain DAP reachability today.

Test coverage: 14/14 in `tests/ai/test_node.py`; 941/941+2 across
`tests/ai/`. No regressions.

* fix(ai): mirror `shared_web_server` from `__main__` to `ai.node`

`engine.exe` spawns `node.py` as a script, so Python loads the file
as `sys.modules['__main__']`. The `global shared_web_server` write
in `run()` therefore lands on `__main__.shared_web_server`. Source
nodes later do `from ai import node as ai_node` and Python imports
the same file AGAIN as `sys.modules['ai.node']` — a separate module
with `shared_web_server = None`. Source nodes saw `None`, fell back
to the legacy self-hosted path, and the data plane was torn between
two competing WebServers (target set on one `app.state`, read from
another).

Mirror the assignment (and the `None` reset in `finally`) into
`sys.modules['ai.node']` so the standard import path returns the
live value. Runs once on the main thread before `processArguments`
spawns C-engine threads; GIL guarantees atomicity, no lock needed.

* feat(nodes): webhook + telegram use shared subprocess web server

Both source nodes drop their own `WebServer` + `.run()` and instead
discover `ai.node.shared_web_server` (bootstrapped by `node.py` from
the `--data_port` flag EaaS passes to every subprocess). They set
`state.target = self.target`, register their routes/modules on the
shared server, and block on a `threading.Event` to preserve the C
engine's `scanObjects()` contract (lifetime == pipeline lifetime).

webhook:
- shared-server path uses sync lifespan helpers
  (`_emit_ready_status_sync` / `_emit_shutdown_status_sync`) so we
  don't spin up a one-shot `asyncio.run` loop that closes before
  the C engine's threads can use it.

telegram:
- `_startup` / `_shutdown` do real async work (aiohttp session,
  polling task), so they're scheduled on `server_loop` via
  `asyncio.run_coroutine_threadsafe(...).result()` instead of the
  webhook's sync-helper trick.
- webhook-mode adds `/telegram/webhook` POST on the shared server.
- polling-mode unchanged behaviourally.

Legacy `_run_legacy_self_hosted_server()` retained for direct CLI
invocations without `--data_port`; emits a `debug(...)` marker on
entry so production logs make it obvious if it's ever exercised
(EaaS always passes `--data_port`, so it shouldn't be).

Tests:
- nodes/test/webhook/test_endpoint.py — 9 unit tests, both paths
- nodes/test/telegram/test_endpoint.py — 12 unit tests, both modes
  and both paths

* fix(ai): alias `ai.node` to `__main__` instead of re-importing

`import ai.node` from inside the `__main__`-loaded `node.py` triggered
a fresh module evaluation — creating a second `server_loop` daemon
thread. Source nodes then saw `ai.node.server_loop` (loop B) while
`WebServer.serve()` was scheduled on `__main__.server_loop` (loop A),
breaking cross-loop scheduling.

Replace the `import` with `sys.modules.setdefault('ai.node',
sys.modules[__name__])` — pure dict aliasing, no second evaluation,
both names point at the same module object.

* fix: correct installation of dependencies

* fix(ai): fail fast on shared `serve()` death; fix test_node race

Production: if `uvicorn.serve()` raises during bind (port collision,
permission denied), `startup_ready` never fires and the 10s timeout
elapses silently, publishing a dead `shared_web_server`. Check
`future.done()` after the wait and call `future.result()` to re-raise.

Test: `test_setup_blocks_until_on_startup_fires` assumed the fake
WebServer had captured `on_startup` after a timed `join()` — racy on
slow CI. Add a `threading.Event` set by the fake server so the test
waits on a signal rather than a guess.

* fix(nodes,ai): propagate telegram `_startup` failures; cover serve()-died branch

- `telegram/IEndpoint.py`: re-raise after the `_startup` try/except so
  startup failures (network, TLS, aiohttp) surface to the engine instead
  of leaving the thread stuck on `_shutdown_event.wait()`.
- `tests/ai/test_node.py`: add regression test for the fail-fast branch
  in `_setup_shared_web_server` — pre-failed future whose `.result()`
  raises must propagate.

* fix(telegram): raise on `_startup` soft-fail paths (missing token, webhook fail)

`_startup` previously returned normally for fatal startup states like
a missing bot token or a failed webhook registration, leaving the
source thread stuck on `_shutdown_event.wait()` with a non-functional
bot. The `raise` in commit `1463bd2b` only fires when `_startup` itself
raises — it can't help the silent-return paths.

Replace `return` with `raise RuntimeError(...)` after `monitorStatus`
for both fatal paths so `_run`'s existing try/except re-raises out of
the source-node thread. Add two unit tests that exercise `_startup`
directly.

* fix(ai): make `DataConn._target` lazy to close race

* test(telegram): stub `aiohttp` for webhook-setup-fail test

`_startup` does `import aiohttp` + `aiohttp.ClientSession()` before reaching
the webhook check the test wants to exercise. In production this works
because `IGlobal.beginGlobal()` ran `depends(requirements.txt)` first.
Unit tests bypass that lifecycle, and the broader test framework
intentionally mocks `depends` to a `MagicMock` in `test_contracts.py`'s
`mock_engine_libs` fixture — so we can't even rely on `beginGlobal()`
as a test setup. The test doesn't actually need real aiohttp (the
session is never used because `_setup_webhook` is stubbed), so stub
`sys.modules['aiohttp']` via `monkeypatch.setitem` for the test's
duration. Same pattern `test_contracts.py:579-580` uses for engine deps.

* test(ai): disconnect `DataConn` in async lazy_target tests

CodeRabbit flagged that 5 async tests construct `DataConn` (which spawns
`_monitor_task` via `asyncio.create_task`) but never call `disconnect()`,
leaking pending tasks at teardown. Wrap each `DataConn` use in
`try/finally` and `await conn.disconnect()` so the monitor task is
cancelled cleanly. Production code unchanged.

* refactor(ai,nodes): post-PR cleanups around shared subprocess web server

- `node.py`: extract `_parse_data_host_port` helper — single source of
  truth for `--data_host`/`--data_port` parsing across the shared-server
  bootstrap and the webhook/telegram legacy fallback paths.
- `node.py`: magic numbers.
- `webhook/IEndpoint.py`: plain sync  `_startup` / `_shutdown`.
- Sweep over inline comments added during the PR — keep only the
  load-bearing "WHY" (race windows, ordering constraints, gotchas);
  drop obvious narration of what the code does.

No production behaviour change. Tests: 980 ai + 31 webhook/telegram
all pass locally.

* refactor(ai,nodes): drop legacy fallback + defer DataConn semaphore init

- node.py: `_setup_shared_web_server` raises on missing `--data_port`
  (EaaS supplies it unconditionally; the legacy "no shared server"
  path is dead).
- webhook/telegram: delete `_run_legacy_self_hosted_server` and the
  `if shared is None` branch in `_run()`.
- data_conn.py: defer semaphore + threadCount init to first `_open`
  via `_ensure_pipe_sem` — folds the race-window wait with the
  threadCount read so the user's config is no longer silently ignored.
- Tests updated; legacy-path tests deleted.

* fix(ai): lazy-init pipe semaphore in DataConn tool handler

The standalone client.tool() path opens no pipe, so `_open` never runs
`_ensure_pipe_sem`, leaving `_pipe_sem` None. Route the borrowed-pipe
acquire through `_ensure_pipe_sem`, matching `_open`.

* fix(ai,nodes): make shared web server optional; guard source-node access

_setup_shared_web_server() returns (None, None) instead of raising when
--data_port is absent, so a subprocess with no data plane (e.g. a
catalog-mode scan) runs rather than failing at startup. This realigns the
code with the contract its own test docstring already documented.

webhook/telegram resolve the server through require_shared_web_server(),
which raises a named error citing the node and --data_port — replacing a
bare 'NoneType' AttributeError surfacing out of scanObjects.

Correct the webhook/telegram READMEs, which still described each node
creating its own FastAPI server on a default port 5567; both untrue since
the server moved into node.py.

* fix(ai): bind shared /task/data channel to 127.0.0.1 not localhost

The name `localhost` resolves to both 127.0.0.1 (IPv4) and ::1 (IPv6),
so asyncio binds one socket per address family and a client can dial
the wrong one, producing intermittent ECONNREFUSED. Bind and dial the
shared subprocess web server's data channel on 127.0.0.1 explicitly,
matching the engine control-port convention.

* test(ai): cover standalone tool call's lazy semaphore init

`_tool` borrows a pipe instead of opening one, so `_open` — the only
caller of `_ensure_pipe_sem` — never runs on that path. The existing
tests exercised `_ensure_pipe_sem` directly and so missed the caller
that skipped it, letting a `None` semaphore reach `.acquire()`.

Both tests fail with "'NoneType' object has no attribute 'acquire'"
when the lazy init is removed from the borrowed path.

* feat(database): Sequelize ORM over pipes + server-side DB transactions (#1467)

* feat(database): server-side multi-statement transaction sessions

Add a TransactionRegistry that holds one pooled DB connection open per session_id (bounded, idle-reaped, connection-leak-guarded), wire it into the DatabaseGlobalBase lifecycle, and expose begin/commit/rollback tool functions plus a session-aware execute with server-side $n parameter binding. All are gated by allow_execute. Includes SQLite-backed unit tests and db_postgres/db_mysql doc updates.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(database): SDK transaction methods + Sequelize ORM (client.database.sequelize)

TS and Python clients gain beginTransaction/commit/rollback and query() session/params. Add a pg-compatible shim injected as Sequelize v6's dialectModule (transaction-control SQL drives the pipe's begin/commit/rollback), a createSequelize() factory, and client.database.sequelize({ Sequelize, ... }) that injects the Sequelize class so the core client uses type-only imports and stays browser-bundle-safe (sequelize is an optional peerDependency). Includes unit + integration tests and the SDK doc.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(database): end-to-end transaction + Sequelize smoke tests

Gated (ROCKETRIDE_E2E=1) end-to-end tests exercising the full client -> pipe -> engine -> Postgres round-trip: multi-statement transaction isolation/commit/rollback via the raw tool functions and via client.database.sequelize(). Verified green against a live engine + Postgres.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(database): address PR review — per-session tx locks, savepoint interception, test/doc coverage

- tx_registry: per-session locking so different sessions run concurrently; the idle reaper skips in-flight sessions (was one global lock held across conn.execute()).
- pg-shim: anchor the transaction-control regex so `ROLLBACK TO SAVEPOINT` (nested rollback) is forwarded as an ordinary query rather than intercepted as a top-level ROLLBACK.
- e2e: close the Sequelize instance in a finally block so each test releases its RocketRide session.
- Tests: session-invalidated-after-rollback, multi-digit ($1..$11) placeholder guard, reaper-skips-in-flight, tool-not-called on validation failure, ROLLBACK TO SAVEPOINT pass-through.
- db_postgres/db_mysql READMEs: document the session-aware query(session_id/sessionId, params) API.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* style(database): ruff-format db_instance_base.py (CI ruff 0.15.20 parity)

* fix(database): roll back on max_execute_rows overflow in stateless execute

_executeRawQuery() logged and returned None inside engine.begin() when a
SELECT/RETURNING overflowed max_execute_rows, so a write still committed while
execute() reported failure (duplicate-retry risk). Raise inside the transaction
so it rolls back — mirroring the session path. Adds a regression test.

Reported by CodeRabbit.

* fix(database): roll back session on a failed session-bound execute

A session-bound execute that raises (e.g. max_execute_rows overflow) left the
held connection pinned until idle-reaping, with the aborted statement still
committable. Roll the session back before re-raising to release the connection
and discard the statement. Adds a regression test.

Reported by CodeRabbit.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(shell-ui): surface dashboard fetch failures instead of loading forever (#1673)

* fix(shell-ui): surface dashboard fetch failures instead of loading forever

_fetchDashboard() caught every error and only console.log'd it, leaving _data
null. Consumers render their loading state whenever data is null, so a failed
fetch was indistinguishable from a slow one and the view sat on "Loading
dashboard data..." indefinitely.

That is what the "Dashboard hangs on Loading" report was (rocketride-saas #373):
the request was returning Permission 'task.monitor' denied and the UI simply
never said so.

Track the failure as first-class state, clear it on success, phrase permission
denials in plain language, and expose it as `error` on the hook so views can
render it. Adding a field is backward compatible — the existing consumer
destructures a subset and is unaffected.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015nTVr6jfSFYm1GppxbjghP

* fix(shell-ui): don't double up the denial wording; clear data on denial only

Both of @anandray's points.

**Doubled wording.** The raw message already says "permission denied" in
exactly the case the friendly sentence exists to explain, so the two
concatenated into "You do not have permission to view the dashboard. Permission
denied". Replaced rather than appended; the verbatim text still goes to the
console line directly above for anyone debugging.

**_data retention — answering rather than confirming.** The retention was
intentional for transient failures and should stay: a poll that blips must
leave the last good numbers on screen with an error beside them, not blank the
dashboard every time the network hiccups. Going empty on each failed poll
would be its own bug.

But it was wrong to apply that uniformly. A denial is not a blip — it means
this user is not entitled to these numbers, and continuing to show a previous
session's data under an "access denied" banner is both confusing and the wrong
default for something access-scoped. So `_data` is now cleared on denial only,
and the split is spelled out in the comment so the next reader does not have to
guess which half was deliberate.

Note the existing 104 tsc errors in apps/shell-ui are unrelated (unbuilt
workspace deps); count is identical with and without this change, and none are
in this file. CI does not typecheck shell-ui, so this was checked locally.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015nTVr6jfSFYm1GppxbjghP

* fix(shell-ui): classify a denial from the structured error, not the prose

Two unresolved CodeRabbit threads. The second is the substantive one and it
breaks this PR on its own terms.

Denial detection was /denied|permission|forbidden|403/ over err.message. But
getDashboard() goes through call(), so a structured DAP failure arrives as
DAPException with the detail in `dapResult` — the human-readable message is not
guaranteed to carry it. And even on the text path the pattern had 403 but not
401 and not "unauthorized", so a plain `Unauthorized` fell through as a
transient blip.

Either way the result is the branch failing to fire for exactly the case it
exists for: a real denial would keep rendering the previous session's numbers
under an "access denied" banner, which is the behaviour this PR is meant to
remove.

Now reads dapResult.status / statusCode / code first and falls back to the
text, with 401 and unauthori[sz]ed added. Checked the classification against
both: Unauthorized, unauthorised, Not authorized, 403 Forbidden, bare 401 and
"failed with status 401" all classify as denied; timeout and network error stay
transient, so cached data still survives a blip.

Also corrected the snapshot comment, which claimed the object is replaced "only
when data or events change" — it is replaced on error too.

Related: #1673.

* fix(shell-ui): treat symbolic DAP denial codes as denials

CodeRabbit caught a real hole in this PR. `Number()` over dapResult.code turns
a symbolic 'UNAUTHORIZED' or 'FORBIDDEN' into NaN, so neither status check
fired, and the prose regex only ever saw `message`. With a generic human
message — which is exactly the case this branch was written for, since the
structured detail is not guaranteed to reach the text — `denied` stayed false.

The consequence is the specific bug this PR is meant to kill: `_data` is
deliberately not cleared on failure so a network blip leaves the last good
numbers on screen, so a missed denial keeps rendering the previous session's
figures under a vague error instead of "you do not have permission".

Now matches the code as a string as well. The pattern is lifted to a named
const rather than duplicated, and carries no `g` flag, so reusing it across two
.test() calls has no lastIndex statefulness.

Verified with tsc against apps/shell-ui/tsconfig.json: no errors in this file.
(The remaining TS2307 "Cannot find module 'rocketride'" errors are the known
unbuilt client-typescript dist/types artifact, not this change — and worth
saying out loud that CI would not have caught a type error here either, since
nothing typechecks apps/** on a PR.)

Refs #1673.

---------

Co-authored-by: Dmitrii Kataraev <dmitrii.kataraev@rocketride.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

* fix(auth): log why has_permission denies instead of swallowing silently (#1670)

* fix(auth): log why has_permission denies instead of swallowing silently

resolve_team_permissions() raises PermissionError when the session's
defaultTeam is not present in account_info.organization (AccountInfo carries
a single org). has_permission() caught that and returned a bare False, so the
caller only ever saw "Permission 'x' denied" with no trace of the real cause —
an unresolvable default team. That silent swallow is why saas #373 (admin
Dashboard hanging on "Loading Dashboard Data" with Permission 'task.monitor'
denied, for a user who genuinely held the grant) took two days to diagnose.

Behaviour is unchanged — still denies — but the reason is now in the log.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015nTVr6jfSFYm1GppxbjghP

* fix(auth): log the denial where we can actually see it (stdlib, not rocketlib)

@anandray asked me to verify rocketlib.debug is enabled in prod before relying
on it. Checked against the running cloud ALB, and the answer kills the original
approach twice over:

1. debug() is gated on the engine trace level and prod runs without one:
   `./engine ./ai/eaas.py --saas --host=0.0.0.0 --port=5565 --modelserver=...`.
   Corroborated in the logs — `[account] upsert_user` fires on every login and
   appears zero times.

2. Bumping to warning() does not fix it either, which is what I would have
   shipped on the first suggestion. rocketlib output goes to the engine JOB
   log, which is delivered to the connected client, not to container stdout.
   A 5000-line sample of the prod ALB log is 100% uvicorn access lines:

     1808  GET /            200
      681  GET /version     200
      237  connection open
      216  WebSocket /task/service [accepted]

   Nothing from the engine at all, and no engine log file in the pod. So this
   PR as written produced no server-side observability whatsoever — and
   denying a client, then reporting the reason only to that same client, is
   not observability in any case.

Switched to stdlib logging, which those uvicorn lines prove reaches stdout and
therefore CloudWatch. Volume is bounded by actual denials, and a flood of them
is itself the signal — that is exactly the shape #373 had.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015nTVr6jfSFYm1GppxbjghP

* test(ai): assert the denial log says who, which team and what failed

CodeRabbit's point on this PR, and it is the same defect I have been catching
in other people's work all week: the existing test asserts only that
has_permission returns False, which passes just as happily with the logging
deleted. The log is the entire point of the change, and nothing was holding it.

The new test asserts the record carries the team, the user, the requested
permission and the original PermissionError text — so removing the line, or
dropping a field from it, fails.

Note the suggestion said to mock task_conn.debug; that is stale. This PR moved
the call to stdlib logging precisely because rocketlib's debug() never reaches
container stdout in the cloud, so the test uses caplog against the module
logger instead.

HONESTY ON VERIFICATION: I could not run this. The suite needs the built
engine — pytest walks fastapi -> uvicorn -> dotenv -> `rocketride` itself, at
which point pip starts trying to fetch our own published package over the local
source, and that is the wrong direction. Assertions were checked statically
against the f-string in task_conn.py: 'team-1', 'user-1', 'task.control' and
the exception text all appear in it.

And CI will not run it either. os-matrix.yml is the only workflow invoking
`./builder test`, and it fires on workflow_dispatch or a monthly cron — so
packages/ai/tests gates nothing on a PR. That is the third instance today of
tests existing with nothing running them (see saas #400 / #401). Filing
separately rather than widening this PR.

Related: #1670.

* test(ai): pin the denial log to one WARNING from this module

Second CodeRabbit finding on this PR, and it is the same criticism I made of
the test I was replacing — mine was also too loose to fail properly.

Joining every captured record meant an unrelated log could satisfy the field
assertions, and the test would keep passing if the denial were downgraded to
debug or raised to error. Both matter here specifically: the whole reason this
PR exists is that rocketlib's debug() never reaches container stdout in the
cloud, so a level change is exactly the regression that would make the message
invisible again while the test stayed green.

Now filters to records from task_conn's own logger at WARNING and asserts
exactly one, so a wrong logger, a wrong level, a missing record or a duplicate
all fail — and the failure message prints what was actually captured.

Refs #1670.

---------

Co-authored-by: Dmitrii Kataraev <dmitrii.kataraev@rocketride.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

* fix(server): dispatch pipeline lifecycle in topological order, including control sub-pipelines (#1669)

* fix(server): dispatch pipeline lifecycle in topological order

Pipeline open/closing/close were dispatched in `.pipe` canvas order, not
dependency order, so a join node could be flushed before an upstream branch
emitted its flush-time data — silently dropping it.

Bind the source-reachable nodes' lifecycle onto the pipe head in topological
order instead: closing/close upstream-first (a node flushes only after all of
its upstream parents), open reversed. Control/invoke sub-pipelines are left
untouched — they keep their per-parent binding so their invoke node (e.g.
`tool_pipe`) can still open/close them.

The order is computed once by `computeLifecycleOrder` (reachability from the
source, then a Kahn topological sort over the reachable set; a cycle in the
reachable region is rejected). Adds unit tests for the ordering and documents
the guarantee.

Closes #1650

* fix(server): extend topological lifecycle to control sub-pipelines

Generalize the open/closing/close dependency-order guarantee to a control node's
inline sub-pipeline (e.g. tool_pipe): it is flushed and closed in dependency order
per invocation, to any nesting depth, so a diamond sub-pipeline returns the merged
output of all branches. tool_pipe now drives closing() then close().

Reject at pipeline open the wirings that break exclusive sub-pipeline ownership:
sharing a sub-pipeline node with the main flow, a second start, or another control
node; and a data-fed invoke node.

Adds unit tests and updates pipeline-rules/observability docs and examples.

* fix(server): name nodes in sub-pipeline ownership errors

Report each node as its service title plus component id, e.g.
"Pipeline Tool" (pipe_tool_1), instead of a pipeStack index, so a rejected
wiring points at what the author sees on the canvas.

Document the exclusive-ownership rule in the tool_pipe README and add a
common-mistakes entry covering the three rejections. Examples switch to
agent_crewai and lane-specific response nodes.

* fix(server): address review feedback and cover the closing order

Remove `Binder::getBound`: public API added with no caller.

Range-check the lifecycle indices once up front rather than inside one of the
bind loops, so every walk is covered whatever order they run in, and correct
the comment that claimed bindFilters was allocation-free per pipe.

Add an end-to-end test for the closing order: a diamond of buffer-and-flush
nodes whose join is declared before its branches, so dispatching in declaration
order would flush the join early and drop a branch's output.

* fix(deps): restore a resolvable onnxruntime pin — 1.20.1 is gone from PyPI (#1725)

The -gpu build of 1.20.1 is absent from the index (not yanked; the release
key itself is gone), so every dependency resolve that misses uv's cache dies
with "no version of onnxruntime-gpu==1.20.1 ... requirements are
unsatisfiable". The CPU build of the same version survives, which is what
let the split sit broken unnoticed.

Move all five consumers to 1.22.0 together. It has wheels for -gpu on
win/linux and for the CPU build on macOS-arm, which 1.20.1 no longer does.
Note that the obvious one-patch bump is a trap: 1.20.2 exists only for -gpu,
so pinning it would break macOS.

Also correct the comment that justified the pin. It claimed rtmlib needs
exactly this version; rtmlib declares onnxruntime unpinned, as does gliner,
and faster-whisper asks for <2,>=1.14. The pin exists so the ~200 MB runtime
installs once across consumers — the number is ours to choose, provided all
five files agree.

Verified with the four consuming nodes, opted in via ROCKETRIDE_INCLUDE_SKIP
(real installs and model downloads, not just resolution):
pose_estimation (rtmlib), anonymize, ner (gliner), audio_transcribe
(faster-whisper) — all pass.

Fixes #1723

* chore(models): sync LLM model lists (#1634)

Co-authored-by: asclearuc <29624014+asclearuc@users.noreply.github.com>

* feat(nodes): add Guild.ai tool node (#1689)

Dual data+tool node running Guild.ai agents from a pipeline step or as an agent tool.
Mirrors tool_n8n (dual-node structure) and tool_v0 (IGlobal/test conventions, raise-not-return errors).

- Pipeline face: lane input runs the configured Guild agent; the answer flows downstream.
- Tool face: run_agent / get_session / get_session_events to delegate to and inspect sessions.
- HTTP Basic trigger-key auth; per-run session budget; POSTs never retried; identifiers sanitised.
- Response shapes verified end-to-end against a live Guild workspace, pinned as regression fixtures.

Closes #1688

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(docs): normalize Discord invite links to a single code (#1746)

The repo referenced three different Discord invites. Standardize every
reference on discord.gg/PMXrtenMsY (the code already used in the
top-level README and client readmes):

- discord.gg/9hr3tdZmEG (docs site + co-located docs) -> PMXrtenMsY
- discord.gg/rocketride (hello-ui placeholder) -> PMXrtenMsY

CI webhook URLs (discord.com/api/webhooks/...) are unrelated and left
untouched.

Co-authored-by: Mithilesh Gaurihar <mithileshgaurihar@Mithileshs-MacBook-Pro.local>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

* fix(agent): optional require_tool_call runtime fabrication guard (#1490)

* fix(agent): optional require_tool_call runtime fabrication guard

No runtime guard asserted that an agent actually invoked a tool, so a weak
planning model could narrate a multi-step tool chain in prose instead of
calling the tools — producing a plausible but ungrounded answer. The
"must call >=1 tool" rule existed only as a prompt instruction, never as code.

Enforce it once in the shared AgentBase so it covers all five agent
frameworks (langchain, llamaindex, rocketride, crewai, deepagent):

- AgentContext gains a run-scoped `invoked_tools` tally (a mutable field on
  the otherwise-frozen context; compare=False keeps the context hashable, and
  list.append is thread-safe under the GIL).
- call_tool — the single choke point every driver routes real tool calls
  through — records each invocation (memory.peek and other local reads bypass
  it and correctly do not count).
- run_agent raises ToolCallRequiredError -> a distinct
  RocketRide.agent.guard.v1 error answer when require_tool_call is on and no
  tool ran; meta.tool_calls now reports the count on success, guard, and error.
- The crewai-manager and deepagent sub-contexts thread the parent's tally so
  tools a sub-agent invokes count toward the parent run's guard.

Exposed as an off-by-default `require_tool_call` boolean on the six top-level
agent service files, read with parse_bool (a stringified "false" leaves it
off, rather than tripping the guard), with README notes + regenerated schema
tables.

Tests (packages/ai/tests/ai/common/agent/ + test_config_shapes.py): AgentBase
guard unit tests, config resolution (flat/nested/default shapes), and
end-to-end integration driving the real RocketRide wave loop and the real
LangChain create_agent loop — fabrication trips the guard, grounding passes,
memory.peek-only trips, and parallel tools in one wave are all counted.

Closes #1403

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(agent): version guard code in docs + retain raw trace on guard answer

Addresses PR review feedback:
- The Fabrication-guard notes in all five agent READMEs referenced the
  unversioned `RocketRide.agent.guard`; the runtime emits
  `RocketRide.agent.guard.v1` (matching the existing `.raw.v1` / `.error.v1`
  identifiers). Use the versioned id consistently.
- The guard-trip answer now also carries the framework's raw output as a
  `RocketRide.agent.raw.v1` stack entry (like the success path), so an operator
  investigating a tripped guard can inspect exactly what was narrated. `raw` is
  bound before the try, so this is safe on every path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: kgarg2468 <181051779+kgarg2468@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(chroma): validate server version and surface silent index failures (#1494)

* fix(chroma): validate server version and surface silent index failures

Chroma silently indexed nothing against incompatible/old servers: the real
error (KeyError('_type')) was swallowed into a transient debug() trace, and
the user only saw a cryptic "'NoneType' object has no attribute 'delete'"
crash with "0 done · 1 error" while downstream chat hallucinated.

- Validate the Chroma server version on connect (floor: major >= 1) and wrap
  the eager client construction so an incompatible/old server yields an
  actionable message instead of a cryptic one.
- _createCollection now re-raises the real cause instead of swallowing it;
  flush() guards a missing collection with a clear message (defense in depth).
- Harden _doesCollectionExist to accept both list_collections() shapes (names
  and Collection objects), fixing a latent always-empty-search bug on newer
  clients.
- Pin chromadb-client>=1.0.0 to match the supported server floor.

Fixes #1405

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(chroma): make server-version check robust to thin client; floor 0.6

The initial connect-time version check broke the working path in CI:

- The node installs the thin `chromadb-client`, whose HttpClient may lack
  get_version. Calling it inside the connect try mis-reported the AttributeError
  as "server too old". Probe defensively via _getServerVersion() so an absent or
  failing get_version is treated as unknown, never a connection error.
- The >= 1.0 floor rejected servers that work (the project's own test infra runs
  chromadb/chroma:0.6.3). Lower the floor to 0.6.0 (the v2 tenant/database API
  boundary), compared on major.minor.
- Revert the chromadb-client>=1.0.0 pin: the incompatibility is server-side and
  the pin was ineffective in CI.

Refs #1405

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: kgarg2468 <181051779+kgarg2468@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(storage): joined filesystem, in-store authorization, task-file identity, and rocketlib.getTask (#1686)

* feat(storage): joined filesystem, in-store authorization, task-file identity, and rocketlib.getTask

Foundation for team-scoped deployments: deployments will belong to teams,
so storage, permissions, and subprocess identity all had to stop assuming
"everything belongs to the calling user". This lands that groundwork;
deployment records and team run logs build on it next.

WHY each piece exists:

* In-store authorization — file_store.py IS the security boundary.
  Storage I/O happens from many places (rrext_store, rrext_log, fetch,
  the run-log writer, deployments). Checking permissions in command
  handlers meant every new caller could forget one — the "forgot to
  check" bug class. Now every path of every operation authorizes INSIDE
  the store, at the single point where paths resolve, against the
  RequestContext bound at construction. Command handlers do zero storage
  permission checks.

* RequestContext identity (copied verbatim from feat/alb, with MERGE
  NOTEs). feat/alb generalizes per-request identity for every handler;
  adopting the identical shape now makes that merge a dedupe instead of
  a rework. The internal()/engine() factories make the entire trusted
  surface greppable: every privileged caller is a
  RequestContext.internal(...) call site.

* Store singleton + RR_STORE_URL (STORE_URL hard-fails). A server has
  exactly one store, so Store.instance()/Store.file_store(ctx) replaces
  the multi-step create()/get_file_store dance that invited divergent
  configuration. The legacy env name fails loudly instead of falling
  back, because a silent fallback would never get migrated. Providers
  became packages (store_providers/filesystem/, memory/). Write locks
  and open handles moved to the shared Store so per-connection FileStore
  instances still exclude each other on one physical path.

* Joined filesystem, grammar v3: '@' is a virtual root listing User/,
  Team/, Org/ so a file browser can walk user, team, and org storage as
  ONE tree with ordinary path joining — while plain paths stay "my
  files" for nodes and SDK code (simple mode). Bare segments are display
  NAMES that resolve ONLY through the caller's own session membership
  (a foreign org's names are inexpressible by construction) and
  round-trip byte-exact: the path you browse to is the path you open.
  =<id> is the only cross-boundary spelling (sys.admin support access).
  Unknown, foreign, permission-less, and ambiguous references all deny
  with ONE identical message so nothing leaks existence. Names starting
  with '=' are uncreatable at scope tops so id references can never be
  shadowed. Case-exact and alias-free: one location, one path string.
  Normalization runs BEFORE parsing, killing the \@\Team\... bypass
  family (fuzz-tested).

* System-owned trees: .logs and .deployments are engine-written and
  served by their domain APIs (rrext_log today, rrext_deploy next).
  Raw file access could corrupt them or expose data the user must not
  see, so through the file API they are invisible in listings and denied
  on access for every session identity except sys.admin (who may do
  anything). Domain APIs act through the internal identity and keep
  their own task.monitor/task.control gates at the command layer.

* Three live cross-team permission holes closed: on_execute accepted an
  arbitrary teamId and merged that team's secrets into the task env
  BEFORE any membership check (secret exfiltration); on_restart
  restarted token-addressed tasks with only a defaultTeam check; the
  cprofile proxy could drive another team's engine subprocess. All three
  now resolve permissions against the ADDRESSED team/task first
  (verify_team_permission / get_task), with uniform denials.

* Task-file identity + storage anchor: subprocess identity rides the
  0600 task file ('identity' block) — never the environment, whose
  ROCKETRIDE_* namespace is caller-influenced by design; the
  ROCKETRIDE_CLIENT_ID env var is gone. The new 'storage' block gives
  nodes chroot semantics: dev runs anchor at the owner's whole tree
  (unchanged behavior), deploy runs at teams/<teamId>/files/tasks/
  <projectId>, so node-level paths are IDENTICAL in both modes, a
  deployed task needs no user, and concurrent deployments cannot fight
  over working storage. The engine context rejects every '@' spelling
  and system tree regardless of anchor — LLM-generated paths cannot
  leave the sandbox.

* rocketlib.getTask() C++ binding: the engine already parses the task
  file, so python re-reading argv would duplicate parsing and couple to
  the launch contract. ITask::execute() publishes the task JSON around
  the begin/end window — before beginTask, because endpoint (and node
  global) construction happens inside it — and clears it on every exit
  path. The store layer resolves identity+anchor itself
  (Store.engine_file_store()), so tool_filesystem carries zero jobConfig
  plumbing and future consumers get task data for free. Single-threaded
  by contract (main task thread only), hence no lock.

* explorer-ui roots its listing at '@' to present the joined tree.

1572 server tests green (suite grew ~90 tests: authorization matrix,
grammar fuzz, system-tree policy, anchor chroot, cprofile proxy).
Engine rebuilt on both dists; rocketlib.getTask verified live.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(storage): resolve PR #1686 review findings — fail-closed handles, universal root guards, real org resolution

Addresses the 14 CodeRabbit review threads:

Security/correctness:
- file_store: _get_handle requires an EXACT conn_id match. The handle
  registry became Store-wide in this PR, so the lenient empty-conn_id
  check would have let any default RequestContext reach another
  connection's live handle. Fail closed.
- file_store: the rename/delete root guards no longer exempt the caller's
  own namespace — rename('') resolved to the account root and turned the
  copy+delete loop into a whole-account move. delete/rename/rmdir now
  reject every root uniformly.
- store: a session-shaped ctx (account present, not internal/engine) with
  an empty userId can no longer anchor a FileStore via explicit client_id;
  that combination (e.g. a task-scoped AccountInfo built from an empty
  control.userId) would unlock an arbitrary user's namespace. Account-LESS
  contexts stay constructible because resolve_scope already denies their
  every operation with 'Not authenticated'.
- task_conn/cmd_task/cmd_debug: new resolve_org_for_team() — members
  resolve via their own membership; callers that pass the team check
  without membership (sys.admin, internal) resolve the target team's REAL
  org through the account backend, and an unresolvable team is denied with
  the uniform message. Previously such callers registered the task with
  orgId='' travelling in the task file as trusted identity.
- task_engine: _storage_root() validates through validate_storage_root at
  launch, so a malformed client-supplied project_id fails with a clear
  launch-time error instead of tool_filesystem disabling itself mid-run
  inside the subprocess.

Minor:
- explorer-ui: handleMove/handleUpload carry the '@' mount through an
  empty targetDir — the tree is rooted at '@', so bare names silently
  targeted the plain user store.
- tool_filesystem: IGlobal docstring, README handwritten sections, and the
  services.json pathWhitelist description describe the task storage anchor
  (dev = user tree, deploy = team task subtree) instead of the universal
  user-root claim. The generated README block regenerates on develop
  (nodes:docs-generate is branch-gated by design).
- file_store: stale usage examples fixed (Store.file_store(ctx), no
  connection_id params); handle-cap comment updated to string conn_ids.
- task.hpp/task.cpp: currentTask()'s main-task-thread-only access model is
  stated on the public API — publish, reads, and clear are strictly
  sequential on the single main task thread, so the value is
  unsynchronized by design.
- store/file_store: per-handle force-close teardown deduped into
  Store._teardown_handle(); FileStore._force_close_handle and
  close_all_handles delegate so the sequences cannot drift.
- task_server: dropped the outer sys.admin short-circuit duplicated inside
  resolve_task_permissions.

Tests: real resolve_org_for_team bound into the cmd_task/cmd_debug stubs,
get_merged_env patched where execution reaches the secret merge, dead
unowned FileStore construction removed, 16 inline FileStore(Store(istore),
...) constructions replaced with a shared make_file_store() helper, and
the sys.admin bypass test now asserts the resolver-side grant.

1572 passed / 122 skipped; ruff clean; explorer-ui rebuilt.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(storage): close the empty-orgId fast path; take develop's services.json description

resolve_org_for_team returned org.get('id', '') straight from the
membership loop, so a membership record whose org carries no id skipped
the empty-org guard the function exists to enforce. Both paths now feed
one guard: membership resolves the id (falling through to the account
backend when it is empty), and an unresolvable org denies with the
uniform no-permissions message.

tool_filesystem/services.json keeps develop's (#1651) description text
verbatim as the conflict resolution for the rebase onto develop; the
sink code's storage wording will be reconciled with the task-anchor
model when this PR merges.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(storage): resolve PR #1686 round-2 review findings — one normalization, sigil reservation, membership-based own-tree access

- file_store: normalize_path extracted to module level as THE wire-path
  normalization; FileStore._validate_path delegates to it. cmd_store's
  system-tree filter now normalizes with this same routine AND resolves the
  listing from the same normalized path — deciding on the raw spelling while
  resolving the normalized one let '@//User'-family spellings reach the user
  root with .logs/.deployments unfiltered.
- file_store: enforce the documented '@' reservation at scope tops alongside
  '=' — a scoped rest could smuggle in an '@'-leading name inside the
  grammar room the docs already reserved.
- file_store: the caller's OWN tree no longer hinges on the defaultTeam
  pointer (an unset or stale defaultTeam denied a user their own storage);
  the file-storage permission grants when ANY membership carries it.
- file_store: rmdir root guard made universal (plain `not rest`) to match
  delete/rename — the kind-conditional form was exactly the guard a future
  normalization change could quietly defeat.
- store: double-checked locking on the Store singleton — two threads racing
  instance() could each build a Store with its own _shared_handles /
  _shared_write_locks registry, silently defeating the cross-instance write
  exclusion this PR introduces (engine-subprocess node init is the realistic
  race).
- task_engine: anonymous dev runs (client_id='' — OSS/standalone) carry NO
  storage anchor instead of failing the launch: identity.userId rides empty
  too, so engine_file_store() yields None and the storage tools disable
  themselves — the same degradable posture as the run-log writer; the shared
  'users//files' prefix is never composed.
- models: RequestContext.account_info docs now name engine-subprocess
  contexts as the second legitimate None case.
- tests: pin the universal own-root guard (delete/rename of ''), the
  normalization-bypass listing matrix ('@//User', '@/./User', '\@\User',
  '/'), the dev-run no-anchor posture, STORE_URL/RR_STORE_URL isolation via
  monkeypatch.delenv, and mirror Store.file_store's real signature
  (root=None) in the cmd_task stub.

1636 server tests green (./builder ai:run-pytest --verbose).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(storage): audit sys.admin =id crossings; pin internal ordering; document resolver contract split (PR #1686)

Addresses the three non-blocking notes from the authorization review:

- file_store: sys.admin '=id' resolutions are the ONE place resolve_scope
  crosses the storage boundary instead of enforcing it (the platform support
  capability). Each crossing — foreign team/org/user id — now emits a
  server-side audit trace (actor, conn, target) via rocketlib.debug. Member
  '=id' resolutions cross nothing and are NOT audited. Trace-only: the wire
  response still cannot distinguish a crossing from an ordinary resolve, so
  the uniform-denial / no-oracle posture is preserved.
- file_store: comment the internal-identity branch as deliberately ordered
  BEFORE the system-tree gate (intent, not fall-through) — the run-log writer
  and domain APIs write .logs/.deployments through that identity;
  test_internal_identity_writes_logs is now also the ordering pin.
- models: banner both resolve_task_permissions (RETURNS []) and
  resolve_team_permissions (RAISES) with their opposite no-membership
  contracts. The rename to carry the difference in the name (get_* vs
  require_*) is deferred to the feat/alb dedupe, where both branches touch
  these functions — renaming here would fight that dedupe.
- tests: TestSysAdminCrossingAudit pins that foreign-id crossings audit and
  member resolutions don't (monkeypatches the hook, since the sink is native
  rocketlib.debug).

757 account+task tests green (./builder ai:run-pytest path).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(storage): resolve PR #1686 round-3 review — sign resolved fetch paths, live Store.reset, single-parse resolver

- get_url now signs the RESOLVED physical store path instead of the wire
  spelling. The fetch handler re-resolved the claim under an internal
  identity, which has no name dictionary and no org context — so signed
  URLs for '@/Team/<name>' and '@/Org' paths died with an unhandled
  PermissionError (HTTP 500). Authorization already ran at issuance under
  the session identity; the signed JWT is the capability, so the handler
  now serves the claim verbatim through backend._get_full_path (which
  guards traversal) with no scope re-resolution at all.
  FileStore.resolve() existed solely for that handler re-resolution and
  lost its only caller — removed rather than left as dead public surface.
  New TestSignedFetchUrls pins the reviewer's failure matrix (plain,
  @/User, @/Team by name, @/Team by id, @/Org), the resolve-free serve
  path, and that issuance still gates (@/Org without org.admin cannot
  mint a URL).

- Store.reset() is now actually wired: an autouse fixture in the ai tests
  conftest drops the process singleton around EVERY test, so no test can
  pin the RR_STORE_URL (plus _shared_handles/_shared_write_locks) that
  happened to be live when the first singleton-reaching test ran —
  singleton state was order-dependent and stateful before this.
  Also corrected the two stale STORE_URL docstrings in store.py to
  RR_STORE_URL (the legacy name hard-fails at create()).

- resolve_scope now returns (kind, scope_root, rest): it already had the
  kind in hand from its own parse_scope call, so _resolve no longer
  re-parses every path just to recover it — one parse per operation, and
  no "cannot fail" reasoning left for readers to re-verify.

- _store_fs_stat's virtual-mount check now runs normalize_path — the same
  ONE-normalization rule _store_fs_list_dir documents and follows —
  instead of .strip('/'), so mount detection and store resolution can
  never disagree about a spelling, and traversal raises instead of being
  stripped.

Full ai suite: 1646 passed, 122 skipped.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore(deps): align @types/react to ~18.3.x across the workspace

The root package.json pinned EXACT @types/react 18.2.79 / react-dom 18.2.25
while shared-ui had moved to ~18.3.29, so pnpm materialized TWO peer-variant
instances of every @types/react-peer package (react-markdown and friends) —
duplicated store copies and forked type-checking between apps. One types
version ends that: root, shell-ui, chat-ui, and dropper-ui now declare the
same ~18.3.29 / ~18.3.7 ranges shared-ui already uses. Types-only change;
chat-ui and dropper-ui type-check clean on the new majors, and the
pre-existing shell-ui contract-check delta is unchanged (verified against a
stashed baseline).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore(deps): land the @types/react package.json pins (completes 4b12d763)

4b12d763 was meant to carry these pins but landed only the lockfile
reshuffle (a stash-pop failure dropped the package.json edits before the
commit — message and content disagreed). This commit lands the actual
alignment: root, shell-ui, chat-ui, dropper-ui at ~18.3.29/~18.3.7;
vscode's floor raised to the resolved ^18.3.31; shared-ui's tilde floor
follows to ~18.3.31. One @types/react instance in the lock (zero 18.2
refs), so @types/react-peer packages (react-markdown et al.) no longer
fork into duplicate peer-variant copies.…
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation module:nodes Python pipeline nodes module:server C++ engine and server components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Pipeline lifecycle (open/closing/close) runs in canvas order, not dependency order — flush-time data silently lost

3 participants