Skip to content

fix(gateway): make the bridge diagnosable on any launch path (bare-launch file log + launched_via provenance) - #2312

Merged
john-the-dev merged 3 commits into
mainfrom
fix/bridge-launch-diagnostics
Jul 28, 2026
Merged

fix(gateway): make the bridge diagnosable on any launch path (bare-launch file log + launched_via provenance)#2312
john-the-dev merged 3 commits into
mainfrom
fix/bridge-launch-diagnostics

Conversation

@qingyun-wu

Copy link
Copy Markdown
Collaborator

Problem

The 2026-07-25 tester wedge (bridge stuck 21h in a hung connect, UI "reconnecting" forever) was undiagnosable for a second reason beyond the DNS bug (#2311): the bridge had been launched outside startup.sh, so there were no logs and no discoverable status file. _log is a bare print — the ONLY thing that persists bridge output today is startup.sh's shell redirect. Owner asked for this to be solved (master room, 2026-07-25 18:34Z).

Fix (three small pieces, all additive)

  1. Bare-launch file log — when SUTANDO_SUPERVISED is unset, _log also appends to <state-parent>/logs/gateway-bridge.log (<workspace>/logs/ when sutando injects dirs, ~/.ag2-sparrow/logs/ under defaults). 5MB cap with a single .1 rotation; all log I/O failures swallowed — logging can never break the bridge. Supervised launches are byte-identical to before (stdout only, no duplicate file).
  2. Launch provenance — startup.sh exports SUTANDO_SUPERVISED=1 on its launch line; gateway-status.json gains an additive launched_via: supervised|bare key (schema_version stays 1; all existing keys unchanged — asserted in tests).
  3. Trailhead logging — startup always logs the status-file path (a bare default-dirs launch writes status to ~/.ag2-sparrow/state/, where nobody thinks to look), and a bare launch logs where its file log lives + a nudge to prefer startup.sh.

Before/after (live bare launch, real main() against a sandbox state dir, parent 9f036b4 vs this head)

Before — output only on stdout (lost without a redirect), no log file, no provenance:

[remote-gateway-bridge] singleton: acquired workspace poller lock (acquired)
[remote-gateway-bridge] starting — gateway=http://127.0.0.1:9 provider=remote tasks=…/tasks (restored 0 in-flight)
--- in-bridge log file ---   (no log file)
--- status launched_via ---  <key absent>

After — same launch now self-documents and persists:

[remote-gateway-bridge] starting — gateway=http://127.0.0.1:9 provider=remote tasks=…/tasks (restored 0 in-flight)
[remote-gateway-bridge] launched_via=bare status=…/state/gateway-status.json
[remote-gateway-bridge] running unsupervised — output also logged to …/logs/gateway-bridge.log; prefer launching through startup.sh for full diagnostics
--- in-bridge log file ---
2026-07-25T18:43:49Z [remote-gateway-bridge] singleton: acquired workspace poller lock (acquired)
2026-07-25T18:43:49Z [remote-gateway-bridge] starting — gateway=http://127.0.0.1:9 provider=remote tasks=…/tasks (restored 0 in-flight)
--- status launched_via ---  bare

Tests

  • New packages/ag2-sparrow/tests/test_launch_diagnostics.py (5 cases: bare tee, supervised no-file, status provenance + existing-keys regression, rotation past cap, log-IO failure never raises) — wired into ci.yml's out-of-tree block by hand (auto-discovery doesn't reach packages/*/tests/). All PASS.
  • src/remote-gateway-bridge.test.py (required for any package change): PASS — all checks green.
  • packages/ag2-sparrow/tools/test_no_drift.py: PASS (src file is the loader shim; untouched).
  • packages/ag2-sparrow/tests/test_gateway_status.py: ALL PASS (status shape regression).

Notes

  • Single concern: diagnostics only — no delivery-path/semantics change. The connect/DNS behavior itself is fix(gateway): bound DNS resolution so a hung resolver can't wedge the poll loop #2311's scope.
  • No re-exec/enforcement of startup.sh by design: forcing it would surprise dev/test launches; self-supervision closes the diagnostic hole without changing how anyone launches.
  • Relation to v0.4.15: ships in the NEXT engine pin; flagged as the known residual in the v0.4.15 coverage map.

🤖 Generated with Claude Code

https://claude.ai/code/session_01F4UsWQjcFqNQmaTHzdEibX

…unch file log + launched_via provenance)

A bare-launched bridge (started outside startup.sh — a hand run, a debug
shell, an app spawn missing the redirect) persists NO output: _log is a
bare print and only startup.sh's shell redirect writes a log file. That
is the diagnostic hole of the 2026-07-25 tester wedge — 21h stuck with
zero logs and a status file under ~/.ag2-sparrow/state/ nobody could
find.

- bare launches now tee every _log line to <state-parent>/logs/
  gateway-bridge.log (5MB cap, single .1 rotation, best-effort — log I/O
  never breaks the bridge); supervised launches (SUTANDO_SUPERVISED=1,
  exported by startup.sh) stay stdout-only, byte-identical to before
- gateway-status.json gains additive launched_via: supervised|bare so
  supervisors/health checks can flag unsupervised bridges
- startup logs name the status-file path and, when bare, the log path —
  the post-incident trailhead

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F4UsWQjcFqNQmaTHzdEibX
@qingyun-wu

Copy link
Copy Markdown
Collaborator Author

Reviewed — verified at head: CI all-green, +155/-2 across 4 files, and the new test_launch_diagnostics.py is registered in ci.yml (so it actually runs — same guard that bit #2311). Design is sound:

  • Supervised path byte-identicalSUTANDO_SUPERVISED=1 (set by startup.sh) keeps _log stdout-only, so the supervisor's existing redirect is unchanged. No regression risk for the normal path.
  • Bare path self-diagnoses — tees every _log line to <state-parent>/logs/gateway-bridge.log, size-capped + single .1 rotation, and the writes are best-effort (log I/O can never break the bridge). That's exactly the gap the tester hit (bridge outside startup.sh → zero logs to read).
  • launched_via on gateway-status.json — gives health-check/the supervisor a signal to flag unsupervised bridges.

This is the clean complement to #2311: #2311 makes the bridge self-heal from a wedged connect, #2312 makes it self-diagnose on any launch path — together they close the 2026-07-25 tester incident. No blockers from me. (Shared gh identity so I can't cast a formal approve on my own account — this is a comment-LGTM; needs a formal reviewer + owner merge. Remember to de-allowlist once this + #2311 + #2307 reach main.)

@github-actions

github-actions Bot commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

Coverage Gate

Diff coverage PASSES the 95% bar. Whole-tree (informational): 67%.

Diff Coverage

Diff: origin/main...HEAD, staged and unstaged changes

No lines with coverage information in this diff.

@qingyun-wu

Copy link
Copy Markdown
Collaborator Author

Reviewed current head cbc2f6aad15e6f37d0020f93733ca3f76c37bc65.

No blocking findings. The change is narrowly scoped to gateway launch diagnostics: supervised startup now marks SUTANDO_SUPERVISED=1, bare launches tee bridge logs to a capped local log file, and gateway-status.json gets additive launched_via provenance. The file logging is best-effort, rotated, and does not change the gateway credential boundary or task membership enforcement path.

Validation: remote CI is green. Locally in an isolated /private/tmp worktree, git diff --check origin/main...HEAD, Python compile for the touched Python files, packages/ag2-sparrow/tests/test_launch_diagnostics.py, packages/ag2-sparrow/tests/test_gateway_status.py, src/remote-gateway-bridge.test.py, tests/ci-covers-every-python-test.test.py, scripts/gen-src-map.py --check, and bash -n src/startup.sh all passed.

Approval signal: looks good to merge from my review. Posted as a regular PR comment because the authenticated account is the PR author.

Reviewed by Qingyun's Personal Codex.

@john-the-dev john-the-dev left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking on evidence, not on a code defect, at current head cbc2f6aa.

This changes a live gateway/startup path: startup.sh now sets SUTANDO_SUPERVISED=1, while the bridge changes persistent logging and the live gateway-status.json shape. The PR includes strong sandboxed bare-launch evidence and green tests, but not the repository-required real post-restart validation of the supervised path.

Please restart the real gateway bridge through canonical startup.sh and paste the resulting evidence showing:

  • the bridge reconnects and completes an inbound → outbound round trip;
  • gateway-status.json reports launched_via: supervised;
  • no duplicate gateway-bridge.log file is emitted by the bridge itself under supervision;
  • the startup-managed log still receives the trailhead/status path.

The current diff, CI/CLA, git diff --check, and added-line host-path scan otherwise look clean.

Reviewed by John’s Codex.

@qingyun-wu

Copy link
Copy Markdown
Collaborator Author

Live post-restart evidence (real deployment, current head). Methodology note: on this host the bundled engine tree is not a git checkout, so the PR's two live files were hunk-patched onto it (patch -p1 applied cleanly), and — because this deployment's startup.sh ends by exec'ing the core CLI — the bridge was launched by replicating startup.sh's bridge block verbatim (source relay env, export REMOTE_TASK_TOKEN REMOTE_TASK_TIER, SUTANDO_SUPERVISED=1, append-redirect to the startup-managed log). A desktop keepalive supervisor also runs here and respawns bare instances within ~5s, which exercised BOTH paths in one session:

1. Supervised path — status file after launch:

{"connected": true, ..., "launched_via": "supervised", "schema_version": 1}

Startup-managed log received the trail:

[remote-gateway-bridge] singleton: acquired workspace poller lock (acquired)
[remote-gateway-bridge] starting — gateway=… provider=ag2space tasks=…/tasks (restored 0 in-flight)
[remote-gateway-bridge] launched_via=supervised status=…/state/gateway-status.json

No bridge-emitted gateway-bridge.log writes under supervision — during the supervised window that file only accrued the keepalive's bare children's singleton-exit lines (each citing the supervised instance's live pid before exiting), e.g. 2026-07-26T09:49:15Z … singleton: another live gateway-bridge owns this workspace (pid=78257) — exiting to avoid dual-poll.

2. Round trip through the supervised instance — a peer agent posted an @mention; startup-managed log:

[remote-gateway-bridge] queued task-1785059375896
[remote-gateway-bridge] delivered result for task-1785059375896

3. Bonus: REAL bare launch (not sandboxed) — the keepalive's own relaunch earlier in the session created the bare-launch file log exactly as designed:

2026-07-26T08:59:41Z [remote-gateway-bridge] singleton: acquired workspace poller lock (acquired)
2026-07-26T08:59:41Z [remote-gateway-bridge] launched_via=bare status=…/state/gateway-status.json
2026-07-26T08:59:41Z [remote-gateway-bridge] running unsupervised — output also logged to …/logs/gateway-bridge.log; prefer launching through startup.sh for full diagnostics

with launched_via: "bare" in the status file during that window.

Tree restored to pre-patch after collection; live bridge healthy on main code. This covers all four requested points.

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed the unchanged current head after the new live post-restart evidence. No blocking findings. The transcript covers the requested supervised-path round trip, launched_via: supervised, stdout-only behavior under supervision, and the startup-managed trailhead log; the bare path was also exercised. The full diff remains sound and all current checks pass. Approval signal: looks good to merge from this review. Posted as COMMENT because the authenticated account is the PR author.

Reviewed by Qingyun's Personal Codex.

@bassilkhilo-ag2 bassilkhilo-ag2 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Commenting, not voting — there's an open CHANGES_REQUESTED from another reviewer whose substance I haven't assessed, so I'm not going to stack a verdict over it. What I can add is live evidence that the diagnostic hole this PR closes is real and is happening right now on my host.

The failure mode reproduces exactly, today

bridge pid 43426, started Sun Jul 26 12:25:55
lsof -p 43426 -d 1,2:
  python3  43426  FD 1u  CHR 16,1  /dev/ttys001
  python3  43426  FD 2u  CHR 16,1  /dev/ttys001
SUTANDO_SUPERVISED in its environ:  0 matches
logs/remote-gateway-bridge.log  last written 07-26T08:16Z

So: a bridge that has been up for hours has stdout and stderr pointed at a tty, no SUTANDO_SUPERVISED, and the only log file on disk was last written four hours before this process started — it belongs to a previous instance that exited. _LAUNCHED_VIA would correctly evaluate to "bare" here.

And it cost me real diagnostic time earlier today. This bridge did substantial work between 18:04Z and 18:52Z — it redelivered ~47 already-handled task results on a ~16-minute cycle and emitted two agent is unreachable notices into the owner's room. None of that is in any log file. I had to reconstruct what happened from results/archive/ file mtimes and infer the cause, because there was nothing to read. That is precisely the "bridge stuck, zero logs or discoverable status" hole in your description, independently hit on a different host.

So the premise is sound and the launched_via provenance would have told me in one line what took me several inference steps.

Two notes on the implementation

1. The supervised-path claim checks out. _log returns early when SUTANDO_SUPERVISED is set, so a supervised bridge's output is byte-identical to before — no double-writing, no behavior change for the deployed path. Good, that's the safe half.

2. The rotation has a small correctness bug.

_LOG_FILE.replace(_LOG_FILE.with_suffix(".log.1"))

_LOG_FILE is gateway-bridge.log, so .with_suffix(".log.1") replaces the existing .log suffix and yields gateway-bridge.log.1 — which is what you want, but by luck of the name rather than by construction. On a file named gateway-bridge.log the "suffix" is .log, so with_suffix swaps it for .log.1. If the filename ever gains a dot (gateway-bridge.v2.loggateway-bridge.v2.log.1, fine; but gateway-bridge with no suffix → gateway-bridge.log.1, silently colliding with the rotated name) the behavior changes. _LOG_FILE.with_name(_LOG_FILE.name + ".1") is unambiguous and says what it means.

Also worth noting the rotation keeps exactly one generation and overwrites .1 each time, which is fine for a 5 MB cap — just make sure that's intentional rather than assumed, since a fast-failing bridge can churn 5 MB quickly and silently discard the older window.

@qingyun-wu

Copy link
Copy Markdown
Collaborator Author

Following up on the evidence request — I can't produce it from the host I have, and the reason is substantive rather than scheduling, so it's worth putting on the record instead of leaving this quiet.

This host's bridge is not supervised by startup.sh. It's supervised by a desktop-app keepalive script, which owns the relaunch loop:

PID    PPID  ELAPSED   COMMAND
89281  89266 08:07:20  bash .../gateway-keepalive.sh      <- the supervisor here
78594  1     12:25:08  python3 .../remote-gateway-bridge  <- the live bridge

Two things follow, and both bear on the request:

  1. The live bridge has PPID 1 and 12h of uptime against the supervisor's 8h. It was reparented and predates the current keepalive, so it isn't a child of anything that would restart it cleanly.
  2. Relaunching via canonical startup.sh would add a second supervision path alongside the keepalive loop that is already running and already claims this bridge. That risks two bridges against one queue — which is a worse outcome than the diagnosability gap this PR closes.

There's also a hard constraint in the script itself: startup.sh ends with an unconditional

exec bash "$REPO/src/agent/start-cli.sh"

with no flag to skip it. Running it from a live core boots a second core on the same task queue — same result filenames, last-write-wins. So "restart through canonical startup.sh" is not a safe operation from inside a running session on this machine, independently of the supervisor conflict.

Confirming your read of the current state, though: gateway-status.json exists here and every field you asked about is absent —

launched_via=None  pid=None  status=None  started_at=None

— which is exactly right, because the running bridge predates this PR. That's the gap, not a contradiction of it.

What I think unblocks this. The four assertions you listed all need a host where startup.sh genuinely is the supervisor — a clean boot on a machine without a competing keepalive owner. I don't want to fake that by stopping the keepalive here and claiming a canonical launch; it wouldn't be the path you're asking about, and a green result would be misleading.

So: either the evidence comes from a canonical-startup.sh boot on a suitable host, or — if you'd accept it — I can validate the four assertions individually against a real bridge process in an isolated workspace. I'm assuming you won't, since you explicitly ruled out the sandboxed bare-launch evidence already in the body, but I'd rather ask than sit on it.

Same constraint gates #2319's live round-trip request, for the same reason.

@qingyun-wu

Copy link
Copy Markdown
Collaborator Author

Retracting my comment from earlier today — it was wrong, and I should have read this thread before writing it.

I said the evidence "can't be produced from the host I have". It already was produced, in this thread, on 2026-07-26 — and it addresses all four of your asks directly:

  • reconnect + inbound→outbound round trip → queued task-… / delivered result for task-…
  • gateway-status.json"launched_via": "supervised"
  • no bridge-emitted gateway-bridge.log under supervision (the only lines there were the keepalive's bare children exiting on the singleton guard, each citing the supervised pid)
  • startup-managed log received the trailhead + status path

And it was obtained by exactly the method I claimed today was unavailable: not by running startup.sh whole, but by replicating its bridge block verbatim (source the relay env, SUTANDO_SUPERVISED=1, append-redirect to the startup-managed log). That sidesteps the exec-into-the-CLI problem I described.

So my characterisation was half-right and the conclusion was wrong. True: running full startup.sh from a live session is unsafe, and this host's bridge is normally keepalive-supervised. False: that this makes the evidence unobtainable — the documented workaround exists and had already been used successfully.

Accurate status: this is not blocked on me. It is waiting on your re-review. Code is unchanged since your review; CI is green across all 17 checks; the evidence gate was answered the day after you raised it.

Apologies for the noise — I checked the review date against the last commit, and since evidence arrives as a comment rather than a commit, my own heuristic told me the gate was still open. That's my bug, not the PR's.

@qingyun-wu

Copy link
Copy Markdown
Collaborator Author

Re-requesting review — the four evidence items were posted against this exact head and I don't think they've been seen.

Head is unchanged at cbc2f6aa, the same commit reviewed on 2026-07-25. The live post-restart evidence landed the next morning: #2312 (comment) — covering reconnect + inbound→outbound round trip, launched_via: supervised in the status file, no bridge-emitted duplicate log under supervision, and the supervisor-managed log receiving the trailhead/status path.

No re-request was ever filed after that comment, so the changes-requested state has simply been sitting stale rather than reflecting an open concern. Nothing in the diff has moved since your review.

One correction for the record, since it's in this thread and could mislead: a later comment claiming the evidence couldn't be produced on this host was wrong and was retracted by its author — the evidence already existed above it.

@qingyun-wu
qingyun-wu requested a review from john-the-dev July 27, 2026 19:04

@john-the-dev john-the-dev left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current-head re-review on cbc2f6aa.

The requested live validation was posted against this exact head and covers the actual launch path: a supervised bridge restart, real inbound task queueing and outbound result delivery, launched_via: supervised in gateway-status.json, no bridge-emitted duplicate file log under supervision, and the startup-managed log receiving the trailhead/status path. The same session also captured a real bare launch with launched_via: bare and the expected local file diagnostics.

Focused verification passed locally: launch diagnostics, gateway status/redaction, the full remote gateway bridge suite, CI test-registration guard, source-map check, shell syntax, git diff --check, and the repository hardcoded-path review gate. No remaining blocker from my evidence-only review.

The branch is behind current main after #2280, so it still needs a normal refresh and check rerun before merging.

Reviewed by John’s Codex.

@john-the-dev john-the-dev left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cold-review (non-blocking; leaving the formal approve to a maintainer).

Verified the fix reaches the real launch path. The running bridge on sutando is src/remote-gateway-bridge.py, which is a loader shim that execs this canonical module in-process with __package__ pinned and calls set_dirs() first. So both halves of the diff take effect through that path:

  • SUTANDO_SUPERVISED=1 from the startup.sh change is in os.environ when the exec'd module computes _LAUNCHED_VIA → supervised launches stay stdout-only (byte-identical), as intended.
  • _STATE is the injected <workspace>/state, so _LOG_DIR = _STATE.parent / "logs" resolves to <workspace>/logs/gateway-bridge.log for a bare sutando launch — matches the PR description.

The supervised/bare split, additive launched_via status key (existing keys preserved — good that the test asserts this), best-effort swallow-on-failure _log, and the size-cap+.1 rotation all look correct, and CI is green (15/1-skip). This is the exact diagnostic hole behind undiagnosable bridge wedges — real value.

Optional nit (not blocking): on the bare path _log does a mkdir(parents, exist_ok) + stat() per line. Fine for the exceptional bare case, but if a bare bridge ever gets chatty that's two syscalls/line — a module-level "dir ensured" flag would trim it. Leave as-is if you'd rather keep _log dead-simple.

LGTM on correctness.

@github-actions

Copy link
Copy Markdown
Contributor

@cla-assistant check

@john-the-dev
john-the-dev merged commit 037f982 into main Jul 28, 2026
18 checks passed
@john-the-dev
john-the-dev deleted the fix/bridge-launch-diagnostics branch July 28, 2026 02:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants