feat(hermes-agent-panda): Docker-in-Docker sandbox on latest panda (0.37.0) + toolstream#3
Open
qu0b wants to merge 10 commits into
Open
feat(hermes-agent-panda): Docker-in-Docker sandbox on latest panda (0.37.0) + toolstream#3qu0b wants to merge 10 commits into
qu0b wants to merge 10 commits into
Conversation
…ream proxy Hermes runs its agent loop server-side and streams tool progress on a custom SSE channel (event: hermes.tool.progress) that Open-WebUI discards — OW only consumes data: chat chunks shaped like choices[].delta. So users never saw the agent run skills/commands, only the final prose. Add a tiny stdlib-only Go reverse proxy that fronts Hermes and rewrites those progress events into standard OpenAI delta.tool_calls chunks, which OW renders natively as tool-call blocks. Everything else is proxied verbatim; the final finish_reason stays "stop" (never "tool_calls") so OW renders rather than tries to execute. Verified against the live Hermes stream + a unit test of the translation; OW 0.9.5 source confirmed it accumulates/renders delta.tool_calls. Wired with zero chart changes: the proxy takes the public port (API_SERVER_PORT) and Hermes moves to +1 internally; entrypoint launches it unless HERMES_TOOLSTREAM_PROXY=0. Probes on the public port now also exercise the proxy.
…lta.tool_calls Investigating why OW showed the tool command but left it spinning "Executing..." with no result: OW 0.9.5 (utils/middleware.py) treats upstream delta.tool_calls as tools IT must execute and enters a tool-call retry loop (line ~4491). The names (terminal/skill_view) are not OW tools, so the cards never complete and the loop starves the final answer. Switch to emitting the tool activity as content-embedded `<details type="tool_calls" done="true" …>` blocks — the exact markup OW renders natively (middleware.py:499) — which bypasses the executor entirely. Built per tool on the `completed` event (command captured from `running`; pending tools flushed at stream end). Hermes streams no per-tool output, so the result body is empty (command is in `arguments`, the answer in the final prose, which now flows through). Verified against live Hermes: 2 cards, 0 delta.tool_calls, 0 leaked events, final answer intact.
OW now shows tool cards (INPUT/command) but OUTPUT was empty. Root cause from Hermes source: gateway/stream_events.py states "No tool output travels here — output is the agent.s concern ... not streamed as presentation", and the chat-completions completed event drops it. But api_server.py _on_tool_complete HAS function_result in scope — it just is not emitted. - Build-time patch (patch-hermes-toolresult.py, fail-loud like open-webui-cf) adds "result" (capped 8000) to the chat-completions tool.completed SSE event. - toolstream-proxy maps that result into each <details type="tool_calls"> card body, which OW renders as the tool OUTPUT. Verified: patch target unique + patched api_server.py compiles; proxy unit test covers result-in-card.
… live guides Another pass taking inspiration directly from ethpandaops/panda source (.claude/skills/query/SKILL.md + runbooks/debug_devnet.md + panda getting-started): - Make the skill a lean ROUTER that points to the code-owned, never-stale guide (`panda getting-started`) and live discovery (`panda docs`/`schema`/`resources`/ `search`) instead of re-documenting (drift-prone) the API and datasource rules. - Correct the log source: devnet CLIENT logs live in clickhouse-raw `external.otel_logs` keyed by ResourceAttributes[network] (verified live: bal-devnet-7 ~7M rows/15 hosts) — NOT internal.otel_logs/k8s.namespace.name (that was the in-cluster support services). Point to the "debug devnet" runbook. - Keep the durable, chat-specific bits panda doesn't carry: scope-to-this-devnet (no os.environ; meta_network_name), PANDA_ON_BEHALF_OF attribution, plain-English answers, never-paste-base64 (storage.upload), fresh-process/workspace sessions. Bump 0.4.0 -> 0.5.0.
panda 0.5.0, faucet 0.1.1, join-devnet 0.1.1 — trim the frontmatter description lines (used for skill selection) to one tight sentence each.
Equip the agent to debug a devnet, grounded in panda's canonical runbooks://debug_devnet: read networks://<network> first, stay fork-aware (ePBS/PeerDAS symptoms are protocol artifacts, not bugs by default), then baseline → classify → drill. Adds the source-authority table (networks/Dora/ Forky/ethnode/otel_logs) and two verified drill-down skeletons — an external.otel_logs error query (filter SeverityNumber>=17; casing of SeverityText is inconsistent) and ethnode finality/syncing/peers. v0.5.0→0.6.0.
…t panda Drop the `direct` sandbox backend (only ever lived in the unmerged panda PR #214 / `pr-214` images) and return to panda's canonical `docker` backend, which panda master/releases require — `pkg/sandbox` now ships only `docker` and `gvisor`, both needing a Docker daemon. - entrypoint.sh: panda-stack starts dockerd (storage driver from DOCKERD_STORAGE_DRIVER, default overlay2), waits for it, pre-pulls the sandbox image from the seeded config, then execs panda-server serve. - Dockerfile: reinstall docker.io/iptables/uidmap; drop the sandbox build stage + /opt/sandbox bundling + libgomp1 (the daemon pulls the sandbox image at runtime); PANDA_REF default latest -> 0.37.0 (latest stable panda; `latest` was never a published tag); restore DOCKERD_STORAGE_DRIVER + root. - CI: panda_ref default latest -> 0.37.0. - README/AGENTS: describe the dockerd/docker-backend dispatch again. The panda-chat chart's docker-backend config (backend: docker, privileged sidecar, sandboxImage) is what this image serves; set DOCKERD_STORAGE_DRIVER=vfs where nested overlay2 is unavailable (the ethpandaops devnet k8s nodes).
…26.7.1 Latest upstream release (Hermes Agent v0.18.0, 'The Judgment Release'). Verified against v2026.7.1 before bumping: - patch-hermes-toolresult.py TARGET block matches exactly once in gateway/platforms/api_server.py (the assert guard would fail-loud otherwise). - /opt/hermes/.venv and docker/stage2-hook.sh still present; s6-overlay /init model unchanged — the entrypoint's hand-rolled bootstrap still applies. Bumps Dockerfile BASE_TAG + workflow hermes_ref/base_tag defaults + README.
🤖 qu0b-reviewerReviewed @ |
…l + telegram SDK Context flows down from the deployment, not probed from inside the pod: - devnet-context.sh: ONE-SHOT briefing writer -> /opt/data/devnet-context.md (identity, fork schedule from the config service, deployment-declared tool URLs via DEVNET_TOOL_URLS, node inventory from panda-server). Renders static facts immediately, waits bounded for the panda sidecar, renders once more, exits. The agent re-runs it with --refresh — no daemon. - skills/devnet-tools: the tooling map (Dora/Forky/Tracoor/Assertoor/Grafana/ panda) + read-the-briefing-first orientation. - Dockerfile: bundle python-telegram-bot so the telegram gateway (enabled at runtime by TELEGRAM_BOT_TOKEN) needs no pip install as uid 10000. - entrypoint: launch the one-shot preloader on the hermes path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Switches the panda-server sandbox from the
directbackend (which only ever existed in the unmerged panda PR #214 /pr-214images) back to panda's canonicaldockersandbox backend on the latest stable panda (0.37.0). panda master/releases ship only thedocker/gvisorbackends — both need a Docker daemon — so this re-aligns us with upstream and lets us track panda releases instead of a stuck PR.Also lands the previously-orphaned toolstream proxy work (commits
2371eee..20c0647) that surfaces tool calls/output in Open-WebUI — PR #2 squash-merged only part of that branch.Image changes
entrypoint.sh:panda-stackstartsdockerdagain (+ pre-pulls the sandbox image), drops the/opt/sandboxPATH.Dockerfile: reinstalldocker.io/iptables/uidmap, drop thesandboxbundling stage +libgomp1,PANDA_REF=0.37.0.panda_refdefaultlatest(never a published panda tag — latent bug) →0.37.0.docker-backend dispatch again.Deploy contract
Pairs with panda-chat chart ≥ 0.3.1 (backend: docker, privileged sidecar,
sandboxImage: sandbox-0.37.0). overlay2 is reliable because the chart mounts/var/lib/dockeron a node-local emptyDir — see the chart PR.Build via the
build-docker-imagelabel (taghermes-agent-panda-qu0b-panda-dind-docker-backend) orworkflow_dispatchwithpanda_ref=0.37.0.