Add in-notebook Jupyter chat agent with .claude skills support (#52)#53
Add in-notebook Jupyter chat agent with .claude skills support (#52)#53IzBrain67 wants to merge 6 commits into
Conversation
Implements a Jupyter notebook chat agent for the July hackathon. The agent loop runs in the kernel (so it can run code and edit files in the live workspace) while the OpenAI key and MCP workflow tools stay on the backend, reached over HTTP. Backend (reuses existing OpenAI/MCP chat services): - ServiceTokenAuthentication: kernel authenticates to stateless endpoints with the shared JupyterHub token. - /api/chat/llm/ streams OpenAI completions; /api/chat/mcp-tools/ and /api/chat/mcp-call/ proxy MCP tools, forwarding the user's Keycloak JWT for per-user workflow access. Kernel package neuroworkflow.agent (mounted via codes/ copy): - %chat / %%chat magics and a ChatPanel ipywidget. - Notebook-native tools (run_code, read_file, write_file) plus MCP workflow tools merged into one tool loop. - Loads git-tracked .claude/skills/*.md into the system prompt. Wiring: spawner injects PYTHONPATH/backend URL/service token and mounts the repo .claude read-only; nest image gains httpx and ipywidgets. Includes backend tests and a sample notebook. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Document how to use the in-notebook chat agent: setup/wiring, the magics and ChatPanel widget, enabling workflow tools with a Keycloak token, skills, the Python API, troubleshooting, and the src/codes sync note. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Skills are authored against the repository layout (src/neuroworkflow/nodes/, NODE_CREATION_GUIDE.md), but the kernel only mounts the codes/ tree. Append an "Environment paths" section to the system prompt that maps repo paths to their container equivalents (writable nodes at /home/jovyan/codes/nodes/, sandbox under it) and tells the agent not to chase repo-root docs that aren't mounted. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds an in-notebook Jupyter chat agent (neuroworkflow.agent) that runs inside the kernel, can execute notebook-local tools (run code / read+write files), and can optionally access workflow MCP tools via backend proxy endpoints. It also loads git-tracked .claude/skills/*.md and injects them into the system prompt, with container-path corrections for the Jupyter environment.
Changes:
- Introduces the in-kernel agent loop, notebook magics (
%chat/%%chat), and anipywidgetschat panel UI. - Adds backend proxy endpoints for LLM streaming (
/api/chat/llm/) and MCP tool listing/calls (/api/chat/mcp-tools/,/api/chat/mcp-call/) plus service-token authentication. - Updates JupyterHub spawner + nest image wiring (mount
.clauderead-only; inject agent env vars; installhttpx/ipywidgets) and adds docs + tests.
Reviewed changes
Copilot reviewed 24 out of 24 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
src/neuroworkflow/agent/__init__.py |
Public notebook agent entrypoints (singleton agent, %load_ext, ChatPanel). |
src/neuroworkflow/agent/client.py |
Kernel → backend HTTP client for SSE LLM streaming + MCP calls. |
src/neuroworkflow/agent/config.py |
Kernel-side config via spawner-injected environment variables. |
src/neuroworkflow/agent/loop.py |
Synchronous in-kernel tool loop driving streamed LLM + tool calls. |
src/neuroworkflow/agent/magic.py |
%chat / %%chat magics wiring. |
src/neuroworkflow/agent/skills.py |
Loads .claude/skills/*.md and builds the system prompt (+ path mapping). |
src/neuroworkflow/agent/tools.py |
Notebook-native tool implementations (run_code, read_file, write_file). |
src/neuroworkflow/agent/widget.py |
ipywidgets-based persistent chat panel UI. |
gui/workflow_backend/django-project/codes/neuroworkflow/agent/* |
Synced copy of the kernel agent package for the mounted codes/ tree. |
gui/workflow_backend/django-project/app/auth/authentication.py |
Adds ServiceTokenAuthentication for kernel → backend stateless auth. |
gui/workflow_backend/django-project/app/chat/views.py |
Adds notebook LLM/MCP proxy endpoints and SSE streaming. |
gui/workflow_backend/django-project/app/chat/urls.py |
Routes for llm/, mcp-tools/, and mcp-call/. |
gui/workflow_backend/django-project/tests/test_notebook_agent.py |
Backend tests for service-token auth and MCP proxy behavior. |
gui/workflow_backend/django-project/neuroworkflow/jupyterhub_config.py |
Spawner mounts .claude and injects agent env vars (PYTHONPATH, backend URL, service token). |
gui/workflow_backend/django-project/neuroworkflow/Dockerfile.nest |
Installs httpx + ipywidgets into the Jupyter image. |
docs/NOTEBOOK_CHAT_AGENT.md |
Usage + troubleshooting documentation for the notebook agent. |
notebooks/notebook_chat_agent.ipynb |
Example notebook showing %chat and ChatPanel() usage. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| authentication_classes = [ServiceTokenAuthentication, KeycloakAuthentication] | ||
| permission_classes = [IsAuthenticated] |
There was a problem hiding this comment.
Fixed in 9900fea: NotebookLLMView.authentication_classes is now [ServiceTokenAuthentication] only, so a Keycloak-authenticated user can no longer reach the LLM proxy. The browser chat uses /api/chat/stream/.
| """Authenticate trusted internal callers via the shared JupyterHub token. | ||
|
|
||
| The notebook chat agent runs inside a Jupyter kernel that has the | ||
| ``JUPYTERHUB_API_TOKEN`` in its environment but no end-user JWT. It uses | ||
| that token to reach stateless backend endpoints (the LLM proxy). A constant | ||
| name avoids leaking the secret into the user table. This authenticates the | ||
| *call*; per-user MCP actions still forward the user's own Keycloak JWT. |
There was a problem hiding this comment.
Fixed in 9900fea: docstring now states the kernel sends the token as NEUROWORKFLOW_SERVICE_TOKEN (the per-server JUPYTERHUB_API_TOKEN is a different value) and the backend validates it against its own JUPYTERHUB_API_TOKEN.
| def on_tool(name, args): | ||
| with log: | ||
| print(f"\n ⚙ {name}({', '.join(args)})") |
There was a problem hiding this comment.
Fixed in 9900fea: the widget on_tool callback now prints key=value pairs (matching the magics) instead of just the dict keys.
| def on_tool(name, args): | ||
| with log: | ||
| print(f"\n ⚙ {name}({', '.join(args)})") |
There was a problem hiding this comment.
Fixed in 9900fea (synced copy): same key=value fix as the src widget.
| def _read_file(path: str) -> str: | ||
| with open(path, encoding="utf-8") as f: | ||
| return _truncate(f.read()) |
There was a problem hiding this comment.
Fixed in 9900fea: read_file now resolves via _resolve_in_workspace and rejects paths outside NEUROWORKFLOW_WORKSPACE_ROOT (default /home/jovyan/codes). (Note: run_code can still reach anything the kernel can; this guard is defense-in-depth for the file tools.)
| def _write_file(path: str, content: str) -> str: | ||
| directory = os.path.dirname(path) | ||
| if directory: | ||
| os.makedirs(directory, exist_ok=True) | ||
| with open(path, "w", encoding="utf-8") as f: | ||
| f.write(content) | ||
| return f"Wrote {len(content)} chars to {path}" |
There was a problem hiding this comment.
Fixed in 9900fea: write_file is confined to NEUROWORKFLOW_WORKSPACE_ROOT via the same _resolve_in_workspace check before creating directories/writing.
| def _read_file(path: str) -> str: | ||
| with open(path, encoding="utf-8") as f: | ||
| return _truncate(f.read()) |
There was a problem hiding this comment.
Fixed in 9900fea (synced copy): same workspace-boundary guard on read_file.
| def _write_file(path: str, content: str) -> str: | ||
| directory = os.path.dirname(path) | ||
| if directory: | ||
| os.makedirs(directory, exist_ok=True) | ||
| with open(path, "w", encoding="utf-8") as f: | ||
| f.write(content) | ||
| return f"Wrote {len(content)} chars to {path}" |
There was a problem hiding this comment.
Fixed in 9900fea (synced copy): same workspace-boundary guard on write_file.
- LLM proxy now accepts only the service token (drop KeycloakAuthentication) so a logged-in user cannot spend the backend OpenAI key. - Confine read_file/write_file to NEUROWORKFLOW_WORKSPACE_ROOT (default /home/jovyan/codes); reject paths outside it. - Widget tool trace now prints arg values, not just keys. - Fix ServiceTokenAuthentication docstring to reflect NEUROWORKFLOW_SERVICE_TOKEN. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Hi @carlosengutierrez 👋 Could you help me verify that the new in-notebook chat agent correctly uses the skills we track in the repo? There's a short usage guide to follow: docs/NOTEBOOK_CHAT_AGENT.md (see the Quick start and Skills sections). 1. Get onto this branchgit fetch origin
git switch feature/52-notebook-chat # or: git checkout feature/52-notebook-chat
git branch --show-current # should print feature/52-notebook-chatIf git says local changes are blocking the switch, set them aside first and restore them afterwards: git stash
git switch feature/52-notebook-chat
# ... after testing ...
git switch - # go back to your previous branch
git stash pop # restore your changesWhen you're done, you can return to the main branch with 2. Run itStart the stack and open a Jupyter notebook (steps are in the guide), then in a cell: 3. What to confirmThe agent should follow our node-creation guidance — for example, it first looks through the existing nodes, asks you for the details it needs (name, stage, parameters, input/output ports with scientific units like Hz), and proposes creating the file in the nodes "sandbox" folder. Could you let me know:
The guide also has a short Skills verification step and a Troubleshooting section in case something doesn't load. Thanks so much! 🙏 |
|
Thanks for the work on this, the skill loading and notebook design are OK. I want to clarify the use case I had in mind, because I would like the participants to understand create nodes locally, without dependency on backend and server (I am assuming low connection to internet, and no server at all), first. First stage:
So the first version of this agent should be fully standalone. Just the repo, a local Python environment, and an API key (OpenAI or Anthropic or Llama?) set as an environment variable. The agent reads the .claude/skills/ files (you already did this), can read/write files and run code in the notebook kernel, and calls the LLM API directly. Second stage (after the hackathon) よろしくお願いします。 |
Let the in-notebook chat agent understand the brain viewer's live state. The viewer (a standalone browser tab) and the chat agent (in the Jupyter kernel) have no communication channel, so add an explicit, manual bridge: - Viewer: a "Copy state for Chat" button serialises the current selection, thresholds, display toggles, BOLD playback, stimulated regions, dataset summary, and camera into a human-readable Markdown block (with an embedded JSON fence) and copies it to the clipboard. The user pastes it into Chat. Reuses the existing visible-connection maths; falls back to window.prompt outside a secure context. Applied to all four brain_viewer.js copies. - Chat: a new .claude/skills/brain-viewer.md skill (auto-loaded into the agent system prompt) documents how to read a pasted snapshot and how to load connectivity_data.json for deeper data questions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The in-kernel notebook agent now runs the Claude Agent SDK instead of the OpenAI LLM-proxy loop. The browser chat (/api/chat/stream/) is unchanged and still uses OpenAI. Backend: - Add an Anthropic passthrough proxy (/api/chat/anthropic/<path>): it validates the shared service token (sent as x-api-key) and injects the real ANTHROPIC_API_KEY, so the key never enters user-accessible kernels. Forwards the request/response untouched and decompresses the streaming body. Kernel agent (src + synced codes copy): - Drive the SDK on a dedicated worker-thread event loop, isolated from the kernel's own loop (avoids the nest_asyncio re-entrancy that destroyed kernel tasks). Let query()'s async generator finish on its own. - Keep live run_code as an in-process SDK MCP tool (exec against the notebook namespace); wrap the workflow MCP tools via the existing backend proxy; file edits use the SDK's Read/Write/Edit confined to the workspace via can_use_tool. - Reach Anthropic through the backend proxy (ANTHROPIC_BASE_URL) presenting the service token as the API key. Infra: - Bundle Node.js + the Claude Code CLI + claude-agent-sdk + nest_asyncio in the nest kernel image (kept at the end of Dockerfile.nest so the expensive source-built layers stay cached). - Inject ANTHROPIC_BASE_URL/ANTHROPIC_MODEL into kernels via the spawner; add ANTHROPIC_API_KEY to the backend and compose/env templates. Docs/tests: - Update CLAUDE.md, docs/NOTEBOOK_CHAT_AGENT.md, gui/DOCKER_SETUP.md, gui/.env.example and gui/env.template. - Add tests for the Anthropic proxy (service-token auth, key injection, streaming). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Summary
Implements a Jupyter notebook chat agent for the July hackathon (closes #52). The agent runs inside the kernel, so it can run code and edit files in the live workspace, while the OpenAI key and MCP workflow tools stay on the backend and are reached over HTTP. It loads the repo's git-tracked
.claude/skills/*.mdand applies them as guidance (currentlycreate-node.md).Backend (reuses existing OpenAI/MCP chat services)
ServiceTokenAuthentication: the kernel authenticates to stateless endpoints with the shared JupyterHub token.POST /api/chat/llm/streams OpenAI completions;GET /api/chat/mcp-tools/andPOST /api/chat/mcp-call/proxy MCP tools, forwarding the user's Keycloak JWT for per-user workflow access.Kernel package
neuroworkflow.agent%chat/%%chatmagics and aChatPanelipywidget.run_code,read_file,write_file) merged with MCP workflow tools into one tool loop..claude/skills/*.mdinto the system prompt, plus an "Environment paths" correction so skill paths map to the container layout (/home/jovyan/codes/...).Wiring
PYTHONPATH/ backend URL / service token and mounts the repo.clauderead-only. Note: does not setJUPYTERHUB_API_TOKEN(reserved by the hub) — usesNEUROWORKFLOW_SERVICE_TOKENinstead.httpxandipywidgets.Docs & tests
docs/NOTEBOOK_CHAT_AGENT.mdusage guide and a sample notebook.Notes for reviewers
src/neuroworkflow/agent/and the mounted copygui/workflow_backend/django-project/codes/neuroworkflow/agent/(samecore//utils/sync convention) — both must stay in sync.Test plan
docker-compose build && up, open Jupyter,%load_ext neuroworkflow.agent%chatgenerates and runs code;ChatPanel()shows a single panel"Skill: create-node.md" in get_agent().messages[0]["content"]isTrue; node-creation requests target/home/jovyan/codes/nodes/sandbox/user_token, MCP tools (e.g.add_node) reflect in the web GUI flowpoetry run pytest tests/test_notebook_agent.py🤖 Generated with Claude Code