Repository navigation
Umbrella: agent-framework-excellence — the substrate LLMs and humans both choose #10542
Description
Umbrella: agent-framework-excellence
Goal
Make AutoBot the agent substrate that both LLMs and humans prefer to work inside — the most flexible and most powerful framework for human↔AI collaboration, with freedom of model choice for everyone (local, hosted, or subscription).
This umbrella came out of a first-person capability assessment (an LLM judging whether it would want to run inside AutoBot, and what it would still lack). The verdict: AutoBot is already top-tier — real reasoning loop + belief state + stagnation detection, approval gates, a Docker sandbox executor, a full MCP client (stdio/SSE/HTTP) and MCP server, tiered memory + knowledge graph, a 10+ provider LLM gateway with fallback/circuit-breaker/semantic-cache, RLM self-critique, and a shipped human↔agent surface (handoff / assign / review-inbox §6.11, approval inboxes). The gaps below are what separate a great agent platform from the human-AI collaboration framework.
Two themes
A. Collaboration core — make the human↔agent relationship live, trustworthy, and self-improving.
B. Freedom of model choice — first-class Mistral & Claude Code providers + subscription/OAuth auth so we never limit anyone to pay-per-token API keys.
Tasks
A. Collaboration core — the relationship, made live & trustworthy
- feat(agent-loop): live mid-task steering — redirect a running agent without restart #10543 — Live mid-task steering: redirect a running agent without restart (priority: high — start here)
- feat(agent): persistent branchable per-task workspace — stateful sandbox with human drop-in #10544 — Persistent branchable per-task workspace: stateful sandbox with human drop-in
- feat(agent): closed feedback→behavior loop — adapt routing/prompts/skills from captured human signals #10545 — Closed feedback→behavior loop: adapt routing/prompts/skills from captured human signals (priority: high — highest-leverage latent asset)
- feat(eval): trajectory-level eval & regression harness — golden-trajectory replay across model/prompt changes #10546 — Trajectory-level eval & regression harness: golden-trajectory replay across model/prompt changes (priority: high)
- feat(agent): adversarial pre-action verification — independent second-model check before consequential acts #10547 — Adversarial pre-action verification: independent second-model check before consequential acts
- feat(provenance): grounded citations on every agent claim by default #10548 — Grounded citations on every agent claim by default
- feat(security): pre-action content firewall — wire prompt-injection defense over MCP/web/RAG/tool-output #10552 — Pre-action content firewall: wire prompt-injection defense over MCP/web/RAG/tool-output (priority: high — highest-risk security gap)
- feat(agent): agent-initiated ask-the-human — pause, ask, resume across channels #10553 — Agent-initiated ask-the-human: pause, ask, resume across channels (priority: high — the agent→human→agent half of collaboration)
- feat(memory): memory transparency & control — see, edit, and forget what the agent knows #10554 — Memory transparency & control: see, edit, forget what the agent knows
B. Freedom of model choice — never limit anyone
- feat(llm): Mistral provider — Le Chat / Codestral / Mistral agents #10549 — Mistral provider: Le Chat / Codestral / Mistral agents
- feat(llm): Claude Code / Claude Agent SDK as an execution provider #10550 — Claude Code / Claude Agent SDK as an execution provider (priority: high)
- feat(llm): subscription & OAuth-based provider auth — choice beyond per-token API keys #10551 — Subscription & OAuth-based provider auth: choice beyond per-token API keys
Principles
- Reuse existing substrate — every gap below has the hard parts already built; most are "connect the last wire," not greenfield.
- No model lock-in: local-first, but every hosted/subscription provider is a drop-in via the existing
BaseProviderABC +ProviderRegistry. - Each child is one PR,
Dev_new_gui-targeted, with file:line entry points in its body.
Activity
mrveiss commentedon Jun 28, 2026
New concrete sub-issues from Ornith-1.0 research (Theme A — closing the loop)
Comparison against DeepReinforce's Ornith-1.0 (self-scaffolding RL: reward shapes the scaffold, not just the route/solution) surfaced three precise, PR-sized "last wire" gaps — machinery AutoBot already built and left disconnected:
best_prompt_template(computed byTaskPatternLearner, persisted, exposed via API, 0 runtime consumers; onlybest_approachis used)similar_trajectoriesin plan construction (retrieved withmin_reward=0.7, parked incontext, docstring admits "step generation always builds from scratch")reasoning_contentfrom reasoning models instead of discarding<think>text#10580 and #10581 are concrete sub-slices of #10545 (closed feedback→behavior loop).
mrveiss commentedon Jun 28, 2026
Follow-up from the same research thread — a 4th, eval-side sub-issue:
TaskOutcomeJudge(best-of-N). Origin: AgentRM (arxiv 2605.02801). The judge exists but only fires post-hoc; nothing uses it to pick a plan before executing.Together #10580 + #10581 + #10583 form the in-context (no-weight-training) version of Ornith-1.0's self-scaffolding loop: propose candidate scaffolds → reward-rank → execute → feed reward back. Blueprint paper for #10580/#10581: Experience as a Compass (evolving orchestration + agent prompts, API-compatible).
mrveiss commentedon Jul 6, 2026
Epic complete — all 12 members delivered ✅
All agent-framework members merged to
Dev_new_gui:Follow-up security hardening for #10551/#10554/#10544 tracked in PR #10989 (IDOR + CSWSH). Efficiency/precision continuation is umbrella #10603.