Built-in Harper Agent Component #676

Proposal / Motivation

Ship an AI agent inside the Harper server process (as opposed to the external harper-agent CLI). It receives prompts from app developers and operators, then autonomously develops, tests, monitors, and debugs Harper components and applications running on that instance.

• Single LLM-backed operation API instead of a separate CLI on the dev machine.
• Zero-friction access to live server state: in-process operation dispatch (no auth round-trips, no HTTP), raw log files, component file edits, V8 inspector attach against worker threads.
• Autonomous mode (timers + self-resumption) lets the agent iterate on a goal — e.g. "build a recommendations component, then test it, then refine" — without a human in the loop on each turn.
• Coexists with harper-agent CLI and Studio chat: those become thin clients to agent_prompt rather than parallel engines.

Relationship to HarperFast/harper#510 / HarperFast/harper#612 / HarperFast/harper#617

This issue depends on and builds on top of the model-access stack rather than duplicating it:

• Loop: the agent uses scope.models.generate(input, { toolMode: 'auto', tools, ... }) from Add unified model-access API (scope.models) harper#510 + Add agent-loop orchestration / toolMode: 'auto' to scope.models harper#612. No bespoke @openai/agents integration; no second loop implementation.
• Tools: most tools come from the unified MCP tool registry (test(cluster): pin three node-lifecycle invariants — hdb_nodes reload auth, clone readiness, remove_node blast radius #615), populated by the Operations profile (test(stress): gate large-catchup on a throughput floor and emit host contention counters #617, e.g. describe_*, search, restart, set_configuration, package_component, deploy_component, …) and the Application profile (test(cluster): make blockCacheEviction actually wait for the restart; pin the pre-crash replication gap #618, auto-generated per-Resource tools). The built-in agent consults the registry as the configured agent user, getting RBAC-filtered tools for free.
• Server-only extensions: a small set of tools is private to this component — never registered in [MCP] Tool registry + class-level introspection + RBAC-aware tools/list filtering harper#615 — because their runtime assumptions only hold on the main thread (see "Operator-only tools" below).

This is distinct from application agents built with HarperFast/harper#612 by app developers in Resource code:

The two share the loop (one implementation in HarperFast/harper#612) but not session state, not tool surface, and not auth identity.

Architecture

Where it runs. Built-in component registered via HARPER_BUILTIN_COMPONENTS, loaded by core/components/componentLoader.ts. Exports startOnMainThread (legacy pattern, like replication/subscriptionManager.ts). Lives on the main thread alongside the operations API server (core/server/operationsServer.ts, default port 9925). Worker threads handle application/REST traffic; the agent stays out of their way and inspects them via CDP.

Request flow:

1. startOnMainThread registers agent_prompt (and friends) via server.registerOperation. These handlers run directly on the main thread.
2. agent_prompt appends the user message to the named session and kicks the agent loop (or attaches to an in-flight loop for that session). Returns { session_id, message_id } immediately.
3. Internally, the loop is a call to scope.models.generate(messages, { toolMode: 'auto', tools: composedToolSet, maxToolIterations, ... }) — the Add agent-loop orchestration / toolMode: 'auto' to scope.models harper#612 orchestrator handles tool dispatch, RBAC, audit-log integration, and re-invocation.
4. Clients poll get_agent_session to see transcript/status updates (SSE streaming can come later).
5. Concurrent sessions interleave on the event loop (turns are mostly awaits on LLM/HTTP). Per-session serialization prevents two prompts in the same session from racing.

Component structure

agent/
  agent.ts              // entry: exports startOnMainThread, handleApplication
  operations.ts         // registerOperation calls
  session.ts            // CombinedSession backed by hdb_agent_session table
  toolset.ts            // composes RBAC-filtered registry + operator-only tools per call
  tools/
    fsTools.ts          // scoped read/write/list/grep against componentsRoot + logDir + configDir
    inspectorTool.ts    // CDP attach/evaluate/breakpoint/logpoint/profile against worker debug ports
    scheduleTool.ts     // schedule_followup via setTimeout(...).unref()
    httpFetchTool.ts    // outbound fetch (web research + self-test against own server)

Wire it up in bin/harper.js:

```js
process.env.HARPER_BUILTIN_COMPONENTS = ... + ',agent=@/dist/agent/agent.js';
```

No new top-level model-loop code, no provider SDK wiring — that all comes from scope.models via HarperFast/harper#510.

Operations to register

All gated on super_user, registered on the main thread in startOnMainThread (same pattern as install_usage_license in licensing/usageLicensing.ts):

Tool composition

Per agent_prompt invocation, the agent's tool set is composed from two sources:

1. Unified MCP tool registry (#615), RBAC-filtered for the configured agent user. With HarperFast/harper#617 + HarperFast/harper#618 this includes:

• Operations API tools (test(stress): gate large-catchup on a throughput floor and emit host contention counters #617): describe_all, describe_table, search, read_audit_log, package_component, deploy_component, drop_component, restart, set_configuration, add_role, etc. (Subject to the per-operation allow/deny list and per-op annotations like destructiveHint.)
• Per-Resource Application tools (test(cluster): make blockCacheEviction actually wait for the restart; pin the pre-crash replication gap #618): auto-generated get_*, search_*, create_*, update_*, delete_* for each @export-ed Resource on the instance.
• Custom Resource methods opted in via static mcpTools (replication: unresolved subscription placeholder Promise causes inbound-message TypeError storm and spurious full-database copy on every restart #622).
• read_log (the operations API), which lives in [MCP] Operations profile: tool generation over OPERATION_FUNCTION_MAP harper#617's profile. Note: this is not the low-level log file reader — that's covered by the scoped FS tools below.

The agent gets these for free; no per-tool wiring in agent/.

2. Operator-only tools, passed inline at the scope.models.generate call site. These are not registered in HarperFast/harper#615 because their runtime assumptions only hold on the main thread, and exposing them to application agents would conflict with Harper's app abstractions:

Operator tools are appended to the per-call tools: list passed to scope.models.generate. Without registration, there's no path by which they appear in tools/list for external MCP clients or for application toolMode: 'auto' callers.

Sessions

New system table system.hdb_agent_session keyed by session_id, holding an ordered array of AgentInputItems plus a pendingApprovals: ApprovalRequest[] field used by the approval flow (approve_agent_action resolves entries here). Implements a CombinedSession interface so the model-access call can hydrate / persist history transparently.

Intentionally separate from ConversationResource (#511), which is the app-developer-facing conversation primitive. Different audience, different lifecycle, different schema — the built-in agent's sessions are operator-owned and server-local; ConversationResource conversations are app-defined and may be tenant-scoped, multi-user, indexed, etc.

Configuration

New agent: block in harperdb-config.yaml, validated against core/config-root.schema.json:

```yaml
agent:
enabled: true # default false; opt-in to avoid surprise LLM bills
provider: anthropic # optional — falls back to scope.models default
model: claude-opus-4-7 # optional — falls back to scope.models default
maxTurns: 50
maxCostUsd: 5.00 # per-session hard cap; loop aborts with structured error when hit
autoApprove: false # if true, agent runs without approval gates (still gated by allowDestructive)
allowDestructive: false # required to enable destructive ops tools (drop_component, restart, set_configuration, ...)
user: hdb_agent # role/user the agent acts as (default: hdb_agent super_user, created at startup if missing)
componentsScope: ./components
```

Provider/model resolution: if agent.provider / agent.model are omitted, fall back to the scope.models default. Single-provider users configure once; power users can override per agent.

API keys: env vars only, read at startup, never in config. Inherited from the same env vars scope.models already uses (ANTHROPIC_API_KEY, OPENAI_API_KEY, …).

Permission / auth model

• agent_prompt and friends require super_user — enforced by the standard operations auth layer (we don't set bypass_auth).
• Tool calls execute under the agent's configured user. Default: a system hdb_agent super_user created at startup if missing. Operators can swap to a restricted role to limit blast radius (e.g. a read-only role for an analytics agent).
• Destructive operator tools (any that mutate config, drop components, or restart) require allowDestructive: true and still emit an approval request when autoApprove: false. restart and set_configuration require approval even when autoApprove: true — a misfire there bricks a node.
• FS tools reject any path resolving outside the scoped roots.

Self-debugging support

threadServer.js opens the V8 inspector on THREADS_DEBUG_PORT (main) or THREADS_DEBUG_STARTINGPORT + workerIndex (workers) when threads_debug=true. The agent's inspector_* tools speak CDP over WS to the worker ports. Document in setup that operators must set threads_debug: true and a sensible starting port before asking the agent to debug.

inspector_attach rejects workerIndex < 0 (the main thread is where the agent itself runs; attaching would deadlock).

Dependencies

To harper-pro/package.json: nothing new for the loop itself — it's all scope.models (#510). The only direct deps:

• ws (already transitively present, for CDP client to worker debug ports)
• zod if we choose Zod-typed tool schemas for the operator-only tools (already transitively present)

No @openai/agents, no ai, no @ai-sdk/* direct in harper-pro — those live in scope.models resolution.

Resolved decisions

• Loop: scope.models.generate(..., { toolMode: 'auto' }) via Add agent-loop orchestration / toolMode: 'auto' to scope.models harper#612. No second implementation.
• Threading: ops API is main-thread; component registers directly, no bridge.
• Tool registry vs private tools: registry (test(cluster): pin three node-lifecycle invariants — hdb_nodes reload auth, clone readiness, remove_node blast radius #615 via [MCP] Operations profile: tool generation over OPERATION_FUNCTION_MAP harper#617/test(cluster): make blockCacheEviction actually wait for the restart; pin the pre-crash replication gap #618) for everything portable; operator-only tools stay private to this component because their runtime constraints (main-thread inspector, FS-as-abstraction-layer, timer persistence) don't translate to application agents.
• Sessions: hdb_agent_session is separate from ConversationResource (fix(replication): close on inbound message-handler errors instead of swallowing them (#440) #511) by design.
• Provider/key config: agent: block falls back to scope.models defaults; keys env-only.
• Status updates: clients poll get_agent_session; SSE/streaming later.
• Approvals: stored on the session; resolved via approve_agent_action.
• Concurrency: multiple sessions interleave on the event loop; per-session serialization.
• Destructive ops: allowDestructive flag; restart and set_configuration always require approval.
• Inspector: workers only.

Open questions

1. harper-agent CLI transition. With the in-process built-in agent doing the work, the CLI's role compresses to a transport/UI in front of agent_prompt. Worth confirming we want to migrate it to that shape rather than continue to maintain two engines.
2. Per-session maxCostUsd enforcement. Add agent-loop orchestration / toolMode: 'auto' to scope.models harper#612 needs to expose a per-call token/cost budget for the orchestrator to enforce, or the built-in agent enforces it externally by inspecting cumulative analytics.model_call rows for the session. Prefer the former; see Add agent-loop orchestration / toolMode: 'auto' to scope.models harper#612 follow-up.
3. Cost telemetry surfacing. Should get_agent_session include per-turn cost so operators can see budget burn-down without joining to analytics.model_call manually?
4. First-run UX. Server-side only — no REPL. Docs page with curl examples + Studio panel later.

Critical files to read / modify

Modify:

• harper-pro/bin/harper.js — add agent=@/dist/agent/agent.js to HARPER_BUILTIN_COMPONENTS.
• harper-pro/core/config-root.schema.json — add agent block schema.
• harper-pro/core/utility/hdbTerms.ts — add config param constants + new OPERATIONS_ENUM entries for agent_prompt, get_agent_session, etc.

Create: harper-pro/agent/ (full new directory, far thinner than the original proposal).

Read & reuse:

• scope.models model-access API (Replication W13: Base-copy & catch-up path (consolidation & correctness) #510) — backend.
• scope.models toolMode: 'auto' orchestrator (test(cluster): promote 4 QA replication/failover/blob-copy regression anchors #612) — loop.
• Unified MCP tool registry (test(cluster): pin three node-lifecycle invariants — hdb_nodes reload auth, clone readiness, remove_node blast radius #615) — tool discovery.
• Operations MCP profile (test(stress): gate large-catchup on a throughput floor and emit host contention counters #617) — ops-as-tools.
• Application MCP profile (test(cluster): make blockCacheEviction actually wait for the restart; pin the pre-crash replication gap #618) — Resources-as-tools.
• harper-pro/licensing/usageLicensing.ts — exemplar for component shape (handleApplication, registerOperation, system table reads).
• harper-agent/tools/files/* — reusable scoped FS tool implementations to lift.

Verification

1. Build & boot: npm run build in harper-pro, start a server with the new env var; check logs for "Agent component initialized". No-op if agent.enabled=false.
2. Round-trip prompt: curl -X POST /agent_prompt -d '{"message":"describe the system schema"}' → returns session_id → curl /get_agent_session/<id> shows the agent called describe_all (from [MCP] Operations profile: tool generation over OPERATION_FUNCTION_MAP harper#617) and produced a response.
3. Autonomous build: prompt "create a Resource called Hello that returns 'hi'" → confirm a component file appears (via scoped FS tools), then curl localhost:9926/Hello to confirm the deployed app works.
4. Self-test loop: "build X, then call it on localhost:, debug any failures, repeat until working" — verifies http_fetch + tool chaining through Add agent-loop orchestration / toolMode: 'auto' to scope.models harper#612.
5. Debugging: with threads_debug=true, prompt "find why the Foo resource throws on POST" — should attach to inspector, set a logpoint, observe a request, report root cause.
6. Scheduled work: "every 5 minutes for the next hour, check cluster status and alert me if any node is down" — verifies schedule_followup + persistence across restarts.
7. Boundary checks: confirm read_file('/etc/passwd') is rejected; confirm inspector_* tools are NOT visible to tools/list over external MCP; confirm non-super_user callers of agent_prompt get 403.
8. Integration test at harper-pro/integrationTests/agent/ with a stub scope.models provider so CI doesn't burn LLM credits.