Let operators turn off or host-limit the agent's http_fetch (agent.httpFetch)

[DavidCockerill]

Operators can now remove the built-in agent's http_fetch tool or limit it to named hosts with a new boot-time agent.httpFetch setting: false drops the tool from the toolset, and { allow: [...] } refuses any request, including any redirect hop, whose host is not listed. Redirects are now checked hop by hop in every mode, which also stops a redirect into the cloud-metadata blocklist, and the blocklist now catches the AWS IPv6 IMDS address and IPv4-mapped spellings it used to miss.

Closes #2974 · Docs: HarperFast/documentation#708

agent:
  enabled: true
  httpFetch:
    allow:
      - localhost:9926 # host:port, or host for any port
      - '*.example.com' # any subdomain, not the apex

For the human reviewer

1. Redirects are followed by hand in every mode, including the default. The planning review advised against this, and I overruled it. The planning review (Codex, better-alternative-exists) proposed leaving native fetch redirect-following in place for httpFetch: true and following hops manually only under an allow-list, so the default path changes nothing. I kept one path. Today a single redirect defeats the tool's own metadata blocklist (from 169.254.169.254 to the AWS IPv6 IMDS address), and the split would leave that open in the mode every enabled deployment runs, Fabric's default render included. The only default-mode behavior change is that a redirect into the blocklist now fails. The cost is about 0.17 ms over three loopback redirects (median 5.18 ms native vs 5.35 ms, 500 runs). Reversible: the default path could go back to redirect: 'follow' in one line, re-opening the bypass. The hop step re-implements fetch's redirect rules (method rewrite, body-header removal, credential headers dropped across origins, 20-hop cap). The tests pin it to native fetch's observed behavior, but it is now code we own.
2. The requirement itself. As specified in the issue: off, or a host allow-list, boot-time only. It is warranted because the agent's read path (tables, logs, component source) and an unrestricted egress path share one toolset. Host-level egress policy can't tell agent fetches from Harper's other outbound traffic, and Fabric operators don't control the host. It does not bound other egress the operations tools might open if mcp.operations.allow is widened (for example deploy_component's package source). That is a separate decision, and the design note says so.
3. Boot-time only, but settable from every config route. set_agent_config rejects an httpFetch patch with a 400 instead of ignoring it, so an operator can't tighten it at runtime either; an emergency stop is a restart. Allowing runtime narrowing only would be easy to add later. The key is registered in CONFIG_PARAMS like every other agent_* key, so AGENT_HTTPFETCH=false, --AGENT_HTTPFETCH false, AGENT_HTTPFETCH_ALLOW='["…"]' and set_configuration all work, each taking effect on restart. My first draft left it out, so that set_configuration couldn't rewrite the policy. That left AGENT_HTTPFETCH=false silently ignored, with the tool fully on, and the protection it bought was thin. set_configuration is destructive and outside the agent's default toolset, needs a restart (also destructive) to take effect, and can already write agent_allowDestructive and agent_user. The residual risk the delta review named is an operator who opts set_configuration and restart into mcp.operations.allow with autoApprove: true. That agent could widen its own egress across one restart, but that configuration already hands it the whole config.
4. A malformed value removes the tool, not the agent. The planned design threw at boot, which takes every agent operation down on a typo. Now the boot log names the bad value and the agent runs without http_fetch. An empty allow-list also removes the tool, since a tool that refuses everything only wastes model turns.
5. Shipped matching semantics, which are hard to tighten later. An entry without a port matches any port, so localhost also admits the operations API port. *.example.com covers any depth but not the apex. Matching is by host name: an allowed name that resolves to an internal address is reached, and the docs say to list names whose DNS you control. Resolving and pinning addresses would need a custom dispatcher, which the planning note rejected because it sees the proxy peer, not the target.

Product and architecture tour

What an operator can set

What can the agent reach, and who can change that?

Before	After
http_fetch is always in the toolset. It refuses only cloud-metadata hosts by literal name, and follows redirects without checking them.	agent.httpFetch: true keeps that reach, but every redirect hop is now checked. false removes the tool. { allow: [...] } refuses any host not listed, and the tool description names the listed hosts.

The policy is read once, at boot. set_agent_config rejects it, and re-composing the toolset after an allowDestructive change reuses the tool built at boot. The config file, HARPER_CONFIG, env vars, CLI flags and set_configuration set it for the next boot.
With the tool disabled, no registry or extra tool can take the http_fetch name.
A malformed value or an empty allow-list removes the tool and leaves the rest of the agent running.

Example: an investigation role with no egress

An operator enables the agent to read logs and tables, and sets httpFetch: false. The boot log reports 29 tools (http_fetch: disabled). The model's tool list has no http_fetch, and the system prompt no longer tells it to verify work over HTTP.

Outcome: A prompt-injected instruction to send data somewhere has no tool to do it with.

• Tool built once from the boot config — compose() receives this instance, never the live config
• set_agent_config refuses the key — before any other key is applied

Every hop is checked before it is sent

Why does the tool follow redirects itself?

fetch with redirect: 'follow' sends the next request before any caller code can see where it goes, so a host check on the first URL says nothing about the second. The tool now requests each hop with redirect: 'manual', cancels the redirect body, resolves Location, and runs the same target check before the next request. Method, body and credential-header handling copy fetch's own redirect step, so allowed redirects behave as they did before.

A redirect toward a host outside the policy

The refused host is never contacted; the model gets a policy error naming it.

sequenceDiagram
    participant Model
    participant Tool as http_fetch
    participant Allowed as allowed host
    participant Other as other host
    Model->>Tool: fetch allowed host
    Tool->>Tool: check target
    Tool->>Allowed: request, redirect manual
    Allowed-->>Tool: 302 Location other host
    Tool->>Tool: cancel body, check Location
    Tool-->>Model: refused by policy
    Note over Other: never contacted

Loading

• The hop loop — check, cancel, rewrite, cross-origin header removal
• The target check — scheme, then metadata and link-local, then the allow-list

How a host matches

What does an allow-list entry admit?

host matches that host on any port; host:port matches only that effective port (80 or 443 when the URL has none).
*.example.com matches any subdomain at any depth, never example.com itself.
Entries and targets compare after URL host parsing, so case, IDNA and IPv4 shorthand compare equal. An IPv4 entry does not admit its IPv4-mapped IPv6 form.
The metadata and link-local check runs first and compares addresses with net.BlockList, so listing a metadata address never admits it.

Example: a wildcard entry

With allow: ['*.example.org', 'api.example.com:8443'], https://a.b.example.org/ is allowed. http://example.org/ is refused, as is https://api.example.com/, whose effective port is 443.

Outcome: An apex domain and every port pin are explicit decisions in the list.

Changes

• agent/tools/httpFetchTool.ts:
  • resolveHttpFetchConfig validates the raw setting.
  • parseAllowEntry normalizes each entry and rejects a bracketed host that is not a plain IPv6 address.
  • checkHttpFetchTarget is the per-target gate, ending in the refused-by-policy error. The blocklist now includes the AWS IPv6 IMDS address.
  • buildHttpFetchTool replaces the module constant and lists the allowed hosts in the description.
  • The request method is upper-cased, so redirect rewriting matches fetch.
  • The hop loop cancels each redirect body and checks the next target before requesting it.
• agent/toolset.ts: composeToolset takes the policy-built tool and reserves the http_fetch name.
• agent/agent.ts:
  • the tool is built once and passed to every compose
  • mergeConfig resolves the setting
  • the system prompt drops fetch when the tool is absent
  • the boot log reports the policy
• agent/operations.ts: set_agent_config rejects httpFetch.
• agent/types.ts: HttpFetchConfig and AgentConfig.httpFetch.
• utility/hdbTerms.ts: AGENT_HTTPFETCH and AGENT_HTTPFETCH_ALLOW in CONFIG_PARAMS, so per-key env vars, CLI flags and set_configuration reach the setting.
• config-root.schema.json: agent.httpFetch as boolean or { allow: string[] } with an entry pattern.
• agent/DESIGN.md, DESIGN.md: the invariant note and its index line.

Verification

Route: live smoke (c). The agent can't run a tool without a model, so no integration test reaches it. I booted the built dist from a scratch root against a local fake OpenAI-compatible model that answers the first prompt with three http_fetch calls: an allowed host, a denied host, and an allowed host that 302s to the denied one. Each row is a fresh boot:

agent.httpFetch	Boot log	Tool results	Denied server hits
omitted	30 tools (http_fetch: enabled)	all three 200	2 (unchanged behavior)
{ allow: ['127.0.0.1:18081'] }	30 tools (http_fetch: allow-list 127.0.0.1:18081)	allowed 200; denied and redirect: refused by policy: 127.0.0.1:18082 is not in agent.httpFetch.allow	0
false	29 tools (http_fetch: disabled)	unknown_tool; model's tool list has no http_fetch, system prompt has no fetch mention	0
{ allow: [] }	29 tools (http_fetch: disabled)	unknown_tool	0
"nope"	error agent.httpFetch must be true, false, or { allow: [...] }; got "nope"; http_fetch is disabled…, then 29 tools	unknown_tool	0

The tool description in the model request listed the allowed host, and set_agent_config {httpFetch: true} returned agent.httpFetch is fixed at startup… in every row. The smoke ran at 8e24c38; later commits only fix an IPv6 zone-ID error message, tighten the design note, and merge main.

The config routes, each on a fresh root at fad656c:

Route	Boot log after restart
AGENT_HTTPFETCH=false	29 tools (http_fetch: disabled)
AGENT_HTTPFETCH_ALLOW='["127.0.0.1:18081"]'	30 tools (http_fetch: allow-list 127.0.0.1:18081); denied and redirected calls refused, denied server 0 hits
harper run --AGENT_HTTPFETCH false	29 tools (http_fetch: disabled)
set_configuration {agent_httpFetch_allow: ["127.0.0.1:18081"]}	allow-list 127.0.0.1:18081, enforced end to end
set_configuration {agent_httpFetch: false}, restart, then set_configuration {agent_httpFetch_allow: [...]}	disabled, then allow-list 127.0.0.1:18081. The boolean parent is replaced, which the delta review had doubted.
AGENT_HTTPFETCH_ALLOW='127.0.0.1:18081,*.example.com'	error agent.httpFetch.allow must be a list of hosts; got "…", then 29 tools (http_fetch: disabled)

Unit tests:

• unitTests/agent/httpFetchTool.test.js:
  • config resolution and every rejection class
  • the matching matrix: ports, wildcard depth and apex, IPv6, IDNA, IPv4 shorthand, trailing dot
  • metadata hosts stay blocked when listed
  • the IPv6 IMDS address in any spelling
  • against real loopback servers: refusal before any connection, redirect refusal, method/body handling matched against native fetch, credential headers, the 20-hop cap, redirect-body release on refusal, one timeout across hops
• unitTests/agent/registryTools.test.js — the reserved name.
• unitTests/agent/operations.test.js — the set_agent_config 400.

The redirect-body test fails when the cancel is removed. On origin/main, http://[fd00:ec2::254]/ and a redirect to 169.254.169.254 both passed the policy and attempted the connection; both are refused here.

Gates, run locally on macOS:

• npx mocha "unitTests/agent/*.test.js": 158 passing, re-run after the merge from main.
• npx mocha "unitTests/config/**/*.test.js" "unitTests/agent/*.test.js": 528 passing at fad656c, after the CONFIG_PARAMS change.
• test:unit:main: 6410 passing, 18 failing. I ran it with unitTests/components/applicationSpawn.test.js excluded, because that file hangs on macOS on origin/main too (harper#2538).
• test:unit:resources: 3915 passing, 3 failing.
• All 21 unit failures reproduce identically on a clean origin/main build of the same files, and none are in agent code. Causes: /var vs /private/var tmpdir paths, a git tag fixture under a signing config, native addons without install scripts, process-group harnesses, and RocksDB flush-premise tests. The Linux CI run is the real gate for these.
• test:integration:all: 2248 passing, 11 failing, 18 skipped. All 11 are worker-thread restart and isolation tests (isolated-application, shutdown-drain-e2e, record-lock-concurrency, log-rotation-write-path, rolling-restart), and they fail the same way on a clean origin/main run of those five files: a UDS mirror path over the macOS socket-path limit, and a single HTTP worker. None of these configs loads the agent.
• npm run lint reports 13 warnings, all in files this PR doesn't touch; prettier --check and check:design-docs pass on the changed files. The schema was checked with ajv against 12 valid and invalid httpFetch documents.

Complexity: medium

🤖 Generated with Claude Code

_{Review-Coverage: authored=claude; ran=gemini,codex; adjudicated=domain; blocked=cursor-grok(not-installed); declined=cursor-composer,cursor-kimi,cursor-muse; rounds=2; full=1 @ fad656c}

_{Human-Review-Need: 4 (decisions: restart-effective-config-writes, boot-only-policy, default-unrestricted, manual-redirects-all-modes, malformed-disables-tool, portless-entry-any-port, name-only-matching, scope-http-fetch-only) @ fad656c}

[github-actions]

Release cherry-pick v5.3: cancelled

Cherry-pick branch cherry-pick/v5.3/pr-3011 was deleted — this PR no longer targets v5.3 (milestone is now v5.4).

[kriszyp]

Great!

_{🤖 Reviewed with Codex}

[kriszyp]

HostRule is used by functions in this module, but nothing in this checkout imports the interface itself. Could we remove export? That keeps this tool-internal type out of the module API while retaining its local type checks.