how to fetch from a local http server?

[panva]

Assuming a configuration such as

cat <<EOT > $(pwd)/tap/.workerd.capnp
using Workerd = import "/workerd/workerd.capnp";

const config :Workerd.Config = (
  services = [
    (name = "main", worker = .tapWorker),
  ],
);

const tapWorker :Workerd.Worker = (
  modules = [
    (name = "worker", esModule = embed "run-workerd.js")
  ],
  compatibilityDate = "$COMPATIBILITY_DATE",
);
EOT

Used for running a library test suite against a mock server like so

workerd test --verbose $(pwd)/tap/.workerd.capnp

I have the need to access a local running http server on a port 3000 using fetch() from within the worker.

I get the following error output from workerd

workerd/server/server.c++:3571: info: [ TEST ] main
workerd/util/symbolizer.c++:98: warning: Not symbolizing stack traces because $LLVM_SYMBOLIZER is not set. To symbolize stack traces, set $LLVM_SYMBOLIZER to the location of the llvm-symbolizer binary. When running tests under bazel, use `--test_env=LLVM_SYMBOLIZER=<path>`.
workerd/jsg/util.c++:281: error: e = kj/async-io-unix.c++:1651: failed: connect() blocked by restrictPeers()
stack: 106b9d49c 106b9dda8 106b9216c 106b925b4 105999017 105999907 10599b003 105978718 10597f2c0 10599549c 104b77a24 10511c734; sentryErrorContext = jsgInternalError
A hanging Promise was canceled. This happens when the worker runtime is waiting for a Promise from JavaScript to resolve, but has detected that the Promise cannot possibly ever resolve because all code and events related to the Promise's I/O context have already finished.
workerd/server/server.c++:3579: info: [ FAIL ] main

I'm able to make changes to the setup, e.g. to use an IP or a hostname to access the service if that's needed.

Any help and guidance for accessing the local http server is welcome.

[kentonv]

By default, Workers are only allowed to fetch from the public internet. This is to protect against SSRF attacks, where an application that tries to fetch user-provided URLs could be tricked by an attacker into fetching internal URLs that the attacker isn't supposed to be able to access. Since only public URLs can be fetched, such an attack is not possible by default.

There are two ways you can give your worker access to internal servers. I'll show examples below, but remember the full config format is documented in workerd.capnp.

Preferred approach: Define a service binding of type ExternalServer.

using Workerd = import "/workerd/workerd.capnp";

const config :Workerd.Config = (
  services = [
    (name = "main", worker = .myWorker),
    (name = "privateServer", external = .myPrivateServer),
  ],
);

const myWorker :Workerd.Worker = (
  modules = [
    (name = "worker", esModule = embed "worker.js")
  ],
  bindings = [
    (name = "PRIVATE_SERVER", service = "privateServer")
  ],
  compatibilityDate = "2023-12-18",
);

const myPrivateServer :Workerd.ExternalServer = (
  address = "127.0.0.1:3000",
  http = ()
);

Now in your code you can fetch from your private server:

export default {
  async fetch(req, env, ctx) {
    let resp = await env.PRIVATE_SERVER.fetch("http://127.0.0.1:3000/foo");
    // ...
  }
}

With this approach, you must explicitly use env.PRIVATE_SERVER.fetch() to reach the private server -- global fetch() still cannot talk to it. This protects you against SSRF, because it means your app will only talk to the private server when it explicitly intended to. So, the app can safely fetch() user-provided URLs without worrying that an attacker can provide the URL of the private server.

Dangerous approach: Disable SSRF protection.

You can redefine the default network for fetch() requests so that it includes private networks. To do this, define a service of Network type which allows all addresses, and set your Worker's globalOutbound to point to it.

using Workerd = import "/workerd/workerd.capnp";

const config :Workerd.Config = (
  services = [
    (name = "main", worker = .myWorker),
    (name = "fullNetwork", network = .myNetwork),
  ],
);

const myWorker :Workerd.Worker = (
  modules = [
    (name = "worker", esModule = embed "worker.js")
  ],
  globalOutbound = "fullNetwork",
  compatibilityDate = "2023-12-18",
);

const myNetwork :Workerd.Network = (
  # Allow all addresses. (You can also list CIDRs here.)
  allow = ["public", "private"],
  deny = []
);

Now you can fetch() to internal addresses. However, you've lost SSRF protection. I strongly recommend against using such a configuration in production.

[panva]

@kentonv thank you so much for guiding me through this, since my use case is testing a library under its different target runtimes in CI (hence only using workerd test), the approach where I don't need to change fetch() for env.PRIVATE_SERVER.fetch() is just perfect for my needs.