Make it possible to use Fetch with proxies or other agents

[pimterry]

What is the problem this feature will solve?

The new fetch API as implemented cannot be used with an HTTP proxy, which is required for connectivity in many environments.

For normal HTTP this is implemented via agents, but there's no way to use any agents with this fetch API. Many of us working in environments with proxies use libraries like global-agent which set node's globalAgent to a proxy agent based on the system settings to automatically configure all libraries, but that doesn't work with fetch either.

Notably, this means the docs are wrong right now, when they say:

http.globalAgent: Global instance of Agent which is used as the default for all HTTP client requests.

This agent is not used for HTTP client requests if you use the fetch API.

Node's fetch implementation comes from Undici, and although Undici doesn't offer an explicit way to set this per fetch request (see nodejs/undici#1350) it does offer an agent-equivalent dispatcher option on all other request methods, and a setGlobalDispatcher method to configure a dispatcher globally (like node's globalAgent) which does work for fetch.

That means it is possible to use proxies with Undici's fetch right now, but not in Node as this isn't exposed anywhere (AFAICT).

What is the feature you are proposing to solve the problem?

• Provide a way to specify a Node.js agent or a Undici dispatcher for a fetch request
• Provide a way to set a global agent/dispatcher, which applies to all HTTP & fetch requests

Since they're very closely related, it seems like it would be sensible to aim to move everything to either agents or dispatchers for all HTTP APIs in future. While fetch is experimental though it seems reasonable to me to implement this only with Undici's existing dispatchers for now and pick one direction or the other to commonize later.

What alternatives have you considered?

As far as I can tell, there's currently no alternative or workaround available to use proxies with fetch in Node. If you need to use an HTTP proxy for connectivity, the current fetch API is unusable.

This is particularly bad because some libraries that support both browsers & node will use the fetch global automatically when available or node-fetch otherwise (which uses http internally) for their requests. Although it used to be possible to use these libraries in a proxy environment by using global-agent or passing an agent explicitly, it's now impossible to use these libraries at all, because they use the new fetch global which ignores all agent configuration.

[targos]

@nodejs/undici

[panva]

Note: the same applies to undici's mocking ability and common userland modules that achieve mocking for node's http/https modules nock. Undici has the affordance, but it is not exposed in Node.js and so node's global fetch cannot be mocked.

[mcollina]

fetch in Node.js is experimental, all of this is planned for when it exits experimental. Could you check out undici and see if it matches your expectations?

On a different note, would it be easier to detect if the global fetch property was non-enumerable?

[pimterry]

Could you check out undici and see if it matches your expectations?

Undici's dispatchers & setGlobalDispatcher on the core request methods (request/stream/etc) do work great for me, yes.

Undici's fetch doesn't have its own dispatcher option available though (nodejs/undici#1350), which is a bit inconvenient (I'm mostly setting a global dispatcher anyway, so in my case it's not a big issue).

It would be a little more useful if it directly supported agents, or there was a dispatcher<->agent compatibility wrapper, since there's so many existing node.js agent implementations that could be reused, but again not a big deal.

On a different note, would it be easier to detect if the global fetch property was non-enumerable?

I don't think enumerability matters here, most checks are just if (!global.fetch) { addPolyfill() } e.g. in isometric-fetch and cross-fetch.

Those libraries are quite widely used (5 million + 8 million installs a week via npm, plus other similar libs & inline checks elsewhere). Everybody using those with Node 18 is automatically using this new experimental implementation everywhere now.

That's how I ran into this myself: my test code had the same inline if-no-fetch-then-ponyfill check, and when I ran my tests in node 18 they all failed because they switched to node's fetch and started ignoring the configured HTTP proxy.

[mcollina]

@nodejs/tsc did we expose fetch too early?

[targos]

@nodejs/tsc did we expose fetch too early?

Not sure. It's difficult to say if we would have gotten useful feedback with the API still behind a flag.

@pimterry note that you can opt out of node's fetch with --no-experimental-fetch.

[panva]

@nodejs/tsc did we expose fetch too early?

The problem isn't exposed fetch. It's that it doesnt utilize node's global agents due to its use of undici. There's a whole ecosystem of mocking and proxying libraries that relied on the fact that regardless of what http client / library you used in node, it always boiled down to http/s.request and its use of global agents

[panva]

fetch in Node.js is experimental, all of this is planned for when it exits experimental

In hindsight, a global that implements a Web Platform API that's actively being polyfilled probably shouldn't be exposed when still experimental / unstable. That being said

It's difficult to say if we would have gotten useful feedback with the API still behind a flag.

Exposing it via a node:fetch module first would probably allow for more feedback to trickle in. WebCrypto API was also not exposed as global and was first added to the crypto module as an export while experimental.

[ronag]

https://github.com/orgs/nodejs/teams/tsc did we expose fetch too early?

Maybe. But I'm not sure what waiting even more would have made any difference? The same things break. The polyfills are still using node streams and http global agents so the primary breakage would be the same.

[mcollina]

I think we should seriously consider exposing node:undici, or possibly node:http-next. What do you think?

[ronag]

I like the idea!