Future of the Node HTTP Client

[ronag]

I see a slight problem with the situation with the node core http client. We are getting issues reported in the issue tracker. However, there is little interest by contributors (that are familiar with the code) to look into these issues and resolve them.

Part of the reason for this is that some of the people that usually engage with http client issues is @mcollina, @dnlup or myself. Whom I believe are mostly focusing on nodejs/undici, which we feel is a better alternative and the future, and have limited interest in putting further effort into the node core http client.

This is a slightly suboptimal situation. I would personally like to discuss whether and how we could encourage users over to NodeJS/undici and eventually deprecating the current node client.

Maybe add something in the docs that point to undici. However, I don’t think this is something we have done in the past.

Thoughts?

[DerekNonGeneric]

Maybe add something in the docs that point to undici. However, I don’t think this is something we have done in the past.

I found some prior art that may help: our util.log(string) core function recommends using a third party module, but it's deprecated, which I'm unsure whether would be an appropriate designation in this situation (perhaps more likely legacy).

Stability: 0 - Deprecated: Use a third party module instead.

Refs: https://nodejs.org/dist/latest-v16.x/docs/api/util.html#util_util_log_string

[mcollina]

I share @ronag concerns and I would also add one of my own.

The reason why very few would like to work on the HTTP client is because every change breaks the huge number of modules in the ecosystem that monkeypatch it. Fixing the majority of those bugs requires either breaking some other users or finding some clever hacks that will come hunting us back over the years.

[Trott]

Since this is on the TSC agenda: @nodejs/tsc

[benjamingr]

I think @jasnell is talking about shipping WHATWG ReadableStream this year (right?), we can also land the polyfill in core in the meantime and just ship fetch (on top of unidici perhaps).

Then we can direct people to the new API and put the old API in maintenance mode and possibly eventually even deprecate it.

Another possible alternative would be to land unidici as part of core (or as a dep) and build the existing HTTP API on top of it (kine of like what browsers did with fetch and XHR or how domains build on async_hooks now).

[mcollina]

Another possible alternative would be to land unidici as part of core (or as a dep) and build the existing HTTP API on top of it (kine of like what browsers did with fetch and XHR or how domains build on async_hooks now).

It's not possible. The current API is an extremely leaky abstraction that multiple modules in the ecosystem monkeypatch. Doing so would not reduce the friction for the community.

[benjamingr]

It's not possible. The current API is an extremely leaky abstraction that multiple modules in the ecosystem monkeypatch. Doing so would not reduce the friction for the community.

I'm sure this is pretty common in servers - but who's patching the clinet?

[targos]

I'd prefer we keep the existing API with a legacy status and introduce a new one based on undici.

[benjamingr]

I'd prefer we keep the existing API with a legacy status and introduce a new one based on undici.

Note that if landing fetch is an eventual goal that would mean we'd have to maintain 3 HTTP APIs. I am fine with this if the fetch we land is based on unidici to the point we only de-facto maintain two APIs and fetch is just a wrapper around that

[ronag]

Just a side note. I actually prefer if undici is not included in core. It is much easier to maintain and develop as npm module.

[ronag]

I'd prefer we keep the existing API with a legacy status and introduce a new one based on undici.

Note that if landing fetch is an eventual goal that would mean we'd have to maintain 3 HTTP APIs. I am fine with this if the fetch we land is based on unidici to the point we only de-facto maintain two APIs and fetch is just a wrapper around that

There is https://github.com/Ethan-Arrowood/undici-fetch which we are considering merging into undici once it's a bit more mature.

[targos]

It would be unfortunate to have to recommend people to use an external module for doing HTTP requests...

[benjamingr]

@ronag

Kind of OT on your side note:

Just a side note. I actually prefer if undici is not included in core. It is much easier to maintain and develop as npm module.

I think that's a bigger problem and we need a process for deps to live in core without requiring a full CI run or the same process. The reason it's much nicer to maintain/develop outside of core is (presumably, from stuff I was involved in):

• You don't need to wait 48 hours (or a week) to land PRs.
• You don't need to wait for mostly flakey CI that's mostly irrelevant.
• You can easily set up your own infra without needing to bother more people.
• You don't need to go through the consensus seeking process.

It would be great if we could solve that and make contributing relatively standalone stuff (like unidico or AbortSignal) more convenient and less tedious.

One approach you may want to take is to keep the nodejs/unidici repo and treat it like a dep - that way you'd get the same "freedom" stuff like libuv gets

[benjamingr]

There is https://github.com/Ethan-Arrowood/undici-fetch which we are considering merging into undici once it's a bit more mature.

That would be cool though I would prefer it if users had fetch as a convenient API and unidici (presumably with a more "standard" name) as a low-level API (assuming callback-land http becomes legacy).

[ronag]

@ronag

Kind of OT on your side note:

What does OT mean?

I think that's a bigger problem and we need a process for deps to live in core without requiring a full CI run or the same process. The reason it's much nicer to maintain/develop outside of core is (presumably, from stuff I was involved in):

Yea I get that. Just thinking that it would be nice if possible.

[ronag]

There is https://github.com/Ethan-Arrowood/undici-fetch which we are considering merging into undici once it's a bit more mature.

That would be cool though I would prefer it if users had fetch as a convenient API and unidici (presumably with a more "standard" name) as a low-level API (assuming callback-land http becomes legacy).

Yea, that's the idea. In terms of API we have a bit of a layered approach in undici. Whether or not we split undici into minimal packages (e.g. undici-core, undici-pool, undici-agent, undici-api, undici-fetch) or just have everything together (undici) is something to consider.