HTTP Server close is taking up to 5 seconds in node 18

[DanielRamosAcosta]

Version

v18.18.2

Platform

Darwin MacBook-Pro-de-Daniel.local 23.0.0 Darwin Kernel Version 23.0.0: Fri Sep 15 14:43:05 PDT 2023; root:xnu-10002.1.13~1/RELEASE_ARM64_T6020 arm64

Subsystem

http

What steps will reproduce the bug?

1. Create an http server
2. Start listening
3. Take at least one request
4. Try to close the server
5. It takes exactly 5 seconds

How often does it reproduce? Is there a required condition?

I have only reproduced the bug in node 18, but is deterministic, it happens every time

What is the expected behavior? Why is that the expected behavior?

In node 19 and 20 it's only taking 0.5 seconds. Thats a normal time that I would expect.

What do you see instead?

I see 5 seconds of delay in order to close

Additional information

I've reproduced the error in this repository with a bare minimum with Github Actions: DanielRamosAcosta/nodejs-close-server-bug

Here is the CI report:

• Node 18: server close: 4.990s
• Node 19: server close: 0.652ms
• Node 20: server close: 0.59ms

This is affecting to a library I'm maintaining similar to supertest. If you see the CI there, Node 18 tests are taking very long due to the servers being closed.

[H4ad]

Started on 18.18.1, the 18.18.0 does not have this weird delay.

EDIT: it has the delay also on 18.18.0

[himself65]

Started on 18.18.1, the 18.18.0 does not have this weird delay.

No, I think all 18 version has such issue

[himself65]

Related PR: #48383

[himself65]

The hot fix:

import * as http from "node:http";
import { promisify } from "node:util";

const server = http.createServer(async (req, res) => {
    res.writeHead(200, { "Content-Type": "text/plain" });
    res.write("Hello world");
    res.end();
});
const listenPromisied = promisify(server.listen.bind(server));
const closePromisied = promisify(server.close.bind(server));

await listenPromisied(0, "127.0.0.1");
const address = server.address();
await fetch(`http://${address.address}:${address.port}`);
console.time("server close");
+ server.closeAllConnections();
await closePromisied();
console.timeEnd("server close");

[H4ad]

Should we bisect or closeAllConnections is expected to be called?

[himself65]

The latest behavior is to close all sockets, I think there are some backport issues on 18. Let me see if I can do the backport

[H4ad]

The 19.0.0 looks like the first release without this delay.

I will try bisect to see if I can find the commit that fixes the delay and then we can backport.

The #48383 was released on 20.4.0, and this issue started on 18.0.0 (or even earlier, I didn't try rewrite the fetch call)

[H4ad]

@himself65 bisect will not help in this case:

The merge base 19064bec341185a8c15fc438cfcf0df8633a179e is bad.
This means the bug has been fixed between 19064bec341185a8c15fc438cfcf0df8633a179e and [cc993fb2760d01457955f5b9ff787d559ed1c34e].

From what I understand, 18.x and 19.x have different git histories, so there's no way to find the bad commit using bisect.
At least, I don't know a way to do it, I'm open to suggestions.

[himself65]

I tried nodejs 16.x. now I think there might have a regression on nodejs 16 -> 18

import * as http from "node:http";
import { promisify } from "node:util";
import fetch from 'node-fetch'

const server = http.createServer(async (req, res) => {
    res.writeHead(200, { "Content-Type": "text/plain" });
    res.write("Hello world");
    res.end();
});
const listenPromisied = promisify(server.listen.bind(server));
const closePromisied = promisify(server.close.bind(server));

await listenPromisied(0, "127.0.0.1");
const address = server.address();
await fetch(`http://${address.address}:${address.port}`);
console.time("server close");
await closePromisied();
console.timeEnd("server close");

➜  nodejs git:(main) ✗ node index.mjs
server close: 0.158ms
➜  nodejs git:(main) ✗ node -v
v16.20.2

[H4ad]

It's a regression from 17.x -> 18.x

h4ad:node-copy-4/ (main✗) $ node test-close-2.mjs                                                                                                                                                                                                                    
server close: 0.108ms
h4ad:node-copy-4/ (main✗) $ node -v
v17.9.1

[H4ad]

Based on https://github.com/nodejs/node/pull/49091/files#diff-d692ac4524379ec6a1201165e8ff8d3267c8130e07014e8221ebf7e6f80c6641R1560-R1567, and this little change:

const server = http.createServer({ keepAliveTimeout: 200 }, async (req, res) => {

h4ad:node-copy-4/ (main✗) $ node test-close-2.mjs
server close: 0.255ms
h4ad:node-copy-4/ (main✗) $ node -v
v18.18.1

---

@himself65 I think you are right, the #48383 should fix the issue, I thought it landed on 18.18.0 but it did not, sorry for the confusion.

Let me know if you will do the backport, if not, I can do that.

[H4ad]

I built 18.18.0 with fix #48383, it solves the delay.

[himself65]

Based on https://github.com/nodejs/node/pull/49091/files#diff-d692ac4524379ec6a1201165e8ff8d3267c8130e07014e8221ebf7e6f80c6641R1560-R1567, and this little change:

const server = http.createServer({ keepAliveTimeout: 200 }, async (req, res) => {

h4ad:node-copy-4/ (main✗) $ node test-close-2.mjs
server close: 0.255ms
h4ad:node-copy-4/ (main✗) $ node -v
v18.18.1

@himself65 I think you are right, the #48383 should fix the issue, I thought it landed on 18.18.0 but it did not, sorry for the confusion.

Let me know if you will do the backport, if not, I can do that.

I might don't have time on backport you can try that.

I read some CIGTM code, i have no idea on this #48383 (comment)