`http` requests with `socketPath` lead to `EPIPE` error

[mhassan1]

Version

v16.7.0

Platform

Linux e332ae582690 5.10.47-linuxkit #1 SMP Sat Jul 3 21:51:47 UTC 2021 x86_64 Linux

Subsystem

No response

What steps will reproduce the bug?

Start a server and client using the following scripts:

server.js:

const http = require('http')
http.createServer((req, res) => {
  res.writeHead(200, {'Content-Type': 'text/plain'})
  res.end('ok')
}).listen('sock.sock')

client.js:

const http = require('http')
;(async () => {
  for (let i = 0; i < 1e4; i++) {
    const result = await new Promise((resolve, reject) => {
      const request = http.request({
        method: 'POST',
        socketPath: 'sock.sock',
        headers: {
          'Content-Length': 2
        }
      }, (resp) => {
        resp.statusCode === 200 ? resolve('ok') : reject(new Error(resp.statusCode))
      })
      request.write('{}', null, () => {
        request.end()
      })
    })
    console.log(i, result)
  }
  console.log('done')
})()

On v16.7.0, this leads to an EPIPE error after a few hundred requests. On v16.6.2, it does not hit an EPIPE error after many thousands of requests.

A full Docker Linux reproduction is here: https://github.com/mhassan1/node-16-7-0-request-epipe

How often does it reproduce? Is there a required condition?

It reproduces every time within a few hundred requests.

What is the expected behavior?

No EPIPE.

What do you see instead?

EPIPE after a few hundred requests:

node:events:371
      throw er; // Unhandled 'error' event
      ^

Error: write EPIPE
    at afterWriteDispatched (node:internal/stream_base_commons:164:15)
    at writeGeneric (node:internal/stream_base_commons:155:3)
    at Socket._writeGeneric (node:net:780:11)
    at Socket._write (node:net:792:8)
    at writeOrBuffer (node:internal/streams/writable:389:12)
    at _write (node:internal/streams/writable:330:10)
    at Socket.Writable.write (node:internal/streams/writable:334:10)
    at ClientRequest._writeRaw (node:_http_outgoing:362:17)
    at ClientRequest._send (node:_http_outgoing:338:15)
    at ClientRequest.end (node:_http_outgoing:880:10)
Emitted 'error' event on ClientRequest instance at:
    at Socket.socketErrorListener (node:_http_client:447:9)
    at Socket.emit (node:events:394:28)
    at emitErrorNT (node:internal/streams/destroy:157:8)
    at emitErrorCloseNT (node:internal/streams/destroy:122:3)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  errno: -32,
  code: 'EPIPE',
  syscall: 'write'
}

Additional information

• This issue is visible when using axios with follow-redirects
• This may be the same underlying issue as Simple http proxy with socketPath leads to EPIPE error #40553
• This does not reproduce when making requests on a port
• This reproduces on v14.18.0 but not v14.17.6

[mhassan1]

I believe the key part of the reproduction is the request.write followed by request.end. If I change the following, it no longer reproduces:

// change
request.write('{}', null, () => {
  request.end()
})
// to
request.end('{}')

This may be a viable workaround, although userland packages like follow-redirects and http-proxy will need to change the way they make requests.

[lpinca]

From a quick look the problem is in the server code. The socket is destroyed when res.end('ok') is called before the request is full written. It should be

const http = require('http')
http.createServer((req, res) => {
  req.resume()
  res.writeHead(200, {'Content-Type': 'text/plain'})
  req.on('end', () => {
    res.end('ok')
  });
}).listen('sock.sock')

[lpinca]

Waiting for the 'end' event as per my previous comment does not seem to make any difference. I think there is a race condition where the socket is closed too early.

[lpinca]

I can reproduce with Node.js v16.7.0 but not with Node.js v16.6.2 so I guess this is caused by #39525.

[mhassan1]

@lpinca Do you think this is a bug or an intended change in behavior?

[lpinca]

It is a bug. With #41062 (comment) server code no error should be emitted.

[mhassan1]

Do you think it's a bug in libuv or node?

[lpinca]

I honestly don't know. The libuv update might have uncovered a latent Node.js bug.

[lpinca]

cc: @vtjnash @nodejs/libuv

[lpinca]

If I revert libuv/libuv@23bebf0 I can no longer reproduce the issue.

cc: @twose

[vtjnash]

It sounds somewhat similar to the bug I fixed in nodejs's TLS around connection management (#36111). That libuv commit is not the primary cause, but it does exacerbate some known bugs in nodejs's network stack. However, it doesn't look like this is using TLS, so that commit won't itself fix this bug. Instead, this is probably the same as the related issues there #39363 and #39683, whereby a malicious client apparently can cause recent versions of nodejs http server to abort the process, by disconnecting network access in the middle of communication.

[twose]

I agree with @vtjnash , there are no incompatible or destructive behavior changes in the libuv patch, it just simplifies the code.
And I'm so sorry that I am not familiar with nodejs, so maybe there's nothing I can do to help...

[lpinca]

I cannot reproduce the issue if I remove the HTTP layer (using only the net module) and I also cannot reproduce the issue if I use the legacy HTTP parser via the --insecure-http-parser flag.

[lpinca]

One more thing. I cannot reproduce the issue if the Content-Length header or alternatively request.end() is removed from the client code. My guess is that the server closes the socket after reading the two bytes the of the request body honouring the Content-Length header. The client then tries to write something on the closed socket via request.end() and this raises the EPIPE error.