http2: How should I be avoiding "New streams cannot be created after receiving a GOAWAY" errors

[murgatroid99]

• Node.js Version: 10+
• Scope (install, code, runtime, meta, other?): usage
• Module (and version) (if relevant): http2

I maintain the module @grpc/grpc-js which uses the http2 module. I have gotten reports of occasional "New streams cannot be created after receiving a GOAWAY" errors. The code I have listens for the close, error, and goaway events, and if it sees any of them it stops using that session. Is that the intended usage pattern to avoid that kind of error? Is there anything else I should be checking to more consistently avoid that error?

[addaleax]

@nodejs/http2

[murgatroid99]

Can anyone help me with this?

[mcollina]

After receiving a GOAWAY you cannot create new streams on the sessions. This is part of the http2 protocol. It is not clear what kind of help do you need on this.. would you mind to add some code to reproduce?

[trivikr]

As @mcollina mentioned, here's relevant HTTP/2 spec which explains new streams can't be created after GOAWAY frame https://http2.github.io/http2-spec/#GOAWAY

GOAWAY allows an endpoint to gracefully stop accepting new streams while still finishing processing of previously established streams.

[murgatroid99]

I understand that new streams cannot be created after receiving a GOAWAY. I am trying to understand how to use the Http2Session API to ensure that that requirement is met. As stated in my question, I am already listening for the goaway event, and not starting new streams on that session after that, but I am still occasionally getting the quoted error. To me this implies that there is something else I should be doing to avoid that and I am trying to understand what that something else is.

[mcollina]

I think having an example that reproduce the error would help, as well as a stacktrace.

[gireeshpunathil]

I don't know much about http2, but just wondering if this code serves any clues:

• If you close the client too quick, then the error is emitted on the client request with goaway as the error type
• sometimes a goaway event is emitted on the session object too
• if you close the client after a while, the request object does not receive anything

is this indication of any racy behavior?

$ cat 2105.js

const h = require('http2')
const s = h.createServer()

s.on('session', (session) => {
  session.on('goaway', (e, l, o) => {
    console.log(`${e}, ${l}, ${o}`)
    s.close()
  })
})

s.listen(12000, () => {
  const c = h.connect('http://localhost:12000')
  const q = c.request()
  q.on('error', (e) => {
    console.log(e)
    s.close()
  })
  setTimeout(() => {
    c.close()
  }, +process.argv[2])
})

#node 2105.js 1
Error [ERR_HTTP2_GOAWAY_SESSION]: New streams cannot be created after receiving a GOAWAY
    at ClientHttp2Stream.requestOnConnect (internal/http2/core.js:642:17)
    at internal/http2/core.js:1609:53
    at Array.forEach (<anonymous>)
    at ClientHttp2Session.<anonymous> (internal/http2/core.js:1609:38)
    at Object.onceWrapper (events.js:307:26)
    at ClientHttp2Session.emit (events.js:222:7)
    at emit (internal/http2/core.js:276:8)
    at processTicksAndRejections (internal/process/task_queues.js:84:21) {
  code: 'ERR_HTTP2_GOAWAY_SESSION'
}

#node 2105.js 1
Error [ERR_HTTP2_GOAWAY_SESSION]: New streams cannot be created after receiving a GOAWAY
    at ClientHttp2Stream.requestOnConnect (internal/http2/core.js:642:17)
    at internal/http2/core.js:1609:53
    at Array.forEach (<anonymous>)
    at ClientHttp2Session.<anonymous> (internal/http2/core.js:1609:38)
    at Object.onceWrapper (events.js:307:26)
    at ClientHttp2Session.emit (events.js:222:7)
    at emit (internal/http2/core.js:276:8)
    at processTicksAndRejections (internal/process/task_queues.js:84:21) {
  code: 'ERR_HTTP2_GOAWAY_SESSION'
}
0, 0, undefined

#node 2105.js 100
0, 0, undefined
^C
#

[JustinBeckwith]

Completely unrelated to gRPC, I just ran into this as well. I am handling the goaway event on my ClientHttp2Session, but it's never called:

const session = http2.connect(`https://${host}`);
    session
      .on('error', e => {
        console.error(`*ERROR*: ${e}`);
        delete sessions[host];
      })
      .on('goaway', (errorCode, lastStreamId) => {
        console.error(`*GOAWAY*: ${errorCode} : ${lastStreamId}`);
        delete sessions[host];
      });

But when making a request by running:

const req = session.request(headers);

I get the error:

Error [ERR_HTTP2_GOAWAY_SESSION]: New streams cannot be created after receiving a GOAWAY
      at ClientHttp2Session.request (internal/http2/core.js:1621:13)
      at Object.request (build/src/http2.js:84:32)
          -> /Users/beckwith/Code/nodejs-googleapis-common/src/http2.ts:112:30
      at createAPIRequestAsync (build/src/apirequest.js:270:23)
          -> /Users/beckwith/Code/nodejs-googleapis-common/src/apirequest.ts:325:17
      at processTicksAndRejections (internal/process/task_queues.js:97:5)
      at async Context.<anonymous> (build/system-test/test.http2.js:119:9)
          -> /Users/beckwith/Code/nodejs-googleapis-common/system-test/test.http2.ts:128:5

I would expect to get a goaway event first 🤷

[RafaelGSS]

Sorry to answer an old issue. I just came to it because @murgatroid99 mentioned it in nodejs/TSC#1168.

is this indication of any racy behavior?

No, in fact, this is expected. When a c.request() is called, it doesn't mean that a request is sent at that moment.

From the source code:

// When a ClientHttp2Session is first created, the socket may not yet be
// connected. If request() is called during this time, the actual request
// will be deferred until the socket is ready to go.

The example provided by @gireeshpunathil is basically sending a request() when the server is already closed. That explains the GOWAY error.

I'll create a PR to improve the clienthttp2session.request explanation. Meanwhile, would be great to have a reproducible code in case if you think it is a bug.