Producer failover and promise reject when a broker is disconnected

[byrne-greg]

Currently running an app as HTTP server on node that emits events on requests. On startup of the app, we connect the producer; and when a request comes in, we emit a message through producer.send.

When one of the kafka brokers are turned off/on during a maintenance window, the application will start reporting issues when emitting events and then throw an unhandled error uncaught by any error handler. Can't see where the error bubbles up from kafkajs and its leading the application crashing.

/Users/myapp/node_modules/kafkajs/src/network/requestQueue/socketRequest.js:108
    this.rejected(new KafkaJSRequestTimeoutError(`Request ${requestInfo} timed out`, eventData))
                  ^
KafkaJSNonRetriableError
  Caused by: KafkaJSRequestTimeoutError: Request Produce(key: 0, version: 7) timed out
    at SocketRequest.timeoutRequest (/Users/myapp/node_modules/kafkajs/src/network/requestQueue/socketRequest.js:108:19)
    ... 4 lines matching cause stack trace ...
    at processTimers (node:internal/timers:512:7) {
  name: 'KafkaJSNumberOfRetriesExceeded',
  retriable: false,
  helpUrl: undefined,
  retryCount: 3,
  retryTime: 596,
  [cause]: KafkaJSRequestTimeoutError: Request Produce(key: 0, version: 7) timed out
      at SocketRequest.timeoutRequest (/Users/myapp/node_modules/kafkajs/src/network/requestQueue/socketRequest.js:108:19)
      at /Users/myapp/node_modules/kafkajs/src/network/requestQueue/index.js:100:21
      at Map.forEach (<anonymous>)
      at Timeout._onTimeout (/Users/myapp/node_modules/kafkajs/src/network/requestQueue/index.js:94:23)
      at listOnTimeout (node:internal/timers:569:17)
      at processTimers (node:internal/timers:512:7) {
    retriable: true,
    helpUrl: undefined,
    broker: 'broker-3.com:9097',
    correlationId: 7,
    createdAt: 1716207461754,
    sentAt: 1716207461755,
    pendingDuration: 1,
    [cause]: undefined
  }
}

What is the correct way to handle the producer in these cases? Ideally i'd prefer the HTTP server stay alive even if we lose some events and some hook that I could restart the producer in the interim. I didn't expect the error to surface from a place that doesn't look like i can wrap with an error handler