Refactor client and/or transport to support middleware

[JoshMock]

This is a meta issue to explore support for adding custom middleware to the client and/or transport that can hook into various steps in the request/response lifecycle, so that requests, responses, metadata, error handling, etc. can be modified to be more flexible and meet a greater set of needs than can be easily anticipated by a generalized API client.

Background

Recently added functionality to the client and transport has increased the complexity and reduced readability of their core classes, like third-party integration hooks or options that conditionally alter connection handling, requests, responses or metadata.

Examples:

• OpenTelemetry support required:
  • writing a wrapper around the entire Transport.request function
  • injecting OTel conditions in several places (e.g. here and here)
• Conditionally redacting sensitive data from exception metadata required modifications to almost every exception class
• Putting the client into serverless mode adds conditional overrides to several defaults in client.ts

Furthermore, a incoming request from the Kibana core team is asking if it's possible to write hooks into each API request to conditionally alter requests that contain certain named parameters before they are sent to the server.

Other integrations

There are other integrations and observability tools built into the client that could easily be refactored into middleware.

• The client currently operates as an EventEmitter, supporting several events during the request lifecycle. While these are helpful for reacting in certain ways (logging, observability) they are not designed to alter the contents or flow of data to or from Elasticsearch.
• Support for unique request IDs and adding context metadata to requests
• Support for X-Opaque-Id
• Helpers exist that transform binary response data into Apache Arrow objects

Prior art

• MongoDB's JavaScript client includes the concept of driver extensions to support additive features like compression, encryption, and custom auth schemes
• Express, the Node.js HTTP server framework, exposes middleware support at multiple layers
• Redux, the functional application state management tool, heavily encourages using middleware to support use cases like logging, modifying state, async state management, scheduled actions and more

Architectural thoughts

In many Node.js and browser-based JavaScript libraries, attaching middleware is done by providing an array of functions that should be called, in that order, at various hook points. These functions are expected to take parameters from the client and return the same parameters, either reacting to them out of band synchronously or asynchronously (logging, observability), or altering those parameters before returning them:

const client = new Client({
 middleware: {
  onBeforeRequest: [compress, addRequestId, changeUserAgent],
  onRequest: [logRequest, openTelemetryWrapper],
  onResponse: [...],
  onError: [redactSecrets],
})

function compress(request) {
  request.body = gzip(request.body)
  return request
}

function redactSecrets(error) {
  error.meta = redact(error.meta)
  return error
}

That is just an example of an approach that could work for this client, not a guarantee of what might be exposed.

More thoughts to come.

Interested parties

• @rudolf
• @afharo
• @mikecote
• @swallez