Add API to refresh authc headers and retry ES request when 401 is enc…

…ountered (elastic#120677)

* initial POC

* remove test code

* update the header holding logic

* add new API to plugin context

* introduce the IAuthHeadersStorage interface

* fix some types, mocks and tests

* export types from server entrypoint

* also export error type

* more doc

* update generated doc

* Fix ES service tests

* add tests for createInternalErrorHandler

* fix type in cli_setup

* generated doc

* add tests for configureClient

* add unit tests for custom transport class

* fix handler propagation to initial clients

* lint

* address review comments

docs/development/core/server/kibana-plugin-core-server.elasticsearchservicesetup.md

@@ -16,4 +16,5 @@ export interface ElasticsearchServiceSetup
 |  Property | Type | Description |
 |  --- | --- | --- |
 |  [legacy](./kibana-plugin-core-server.elasticsearchservicesetup.legacy.md) | { readonly config$: Observable<ElasticsearchConfig>; } |  |
+|  [setUnauthorizedErrorHandler](./kibana-plugin-core-server.elasticsearchservicesetup.setunauthorizederrorhandler.md) | (handler: UnauthorizedErrorHandler) =&gt; void | Register a handler that will be called when unauthorized (401) errors are returned from any API call to elasticsearch performed on behalf of a user via a [scoped cluster client](./kibana-plugin-core-server.iscopedclusterclient.md)<!-- -->. |
 

docs/development/core/server/kibana-plugin-core-server.elasticsearchservicesetup.setunauthorizederrorhandler.md

@@ -0,0 +1,35 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-core-server](./kibana-plugin-core-server.md) &gt; [ElasticsearchServiceSetup](./kibana-plugin-core-server.elasticsearchservicesetup.md) &gt; [setUnauthorizedErrorHandler](./kibana-plugin-core-server.elasticsearchservicesetup.setunauthorizederrorhandler.md)
+
+## ElasticsearchServiceSetup.setUnauthorizedErrorHandler property
+
+Register a handler that will be called when unauthorized (401) errors are returned from any API call to elasticsearch performed on behalf of a user via a [scoped cluster client](./kibana-plugin-core-server.iscopedclusterclient.md)<!-- -->.
+
+<b>Signature:</b>
+
+```typescript
+setUnauthorizedErrorHandler: (handler: UnauthorizedErrorHandler) => void;
+```
+
+## Remarks
+
+The handler will only be invoked for scoped client bound to real [request](./kibana-plugin-core-server.kibanarequest.md) instances.
+
+## Example
+
+
+```ts
+const handler: UnauthorizedErrorHandler = ({ request, error }, toolkit) => {
+  const reauthenticationResult = await authenticator.reauthenticate(request, error);
+  if (reauthenticationResult.succeeded()) {
+    return toolkit.retry({
+      authHeaders: reauthenticationResult.authHeaders,
+    });
+  }
+  return toolkit.notHandled();
+}
+
+coreSetup.elasticsearch.setUnauthorizedErrorHandler(handler);
+```
+

docs/development/core/server/kibana-plugin-core-server.md

@@ -230,6 +230,11 @@ The plugin integrates with the core system via lifecycle events: `setup`<!-- -->
 |  [UiSettingsParams](./kibana-plugin-core-server.uisettingsparams.md) | UiSettings parameters defined by the plugins. |
 |  [UiSettingsServiceSetup](./kibana-plugin-core-server.uisettingsservicesetup.md) |  |
 |  [UiSettingsServiceStart](./kibana-plugin-core-server.uisettingsservicestart.md) |  |
+|  [UnauthorizedErrorHandlerNotHandledResult](./kibana-plugin-core-server.unauthorizederrorhandlernothandledresult.md) |  |
+|  [UnauthorizedErrorHandlerOptions](./kibana-plugin-core-server.unauthorizederrorhandleroptions.md) |  |
+|  [UnauthorizedErrorHandlerResultRetryParams](./kibana-plugin-core-server.unauthorizederrorhandlerresultretryparams.md) |  |
+|  [UnauthorizedErrorHandlerRetryResult](./kibana-plugin-core-server.unauthorizederrorhandlerretryresult.md) |  |
+|  [UnauthorizedErrorHandlerToolkit](./kibana-plugin-core-server.unauthorizederrorhandlertoolkit.md) | Toolkit passed to a [UnauthorizedErrorHandler](./kibana-plugin-core-server.unauthorizederrorhandler.md) used to generate responses from the handler |
 |  [UserProvidedValues](./kibana-plugin-core-server.userprovidedvalues.md) | Describes the values explicitly set by user. |
 
 ## Variables
@@ -329,4 +334,7 @@ The plugin integrates with the core system via lifecycle events: `setup`<!-- -->
 |  [SharedGlobalConfig](./kibana-plugin-core-server.sharedglobalconfig.md) |  |
 |  [StartServicesAccessor](./kibana-plugin-core-server.startservicesaccessor.md) | Allows plugins to get access to APIs available in start inside async handlers. Promise will not resolve until Core and plugin dependencies have completed <code>start</code>. This should only be used inside handlers registered during <code>setup</code> that will only be executed after <code>start</code> lifecycle. |
 |  [UiSettingsType](./kibana-plugin-core-server.uisettingstype.md) | UI element type to represent the settings. |
+|  [UnauthorizedError](./kibana-plugin-core-server.unauthorizederror.md) |  |
+|  [UnauthorizedErrorHandler](./kibana-plugin-core-server.unauthorizederrorhandler.md) | A handler used to handle unauthorized error returned by elasticsearch |
+|  [UnauthorizedErrorHandlerResult](./kibana-plugin-core-server.unauthorizederrorhandlerresult.md) |  |
 

docs/development/core/server/kibana-plugin-core-server.unauthorizederror.md

@@ -0,0 +1,14 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-core-server](./kibana-plugin-core-server.md) &gt; [UnauthorizedError](./kibana-plugin-core-server.unauthorizederror.md)
+
+## UnauthorizedError type
+
+
+<b>Signature:</b>
+
+```typescript
+export declare type UnauthorizedError = errors.ResponseError & {
+    statusCode: 401;
+};
+```

docs/development/core/server/kibana-plugin-core-server.unauthorizederrorhandler.md

@@ -0,0 +1,13 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-core-server](./kibana-plugin-core-server.md) &gt; [UnauthorizedErrorHandler](./kibana-plugin-core-server.unauthorizederrorhandler.md)
+
+## UnauthorizedErrorHandler type
+
+A handler used to handle unauthorized error returned by elasticsearch
+
+<b>Signature:</b>
+
+```typescript
+export declare type UnauthorizedErrorHandler = (options: UnauthorizedErrorHandlerOptions, toolkit: UnauthorizedErrorHandlerToolkit) => MaybePromise<UnauthorizedErrorHandlerResult>;
+```

docs/development/core/server/kibana-plugin-core-server.unauthorizederrorhandlernothandledresult.md

@@ -0,0 +1,19 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-core-server](./kibana-plugin-core-server.md) &gt; [UnauthorizedErrorHandlerNotHandledResult](./kibana-plugin-core-server.unauthorizederrorhandlernothandledresult.md)
+
+## UnauthorizedErrorHandlerNotHandledResult interface
+
+
+<b>Signature:</b>
+
+```typescript
+export interface UnauthorizedErrorHandlerNotHandledResult 
+```
+
+## Properties
+
+|  Property | Type | Description |
+|  --- | --- | --- |
+|  [type](./kibana-plugin-core-server.unauthorizederrorhandlernothandledresult.type.md) | 'notHandled' |  |
+

docs/development/core/server/kibana-plugin-core-server.unauthorizederrorhandlernothandledresult.type.md

@@ -0,0 +1,11 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-core-server](./kibana-plugin-core-server.md) &gt; [UnauthorizedErrorHandlerNotHandledResult](./kibana-plugin-core-server.unauthorizederrorhandlernothandledresult.md) &gt; [type](./kibana-plugin-core-server.unauthorizederrorhandlernothandledresult.type.md)
+
+## UnauthorizedErrorHandlerNotHandledResult.type property
+
+<b>Signature:</b>
+
+```typescript
+type: 'notHandled';
+```

docs/development/core/server/kibana-plugin-core-server.unauthorizederrorhandleroptions.error.md

@@ -0,0 +1,11 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-core-server](./kibana-plugin-core-server.md) &gt; [UnauthorizedErrorHandlerOptions](./kibana-plugin-core-server.unauthorizederrorhandleroptions.md) &gt; [error](./kibana-plugin-core-server.unauthorizederrorhandleroptions.error.md)
+
+## UnauthorizedErrorHandlerOptions.error property
+
+<b>Signature:</b>
+
+```typescript
+error: UnauthorizedError;
+```

docs/development/core/server/kibana-plugin-core-server.unauthorizederrorhandleroptions.md

@@ -0,0 +1,20 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-core-server](./kibana-plugin-core-server.md) &gt; [UnauthorizedErrorHandlerOptions](./kibana-plugin-core-server.unauthorizederrorhandleroptions.md)
+
+## UnauthorizedErrorHandlerOptions interface
+
+
+<b>Signature:</b>
+
+```typescript
+export interface UnauthorizedErrorHandlerOptions 
+```
+
+## Properties
+
+|  Property | Type | Description |
+|  --- | --- | --- |
+|  [error](./kibana-plugin-core-server.unauthorizederrorhandleroptions.error.md) | UnauthorizedError |  |
+|  [request](./kibana-plugin-core-server.unauthorizederrorhandleroptions.request.md) | KibanaRequest |  |
+

docs/development/core/server/kibana-plugin-core-server.unauthorizederrorhandleroptions.request.md

@@ -0,0 +1,11 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-core-server](./kibana-plugin-core-server.md) &gt; [UnauthorizedErrorHandlerOptions](./kibana-plugin-core-server.unauthorizederrorhandleroptions.md) &gt; [request](./kibana-plugin-core-server.unauthorizederrorhandleroptions.request.md)
+
+## UnauthorizedErrorHandlerOptions.request property
+
+<b>Signature:</b>
+
+```typescript
+request: KibanaRequest;
+```

docs/development/core/server/kibana-plugin-core-server.unauthorizederrorhandlerresult.md

@@ -0,0 +1,12 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-core-server](./kibana-plugin-core-server.md) &gt; [UnauthorizedErrorHandlerResult](./kibana-plugin-core-server.unauthorizederrorhandlerresult.md)
+
+## UnauthorizedErrorHandlerResult type
+
+
+<b>Signature:</b>
+
+```typescript
+export declare type UnauthorizedErrorHandlerResult = UnauthorizedErrorHandlerRetryResult | UnauthorizedErrorHandlerNotHandledResult;
+```

docs/development/core/server/kibana-plugin-core-server.unauthorizederrorhandlerresultretryparams.authheaders.md

@@ -0,0 +1,11 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-core-server](./kibana-plugin-core-server.md) &gt; [UnauthorizedErrorHandlerResultRetryParams](./kibana-plugin-core-server.unauthorizederrorhandlerresultretryparams.md) &gt; [authHeaders](./kibana-plugin-core-server.unauthorizederrorhandlerresultretryparams.authheaders.md)
+
+## UnauthorizedErrorHandlerResultRetryParams.authHeaders property
+
+<b>Signature:</b>
+
+```typescript
+authHeaders: AuthHeaders;
+```

docs/development/core/server/kibana-plugin-core-server.unauthorizederrorhandlerresultretryparams.md

@@ -0,0 +1,19 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-core-server](./kibana-plugin-core-server.md) &gt; [UnauthorizedErrorHandlerResultRetryParams](./kibana-plugin-core-server.unauthorizederrorhandlerresultretryparams.md)
+
+## UnauthorizedErrorHandlerResultRetryParams interface
+
+
+<b>Signature:</b>
+
+```typescript
+export interface UnauthorizedErrorHandlerResultRetryParams 
+```
+
+## Properties
+
+|  Property | Type | Description |
+|  --- | --- | --- |
+|  [authHeaders](./kibana-plugin-core-server.unauthorizederrorhandlerresultretryparams.authheaders.md) | AuthHeaders |  |
+

docs/development/core/server/kibana-plugin-core-server.unauthorizederrorhandlerretryresult.md

@@ -0,0 +1,20 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-core-server](./kibana-plugin-core-server.md) &gt; [UnauthorizedErrorHandlerRetryResult](./kibana-plugin-core-server.unauthorizederrorhandlerretryresult.md)
+
+## UnauthorizedErrorHandlerRetryResult interface
+
+
+<b>Signature:</b>
+
+```typescript
+export interface UnauthorizedErrorHandlerRetryResult extends UnauthorizedErrorHandlerResultRetryParams 
+```
+<b>Extends:</b> UnauthorizedErrorHandlerResultRetryParams
+
+## Properties
+
+|  Property | Type | Description |
+|  --- | --- | --- |
+|  [type](./kibana-plugin-core-server.unauthorizederrorhandlerretryresult.type.md) | 'retry' |  |
+

docs/development/core/server/kibana-plugin-core-server.unauthorizederrorhandlerretryresult.type.md

@@ -0,0 +1,11 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-core-server](./kibana-plugin-core-server.md) &gt; [UnauthorizedErrorHandlerRetryResult](./kibana-plugin-core-server.unauthorizederrorhandlerretryresult.md) &gt; [type](./kibana-plugin-core-server.unauthorizederrorhandlerretryresult.type.md)
+
+## UnauthorizedErrorHandlerRetryResult.type property
+
+<b>Signature:</b>
+
+```typescript
+type: 'retry';
+```

docs/development/core/server/kibana-plugin-core-server.unauthorizederrorhandlertoolkit.md

@@ -0,0 +1,21 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-core-server](./kibana-plugin-core-server.md) &gt; [UnauthorizedErrorHandlerToolkit](./kibana-plugin-core-server.unauthorizederrorhandlertoolkit.md)
+
+## UnauthorizedErrorHandlerToolkit interface
+
+Toolkit passed to a [UnauthorizedErrorHandler](./kibana-plugin-core-server.unauthorizederrorhandler.md) used to generate responses from the handler
+
+<b>Signature:</b>
+
+```typescript
+export interface UnauthorizedErrorHandlerToolkit 
+```
+
+## Properties
+
+|  Property | Type | Description |
+|  --- | --- | --- |
+|  [notHandled](./kibana-plugin-core-server.unauthorizederrorhandlertoolkit.nothandled.md) | () =&gt; UnauthorizedErrorHandlerNotHandledResult | The handler cannot handle the error, or was not able to authenticate. |
+|  [retry](./kibana-plugin-core-server.unauthorizederrorhandlertoolkit.retry.md) | (params: UnauthorizedErrorHandlerResultRetryParams) =&gt; UnauthorizedErrorHandlerRetryResult | The handler was able to authenticate. Will retry the failed request with new auth headers |
+

docs/development/core/server/kibana-plugin-core-server.unauthorizederrorhandlertoolkit.nothandled.md

@@ -0,0 +1,13 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-core-server](./kibana-plugin-core-server.md) &gt; [UnauthorizedErrorHandlerToolkit](./kibana-plugin-core-server.unauthorizederrorhandlertoolkit.md) &gt; [notHandled](./kibana-plugin-core-server.unauthorizederrorhandlertoolkit.nothandled.md)
+
+## UnauthorizedErrorHandlerToolkit.notHandled property
+
+The handler cannot handle the error, or was not able to authenticate.
+
+<b>Signature:</b>
+
+```typescript
+notHandled: () => UnauthorizedErrorHandlerNotHandledResult;
+```

docs/development/core/server/kibana-plugin-core-server.unauthorizederrorhandlertoolkit.retry.md

@@ -0,0 +1,13 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-core-server](./kibana-plugin-core-server.md) &gt; [UnauthorizedErrorHandlerToolkit](./kibana-plugin-core-server.unauthorizederrorhandlertoolkit.md) &gt; [retry](./kibana-plugin-core-server.unauthorizederrorhandlertoolkit.retry.md)
+
+## UnauthorizedErrorHandlerToolkit.retry property
+
+The handler was able to authenticate. Will retry the failed request with new auth headers
+
+<b>Signature:</b>
+
+```typescript
+retry: (params: UnauthorizedErrorHandlerResultRetryParams) => UnauthorizedErrorHandlerRetryResult;
+```

src/cli_setup/utils.ts

@@ -37,17 +37,17 @@ export const elasticsearch = new ElasticsearchService(logger, kibanaPackageJson.
   elasticsearch: {
     createClient: (type, config) => {
       const defaults = configSchema.validate({});
-      return new ClusterClient(
-        merge(
+      return new ClusterClient({
+        config: merge(
           defaults,
           {
             hosts: Array.isArray(defaults.hosts) ? defaults.hosts : [defaults.hosts],
           },
           config
         ),
         logger,
-        type
-      );
+        type,
+      });
     },
   },
 });

src/core/server/elasticsearch/client/cluster_client.test.mocks.ts

@@ -10,3 +10,13 @@ export const configureClientMock = jest.fn();
 jest.doMock('./configure_client', () => ({
   configureClient: configureClientMock,
 }));
+
+export const createTransportMock = jest.fn();
+jest.doMock('./create_transport', () => ({
+  createTransport: createTransportMock,
+}));
+
+export const createInternalErrorHandlerMock = jest.fn();
+jest.doMock('./retry_unauthorized', () => ({
+  createInternalErrorHandler: createInternalErrorHandlerMock,
+}));