[MD] Add data source signing support (#2510) (#2547)

* Add data source signing support
* Optimize error handling and logging
* Update wording on error message and readme

Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
(cherry picked from commit e3bbdef)

Co-authored-by: Louis Chu <lingzhichu.clz@gmail.com>

CHANGELOG.md

@@ -11,6 +11,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 ### 📈 Features/Enhancements
 
 * [MD] Support legacy client for data source ([#2204](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2204))
+* [MD] Add data source signing support ([#2510](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2510)) 
 * [Plugin Helpers] Facilitate version changes ([#2398](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2398))
 * [MD] Display error toast for create index pattern with data source ([#2506](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2506))
 * [Multi DataSource] UX enhancement on index pattern management stack ([#2505](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2505))

src/plugins/data_source/README.md

@@ -5,50 +5,61 @@ An OpenSearch Dashboards plugin
 This plugin introduces support for multiple data sources into OpenSearch Dashboards and provides related functions to connect to OpenSearch data sources.
 
 ## Configuration
+
 Update the following configuration in the `opensearch_dashboards.yml` file to apply changes. Refer to the schema [here](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/main/src/plugins/data_source/config.ts) for supported configurations.
 
 1. The dataSource plugin is disabled by default; to enable it:
-`data_source.enabled: true`
+   `data_source.enabled: true`
 
 2. The audit trail is enabled by default for logging the access to data source; to disable it:
-`data_source.audit.enabled: false`
+   `data_source.audit.enabled: false`
+
+- Current auditor configuration:
 
- - Current auditor configuration:
 ```
 data_source.audit.appender.kind: 'file'
 data_source.audit.appender.layout.kind: 'pattern'
 data_source.audit.appender.path: '/tmp/opensearch-dashboards-data-source-audit.log'
 ```
 
 3. The default encryption-related configuration parameters are:
+
 ```
 data_source.encryption.wrappingKeyName: 'changeme'
 data_source.encryption.wrappingKeyNamespace: 'changeme'
 data_source.encryption.wrappingKey: [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
 ```
+
 Note that if any of the encryption keyring configuration values change (wrappingKeyName/wrappingKeyNamespace/wrappingKey), none of the previously-encrypted credentials can be decrypted; therefore, credentials of previously created data sources must be updated to continue use.
 
 **What are the best practices for generating a secure wrapping key?**  
 WrappingKey is an array of 32 random numbers. Read [more](https://en.wikipedia.org/wiki/Cryptographically_secure_pseudorandom_number_generator) about best practices for generating a secure wrapping key.
 
 ## Public
+
 The public plugin is used to enable and disable the features related to multi data source available in other plugins. e.g. data_source_management, index_pattern_management
 
 - Add as a required dependency for whole plugin on/off switch
 - Add as opitional dependency for partial flow changes control
 
 ## Server
+
 The provided data source client is integrated with default search strategy in data plugin. When data source id presented in IOpenSearchSearchRequest, data source client will be used.
 
 ### Data Source Service
-The data source service will provide a data source client given a data source id and optional client configurations. 
+
+The data source service will provide a data source client given a data source id and optional client configurations.
 
 Currently supported client config is:
+
 - `data_source.clientPool.size`
 
 Data source service uses LRU cache to cache the root client to improve client pool usage.
+
 #### Example usage:
+
 In the RequestHandler, get an instance of the client using:
+
 ```ts
 client: OpenSearchClient = await context.dataSource.opensearch.getClient(dataSourceId);
 
@@ -57,17 +68,21 @@ apiCaller: LegacyAPICaller = context.dataSource.opensearch.legacy.getClient(data
 ```
 
 ### Data Source Client Wrapper
+
 The data source saved object client wrapper overrides the write related action for data source object in order to perform validation and encryption actions of the authentication information inside data source.
 
-### Cryptography Client
-The research for choosing a suitable stack can be found in: [#1756](https://github.com/opensearch-project/OpenSearch-Dashboards/issues/1756)
-#### Example usage:
-```ts
-//Encrypt
-const encryptedPassword = await this.cryptographyClient.encryptAndEncode(password);
-//Decrypt
-const decodedPassword = await this.cryptographyClient.decodeAndDecrypt(password);
-```
+### Cryptography service
+
+The cryptography service encrypts and decrypts data source credentials (support no_auth and username_password credential types). Highlight the best security practices listed below:
+
+a. Envelope encryption - provides strong protection on data keys. Read more details [here](https://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/concepts.html#envelope-encryption)
+
+b. Key derivation with HMAC - KDF with SHA-384 protects against accidental reuse of a data encryption keys and reduces the risk of overusing data keys.
+
+c. Signature algorithm - ECDSA with P-384 and SHA-384. Under multiple data source case, data source documents stored on OpenSearch can be modified / replaced by attacker. With ECDSA signature, ciphertext decryption will fail if it’s getting pullted. No one will be able to create another signature that verifies with the public key because the private key has been dropped.
+
+Please check https://github.com/opensearch-project/OpenSearch-Dashboards/issues/1756 for more details.
+
 ---
 
 ## Development

src/plugins/data_source/config.ts

@@ -8,7 +8,7 @@ import { fileAppenderSchema } from './audit_config';
 
 const KEY_NAME_MIN_LENGTH: number = 1;
 const KEY_NAME_MAX_LENGTH: number = 100;
-// Wrapping key size shoule be 32 bytes, as used in envelope encryption algorithms.
+// Wrapping key size should be 32 bytes, as used in envelope encryption algorithms.
 const WRAPPING_KEY_SIZE: number = 32;
 
 export const configSchema = schema.object({

src/plugins/data_source/server/client/configure_client.test.ts

@@ -14,17 +14,18 @@ import { configureClient } from './configure_client';
 import { ClientOptions } from '@opensearch-project/opensearch';
 // eslint-disable-next-line @osd/eslint/no-restricted-paths
 import { opensearchClientMock } from '../../../../core/server/opensearch/client/mocks';
-import { CryptographyClient } from '../cryptography';
+import { cryptographyServiceSetupMock } from '../cryptography_service.mocks';
+import { CryptographyServiceSetup } from '../cryptography_service';
 import { DataSourceClientParams } from '../types';
 
 const DATA_SOURCE_ID = 'a54b76ec86771ee865a0f74a305dfff8';
-const cryptoClient = new CryptographyClient('test', 'test', new Array(32).fill(0));
 
 // TODO: improve UT
 describe('configureClient', () => {
   let logger: ReturnType<typeof loggingSystemMock.createLogger>;
   let config: DataSourcePluginConfigType;
   let savedObjectsMock: jest.Mocked<SavedObjectsClientContract>;
+  let cryptographyMock: jest.Mocked<CryptographyServiceSetup>;
   let clientPoolSetup: OpenSearchClientPoolSetup;
   let clientOptions: ClientOptions;
   let dataSourceAttr: DataSourceAttributes;
@@ -35,6 +36,8 @@ describe('configureClient', () => {
     dsClient = opensearchClientMock.createInternalClient();
     logger = loggingSystemMock.createLogger();
     savedObjectsMock = savedObjectsClientMock.create();
+    cryptographyMock = cryptographyServiceSetupMock.create();
+
     config = {
       enabled: true,
       clientPool: {
@@ -75,7 +78,7 @@ describe('configureClient', () => {
     dataSourceClientParams = {
       dataSourceId: DATA_SOURCE_ID,
       savedObjects: savedObjectsMock,
-      cryptographyClient: cryptoClient,
+      cryptography: cryptographyMock,
     };
 
     ClientMock.mockImplementation(() => dsClient);
@@ -109,14 +112,48 @@ describe('configureClient', () => {
     expect(client).toBe(dsClient.child.mock.results[0].value);
   });
 
-  test('configure client with auth.type == username_password, will first call decrypt()', async () => {
-    const spy = jest.spyOn(cryptoClient, 'decodeAndDecrypt').mockResolvedValue('password');
+  test('configure client with auth.type == username_password, will first call decodeAndDecrypt()', async () => {
+    const decodeAndDecryptSpy = jest.spyOn(cryptographyMock, 'decodeAndDecrypt').mockResolvedValue({
+      decryptedText: 'password',
+      encryptionContext: { endpoint: 'http://localhost' },
+    });
 
     const client = await configureClient(dataSourceClientParams, clientPoolSetup, config, logger);
 
     expect(ClientMock).toHaveBeenCalledTimes(1);
     expect(savedObjectsMock.get).toHaveBeenCalledTimes(1);
-    expect(spy).toHaveBeenCalledTimes(1);
+    expect(decodeAndDecryptSpy).toHaveBeenCalledTimes(1);
     expect(client).toBe(dsClient.child.mock.results[0].value);
   });
+
+  test('configure client with auth.type == username_password and password contaminated', async () => {
+    const decodeAndDecryptSpy = jest
+      .spyOn(cryptographyMock, 'decodeAndDecrypt')
+      .mockImplementation(() => {
+        throw new Error();
+      });
+
+    await expect(
+      configureClient(dataSourceClientParams, clientPoolSetup, config, logger)
+    ).rejects.toThrowError();
+
+    expect(ClientMock).toHaveBeenCalledTimes(1);
+    expect(savedObjectsMock.get).toHaveBeenCalledTimes(1);
+    expect(decodeAndDecryptSpy).toHaveBeenCalledTimes(1);
+  });
+
+  test('configure client with auth.type == username_password and endpoint contaminated', async () => {
+    const decodeAndDecryptSpy = jest.spyOn(cryptographyMock, 'decodeAndDecrypt').mockResolvedValue({
+      decryptedText: 'password',
+      encryptionContext: { endpoint: 'http://dummy.com' },
+    });
+
+    await expect(
+      configureClient(dataSourceClientParams, clientPoolSetup, config, logger)
+    ).rejects.toThrowError();
+
+    expect(ClientMock).toHaveBeenCalledTimes(1);
+    expect(savedObjectsMock.get).toHaveBeenCalledTimes(1);
+    expect(decodeAndDecryptSpy).toHaveBeenCalledTimes(1);
+  });
 });

src/plugins/data_source/server/client/configure_client.ts

@@ -12,14 +12,14 @@ import {
   UsernamePasswordTypedContent,
 } from '../../common/data_sources';
 import { DataSourcePluginConfigType } from '../../config';
-import { CryptographyClient } from '../cryptography';
+import { CryptographyServiceSetup } from '../cryptography_service';
 import { DataSourceConfigError } from '../lib/error';
 import { DataSourceClientParams } from '../types';
 import { parseClientOptions } from './client_config';
 import { OpenSearchClientPoolSetup } from './client_pool';
 
 export const configureClient = async (
-  { dataSourceId, savedObjects, cryptographyClient }: DataSourceClientParams,
+  { dataSourceId, savedObjects, cryptography }: DataSourceClientParams,
   openSearchClientPoolSetup: OpenSearchClientPoolSetup,
   config: DataSourcePluginConfigType,
   logger: Logger
@@ -28,12 +28,12 @@ export const configureClient = async (
     const dataSource = await getDataSource(dataSourceId, savedObjects);
     const rootClient = getRootClient(dataSource.attributes, config, openSearchClientPoolSetup);
 
-    return await getQueryClient(rootClient, dataSource, cryptographyClient);
+    return await getQueryClient(rootClient, dataSource, cryptography);
   } catch (error: any) {
-    logger.error(`Fail to get data source client for dataSourceId: [${dataSourceId}]`);
+    logger.error(`Failed to get data source client for dataSourceId: [${dataSourceId}]`);
     logger.error(error);
     // Re-throw as DataSourceConfigError
-    throw new DataSourceConfigError('Fail to get data source client: ', error);
+    throw new DataSourceConfigError('Failed to get data source client: ', error);
   }
 };
 
@@ -50,13 +50,28 @@ export const getDataSource = async (
 
 export const getCredential = async (
   dataSource: SavedObject<DataSourceAttributes>,
-  cryptographyClient: CryptographyClient
+  cryptography: CryptographyServiceSetup
 ): Promise<UsernamePasswordTypedContent> => {
+  const { endpoint } = dataSource.attributes!;
+
   const { username, password } = dataSource.attributes.auth.credentials!;
-  const decodedPassword = await cryptographyClient.decodeAndDecrypt(password);
+
+  const { decryptedText, encryptionContext } = await cryptography
+    .decodeAndDecrypt(password)
+    .catch((err: any) => {
+      // Re-throw as DataSourceConfigError
+      throw new DataSourceConfigError('Unable to decrypt "auth.credentials.password".', err);
+    });
+
+  if (encryptionContext!.endpoint !== endpoint) {
+    throw new Error(
+      'Data source "endpoint" contaminated. Please delete and create another data source.'
+    );
+  }
+
   const credential = {
     username,
-    password: decodedPassword,
+    password: decryptedText,
   };
 
   return credential;
@@ -67,13 +82,13 @@ export const getCredential = async (
  *
  * @param rootClient root client for the connection with given data source endpoint.
  * @param dataSource data source saved object
- * @param cryptographyClient cryptography client for password encryption / decryption
+ * @param cryptography cryptography service for password encryption / decryption
  * @returns child client.
  */
 const getQueryClient = async (
   rootClient: Client,
   dataSource: SavedObject<DataSourceAttributes>,
-  cryptographyClient: CryptographyClient
+  cryptography: CryptographyServiceSetup
 ): Promise<Client> => {
   const authType = dataSource.attributes.auth.type;
 
@@ -82,7 +97,7 @@ const getQueryClient = async (
       return rootClient.child();
 
     case AuthType.UsernamePasswordType:
-      const credential = await getCredential(dataSource, cryptographyClient);
+      const credential = await getCredential(dataSource, cryptography);
       return getBasicAuthClient(rootClient, credential);
 
     default: