[NodeJS] Dashboards is not compatible with nodejs client

[ananzh]

Tasks

• hookup @opensearch-project/opensearch to dashboards
  PR: [nodejs client] hookup js client with dashboards #1181
• fix issues in scr/core/sever/saved_objects
  Issue: [nodejs client] hookup issues in src/core/server/saved_objects #1216
  PR: [nodejs client] fix hookup issues in src/core/server/saved_objects #1223
• fix issues in src/core/server/core_usage_data
  Issue: [nodejs client] hookup issues in src/core/server/core_usage_data #1224
  PR: [nodejs client] fix hookup issues in src/core/server/core_usage_data_service.ts #1225
• fix issues in src/core/server/opensearch
  Issue: [nodejs client] hookup issues in src/core/server/opensearch #1193
  PR: [nodejs client] fix hookup issues in src/core/server/opensearch #1194
• yarn osd bootstrap works
• Unit tests all pass
  PR: [nodejs client] modify tsconfig to route types to new.d.ts #1255
• Integration tests all pass
  Issue: [nodejs client] Integration test passing #1245
• Functional tests all pass
  Issue: [nodejs client] Functional tests passing #1246

Describe the bug

We forked nodejs client (repo here) from v7.14 and delete all the non-oss api between v7.10 to v7.14. nodejs client v7.14 is not compatible with current Dashboards due to type change.

For example, below is the indices.get defined in nodejs v7.14:

get<TContext = unknown>(params: T.IndicesGetRequest, options?: TransportRequestOptions): TransportRequestPromise<ApiResponse<T.IndicesGetResponse, TContext>>

which is different from its definition in nodejs v7.10:

get<TResponse = Record<string, any>, TContext = Context>(params?: RequestParams.IndicesGet, options?: TransportRequestOptions): TransportRequestPromise<ApiResponse<TResponse, TContext>>

If we refers to @opensearch-project/opensearch and change all the client from KibanaClient to OpenSearchDashboardsClient, we could see type undefined errors from bootstrap. Below is one example:

ERROR [bootstrap] failed:
ERROR Error: Command failed with exit code 1: /usr/share/yarn/bin/yarn.js run osd:bootstrap
      error Command failed with exit code 1.
      $ node scripts/build_ts_refs && node scripts/register_git_hook
      ERROR Error: Command failed with exit code 1: /home/anan/work/OpenSearch-Dashboards/node_modules/typescript/bin/tsc -b tsconfig.refs.json --pretty
            src/core/server/saved_objects/migrations/core/opensearch_index.ts:71:33 - error TS2345: Argument of type '{ exists: true; indexName: string; aliases?: Record<string, IndicesAlias> | undefined; mappings?: MappingTypeMapping | undefined; settings: IndicesIndexSettings | IndicesIndexStatePrefixedSettings; }' is not assignable to parameter of type 'FullIndexInfo'.
              Types of property 'aliases' are incompatible.
                Type 'Record<string, IndicesAlias> | undefined' is not assignable to type '{ [name: string]: object; }'.
                  Type 'undefined' is not assignable to type '{ [name: string]: object; }'.

            71   return assertIsSupportedIndex({ ...indexInfo, exists: true, indexName });
                                               ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To Reproduce
1)yarn add @opensearch-project/opensearch
2) search @elastic/elasticsearch in non package.json files and replace it with @opensearch-project/opensearch
3) go to src/core/server/saved_objects/migrations/core/opensearch_index.ts change KibanaClient to OpenSearchDashboardsClient

Impact to Plugins

The major change brought by new opensearch-js client is that the api types are looser ( here ) than the original ( here ) . For example, comparing how bulk api is previously defined as

bulk<TResponse = Record<string, any>, TRequestBody extends RequestNDBody = Record<string, any>[], TContext = Context>(params?: RequestParams.Bulk<TRequestBody>, options?: TransportRequestOptions): TransportRequestPromise<ApiResponse<TResponse, TContext>> 

While in the new.d.ts, the bulk api is now changed to:

bulk<TSource = unknown, TContext = unknown>(params: T.BulkRequest<TSource>, options?: TransportRequestOptions): TransportRequestPromise<ApiResponse<T.BulkResponse, TContext>>

Since it is a looser typed client, we don’t see much breaking changes in the dashboards code base. Most changes are related to type fixes which should not affect plugin owners. For most plugins, they use client interact with data plugin. For index management plugin, it uses Client defined in /src/core/server/opensearch/legacy. Though this client is imported from 'elasticsearch' but this is actually "@types/elasticsearch": "^5.0.33" from here. Currently, we don’t plan to change the legacy server and client.

To sum up, we don’t see breaking changes for plugins. I listed some steps to try using anomaly-detection below. SInce we are still fixing the PRs, could try with my test branch. However, since none of the plugins are using opensearch/opensearch-dashboards 2.0 or bump the node&package version accordingly, it is hard to evaluate the full impact. I would highly suggest that plugins bump to version 2.0 with node v14 and upgraded packages. Use the below steps to hook up with the opensearch-dashboards. Run all the unit tests, integration tests, and sanity tested the plugin.

Anomaly Detection build & run steps

1. Use OpenSearch main branch (v2.0). Fork and Clone anomaly-detection backend plugin (https://github.com/opensearch-project/anomaly-detection)
2. run ./gradlew build in anomaly-detection directory
3. run ./gradlew :run to run a single node cluster with anomaly-detection plugin
4. Fork and clone anomaly-detection-dashboards-plugin (https://github.com/opensearch-project/anomaly-detection-dashboards-plugin) into the <rootDir>/plugins for OpenSearch Dashboards with the nodejs client hooked up
5. make sure to use node 14.18.2 and run yarn osd bootstrap inside of OpenSearch Dashboards <rootDir>
6. update "opensearchDashboardsVersion" in <rootDir>/plugins/anomaly-detection-dashboards-plugin/opensearch_dashboards.json from "1.3.0" to "2.0.0"
7. update <rootDir>/config/opensearch_dashboards.yml and add "openserch.ignoreVersionMismatch: true" in this config file. This is because plugin is still using v1.3.0 while dashboards is v2.0.0. This line will ignore the version mismatch check (

   OpenSearch-Dashboards/src/core/server/opensearch/version_check/ensure_opensearch_version.ts

   Line 167 in 8951429

   if (ignoreVersionMismatch) {

   ).
8. run yarn start inside of OpenSearch Dashboards and verify anomaly-detection plugin is running up
9. run tests and play with anomaly-detection plugin

Running anomaly-detection:

Unit test results:

Integration test results:

Expected behavior
Update Dashboards in 2.0 with type changes

OpenSearch Version
2.0

Dashboards Version
2.0

Plugins
plugins also need to update the types