[Auto Suggest] Add MDS Support Along with A Few Cleanup and tests (#7463

) * add tests for sql autocomplete rule processing Signed-off-by: Eric <menwe@amazon.com> * refer to monaco type directly Signed-off-by: Eric <menwe@amazon.com> * remove unnecessary antlr auto generated files Signed-off-by: Eric <menwe@amazon.com> * inital adoption of dataSet manager Signed-off-by: Eric <menwe@amazon.com> * mds support Signed-off-by: Eric <menwe@amazon.com> * remove test that are failed due to adopting dataSet manager Signed-off-by: Eric <menwe@amazon.com> * add changelog Signed-off-by: Eric <menwe@amazon.com> * fix(query assist): update reading data source id from dataset manager (#7464) * revert to read datasource id from index pattern Signed-off-by: Joshua Li <joshuali925@gmail.com> * add dataset mock to query mock Signed-off-by: Joshua Li <joshuali925@gmail.com> * update query assist to use dataset manager Signed-off-by: Joshua Li <joshuali925@gmail.com> * use selected dataset state instead of relying on rerender Signed-off-by: Joshua Li <joshuali925@gmail.com> * remove skip 1 in dataset observable Signed-off-by: Joshua Li <joshuali925@gmail.com> * update dataset_manager tests Signed-off-by: Joshua Li <joshuali925@gmail.com> --------- Signed-off-by: Joshua Li <joshuali925@gmail.com> * update utils Signed-off-by: Eric <menwe@amazon.com> * keep with observable and remove values suggestion Signed-off-by: Eric <menwe@amazon.com> * update unit tests Signed-off-by: Eric <menwe@amazon.com> * [Auto Suggest] DQL autosuggest with ANTLR (#7467) * Antlr autocomplete (#7159) * dql grammar with rudamentary testing parser Signed-off-by: Paul Sebastian <paulstn@amazon.com> * show suggestion of fields depending on current index pattern Signed-off-by: Paul Sebastian <paulstn@amazon.com> * basic code completion with fields populated Signed-off-by: Paul Sebastian <paulstn@amazon.com> * updated grammar and generated for better group handling Signed-off-by: Paul Sebastian <paulstn@amazon.com> * add ignored tokens Signed-off-by: Paul Sebastian <paulstn@amazon.com> * remove console logs Signed-off-by: Paul Sebastian <paulstn@amazon.com> --------- Signed-off-by: Paul Sebastian <paulstn@amazon.com> * dql Antlr autocomplete (#7160) * re-add provider for sql Signed-off-by: Paul Sebastian <paulstn@amazon.com> * added temporary fix for language providor to appear for more than one language Signed-off-by: Paul Sebastian <paulstn@amazon.com> --------- Signed-off-by: Paul Sebastian <paulstn@amazon.com> * remove EOF in parser to fix suggestions Signed-off-by: Paul Sebastian <paulstn@amazon.com> * use custom version of cursor token index for dql Signed-off-by: Paul Sebastian <paulstn@amazon.com> * implemented value suggestions based on field Signed-off-by: Paul Sebastian <paulstn@amazon.com> * set param type Signed-off-by: Paul Sebastian <paulstn@amazon.com> * update grouping grammar Signed-off-by: Paul Sebastian <paulstn@amazon.com> * fix grammar for dots in field and value term search with spaces Signed-off-by: Paul Sebastian <paulstn@amazon.com> * value suggestions match field to avoid failing api call and to find assc keyword field Signed-off-by: Paul Sebastian <paulstn@amazon.com> * update value suggestions from partially formed value Signed-off-by: Paul Sebastian <paulstn@amazon.com> * refactor value suggestions and change fieldval listener to visitor Signed-off-by: Paul Sebastian <paulstn@amazon.com> * implement value suggestions within phrases Signed-off-by: Paul Sebastian <paulstn@amazon.com> * make grammar more readable Signed-off-by: Paul Sebastian <paulstn@amazon.com> * rename grammar parser rules Signed-off-by: Paul Sebastian <paulstn@amazon.com> * bring back minimal autocomplete optimized grammar Signed-off-by: Paul Sebastian <paulstn@amazon.com> * enable partially complete value suggestion for value groups Signed-off-by: Paul Sebastian <paulstn@amazon.com> * remove number as lexer rule Signed-off-by: Paul Sebastian <paulstn@amazon.com> * fix cursor import and clean up Signed-off-by: Paul Sebastian <paulstn@amazon.com> * fix completion item range to be current word Signed-off-by: Paul Sebastian <paulstn@amazon.com> * update cursor to use monaco position Signed-off-by: Paul Sebastian <paulstn@amazon.com> * cursor index to use position directly Signed-off-by: Paul Sebastian <paulstn@amazon.com> * move language registration into render function to handle new languages Signed-off-by: Paul Sebastian <paulstn@amazon.com> * include auto closing quotes and parenthesis for dql Signed-off-by: Paul Sebastian <paulstn@amazon.com> * rename generated file Signed-off-by: Paul Sebastian <paulstn@amazon.com> * include single line editor closing pairs Signed-off-by: Paul Sebastian <paulstn@amazon.com> * Changeset file for PR #7391 created/updated * add license and fix linting Signed-off-by: Paul Sebastian <paulstn@amazon.com> * modify grammar Signed-off-by: Paul Sebastian <paulstn@amazon.com> * add tests for fields and keywords Signed-off-by: Paul Sebastian <paulstn@amazon.com> * move dql test constants to separate file Signed-off-by: Paul Sebastian <paulstn@amazon.com> * pass core setup from autocomplete constructor to query sugg provider and utilize selectionEnd if no position Signed-off-by: Paul Sebastian <paulstn@amazon.com> * update an import Signed-off-by: Paul Sebastian <paulstn@amazon.com> * use updated dataset for index pattern Signed-off-by: Paul Sebastian <paulstn@amazon.com> * remove console log Signed-off-by: Paul Sebastian <paulstn@amazon.com> --------- Signed-off-by: Paul Sebastian <paulstn@amazon.com> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com> * [tests][discover-next] update the tests and async nature of the dataset navigator (#7489) * [tests][discover-next] update the tests and async nature of the dataset manager Address test failures related to the dataset navigator. Signed-off-by: Kawika Avilla <kavilla414@gmail.com> * bad fingers accidentally hit the x button Signed-off-by: Kawika Avilla <kavilla414@gmail.com> --------- Signed-off-by: Kawika Avilla <kavilla414@gmail.com> * resolve conflicts Signed-off-by: Eric <menwe@amazon.com> * fix one minor linting Signed-off-by: Eric <menwe@amazon.com> --------- Signed-off-by: Eric <menwe@amazon.com> Signed-off-by: Joshua Li <joshuali925@gmail.com> Signed-off-by: Paul Sebastian <paulstn@amazon.com> Signed-off-by: Kawika Avilla <kavilla414@gmail.com> Signed-off-by: Eric Wei <menwe@amazon.com> Co-authored-by: Joshua Li <joshuali925@gmail.com> Co-authored-by: Paul Sebastian <paulstn@amazon.com> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com> Co-authored-by: Kawika Avilla <kavilla414@gmail.com> Co-authored-by: Ashwin P Chandran <ashwinpc@amazon.com>
opensearch-project · Jul 25, 2024 · 9f68352 · 9f68352
1 parent 9f49ce3
commit 9f68352
Show file tree

Hide file tree

Showing 17 changed files with 286 additions and 20,954 deletions.
diff --git a/.gitignore b/.gitignore
@@ -68,3 +68,6 @@ snapshots.js
 
 # Yarn local mirror content
 .yarn-local-mirror
+
+# Ignore the generated antlr files
+/src/plugins/data/public/antlr/opensearch_sql/grammar/.antlr
diff --git a/changelogs/fragments/7463.yml b/changelogs/fragments/7463.yml
@@ -0,0 +1,2 @@
+feat:
+- Add MDS support along with a few cleanup and tests update ([#7463](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/7463))
diff --git a/src/plugins/data/public/antlr/opensearch_sql/code_completion.ts b/src/plugins/data/public/antlr/opensearch_sql/code_completion.ts
@@ -3,7 +3,7 @@
  * SPDX-License-Identifier: Apache-2.0
  */
 
-import { monaco } from 'packages/osd-monaco/target';
+import { monaco } from '@osd/monaco';
 import { Lexer as LexerType, ParserRuleContext, Parser as ParserType } from 'antlr4ng';
 import { CodeCompletionCore } from 'antlr4-c3';
 import {
@@ -21,10 +21,9 @@ import { createParser } from './parse';
 import { SqlErrorListener } from './sql_error_listerner';
 import { findCursorTokenIndex } from '../shared/cursor';
 import { openSearchSqlAutocompleteData } from './opensearch_sql_autocomplete';
-import { getUiSettings } from '../../services';
 import { SQL_SYMBOLS } from './constants';
-import { QuerySuggestionGetFnArgs } from '../../autocomplete';
-import { fetchColumnValues, fetchTableSchemas } from '../shared/utils';
+import { QuerySuggestion, QuerySuggestionGetFnArgs } from '../../autocomplete';
+import { fetchTableSchemas } from '../shared/utils';
 
 export interface SuggestionParams {
   position: monaco.Position;
@@ -44,68 +43,48 @@ export const getSuggestions = async ({
   selectionEnd,
   position,
   query,
-  connectionService,
-}: QuerySuggestionGetFnArgs): Promise<ISuggestionItem[]> => {
-  const { api } = getUiSettings();
+  openSearchDashboards,
+}: QuerySuggestionGetFnArgs): Promise<QuerySuggestion[]> => {
+  const { api } = openSearchDashboards.uiSettings;
+  const dataSetManager = openSearchDashboards.data.query.dataSet;
   const suggestions = getOpenSearchSqlAutoCompleteSuggestions(query, {
     line: position?.lineNumber || selectionStart,
     column: position?.column || selectionEnd,
   });
 
-  const finalSuggestions = [];
+  const finalSuggestions = [] as QuerySuggestion[];
 
   try {
     // Fetch columns and values
     if ('suggestColumns' in suggestions && (suggestions.suggestColumns?.tables?.length ?? 0) > 0) {
       const tableNames = suggestions.suggestColumns?.tables?.map((table) => table.name) ?? [];
-      const schemas = await fetchTableSchemas(tableNames, api, connectionService);
+      const schemas = await fetchTableSchemas(tableNames, api, dataSetManager);
 
       schemas.forEach((schema) => {
         if (schema.body?.fields?.length > 0) {
           const columns = schema.body.fields.find((col) => col.name === 'COLUMN_NAME');
           const fieldTypes = schema.body.fields.find((col) => col.name === 'DATA_TYPE');
           if (columns && fieldTypes) {
             finalSuggestions.push(
-              ...columns.values.map((col: string, index: number) => ({
+              ...columns.values.map((col: string) => ({
                 text: col,
-                type: 'field',
-                fieldType: fieldTypes.values[index],
+                type: monaco.languages.CompletionItemKind.Field,
               }))
             );
           }
         }
       });
 
-      if (
-        'suggestValuesForColumn' in suggestions &&
-        /\S/.test(suggestions.suggestValuesForColumn as string) &&
-        suggestions.suggestValuesForColumn !== undefined
-      ) {
-        const values = await fetchColumnValues(
-          tableNames,
-          suggestions.suggestValuesForColumn as string,
-          api,
-          connectionService
-        );
-        values.forEach((value) => {
-          if (value.body?.fields?.length > 0) {
-            finalSuggestions.push(
-              ...value.body.fields[0].values.map((colVal: string) => ({
-                text: `'${colVal}'`,
-                type: 'value',
-              }))
-            );
-          }
-        });
-      }
+      // later TODO: fetch column values, currently within the industry, it's not a common practice to suggest
+      // values due to different types of the columns as well as the performance impact. For now just avoid it.
     }
 
     // Fill in aggregate functions
     if ('suggestAggregateFunctions' in suggestions && suggestions.suggestAggregateFunctions) {
       finalSuggestions.push(
         ...SQL_SYMBOLS.AGREGATE_FUNCTIONS.map((af) => ({
           text: af,
-          type: 'function',
+          type: monaco.languages.CompletionItemKind.Function,
         }))
       );
     }
@@ -115,12 +94,13 @@ export const getSuggestions = async ({
       finalSuggestions.push(
         ...(suggestions.suggestKeywords ?? []).map((sk) => ({
           text: sk.value,
-          type: 'keyword',
+          type: monaco.languages.CompletionItemKind.Keyword,
         }))
       );
     }
   } catch (error) {
     // TODO: pipe error to the UI
+    return [];
   }
 
   return finalSuggestions;