[2.x][Manual Backport ] Back port main to 2.x (#7477)

* Bump OUI to 1.8.0 (#7363) Signed-off-by: Miki <miki@amazon.com> (cherry picked from commit c1bae75) * [Auto Suggest] OpenSearch SQL autosuggest with ANTLR (#7336) * cherry pick Signed-off-by: Eric <menwe@amazon.com> * SQL keyword suggestion Signed-off-by: Eric <menwe@amazon.com> * column and values suggestions Signed-off-by: Eric <menwe@amazon.com> * adjust values suggestions and ignored token Signed-off-by: Eric <menwe@amazon.com> * suggestion type casting Signed-off-by: Eric <menwe@amazon.com> * ignore functions Signed-off-by: Eric <menwe@amazon.com> * case insensitive Signed-off-by: Eric <menwe@amazon.com> * change to use grammar for insensitive case Signed-off-by: Eric <menwe@amazon.com> * fix multi-line issue Signed-off-by: Eric <menwe@amazon.com> * code cleanup Signed-off-by: Eric <menwe@amazon.com> * move ppl out of scope Signed-off-by: Eric <menwe@amazon.com> * rename folder and add ignoring rules Signed-off-by: Eric <menwe@amazon.com> * resolve type issue Signed-off-by: Eric <menwe@amazon.com> * fix ppl suggestion provider issue Signed-off-by: Eric <menwe@amazon.com> * remove function suggestion Signed-off-by: Eric <menwe@amazon.com> * some code clean up and adding tests Signed-off-by: Eric <menwe@amazon.com> * cursor tests Signed-off-by: Eric <menwe@amazon.com> * remove testing setup Signed-off-by: Eric <menwe@amazon.com> * add changelog Signed-off-by: Eric <menwe@amazon.com> * update yarn file Signed-off-by: Eric <menwe@amazon.com> * add missing testing library Signed-off-by: Eric <menwe@amazon.com> * MDS integration Signed-off-by: Eric <menwe@amazon.com> * minor interface change and disable word based suggestion Signed-off-by: Eric <menwe@amazon.com> * update ID_LITERAL and recompile grammar Signed-off-by: Eric <menwe@amazon.com> * column suggest for agg function Signed-off-by: Eric <menwe@amazon.com> * revert version Signed-off-by: Eric <menwe@amazon.com> * opensearch sql syntax highlighting Signed-off-by: Eric <menwe@amazon.com> * add utility tests Signed-off-by: Eric <menwe@amazon.com> * symbol table test Signed-off-by: Eric <menwe@amazon.com> * error listener test Signed-off-by: Eric <menwe@amazon.com> * parse test Signed-off-by: Eric <menwe@amazon.com> * add suggestion provider to single line query editor Signed-off-by: Eric <menwe@amazon.com> * remove one test Signed-off-by: Eric <menwe@amazon.com> * add connection service type Signed-off-by: Eric <menwe@amazon.com> --------- Signed-off-by: Eric <menwe@amazon.com> (cherry picked from commit 9348bd4) * Move TopNavLinks to new Nav Bar Discover (#7326) This PR aims to introduce a new Nav bar in Discover that would contain the Date Picker as well as the Navigation links that are currently part of headers. The Navigation Links have been replaced with the corresponding Icons to provide a modern UX. These setting are currently controlled by query:enhancements:enabled Advanced Setting Flag and would be visible when the above flag is turned on. Signed-off-by: Suchit Sahoo <suchsah@amazon.com> (cherry picked from commit e46e54a) * [navigation-next]fix: breadcrumb issue found in data source management and other similar cases. (#7401) * feat: add scoped breadcrumbs Signed-off-by: SuZhou-Joe <suzhou@amazon.com> * Changeset file for PR #7401 created/updated * feat: move data source management register logic up a little bit Signed-off-by: SuZhou-Joe <suzhou@amazon.com> * feat: add unit test Signed-off-by: SuZhou-Joe <suzhou@amazon.com> * feat: revert the home related change Signed-off-by: SuZhou-Joe <suzhou@amazon.com> * fix: overview error in all use case when workspace is enabled Signed-off-by: SuZhou-Joe <suzhou@amazon.com> * fix: update snapshot Signed-off-by: SuZhou-Joe <suzhou@amazon.com> * Changeset file for PR #7401 created/updated * fix: hide nav groups that should be displayed Signed-off-by: SuZhou-Joe <suzhou@amazon.com> * feat: hide expand icon in left navigation Signed-off-by: SuZhou-Joe <suzhou@amazon.com> * feat: update Signed-off-by: SuZhou-Joe <suzhou@amazon.com> --------- Signed-off-by: SuZhou-Joe <suzhou@amazon.com> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com> Co-authored-by: ZilongX <99905560+ZilongX@users.noreply.github.com> (cherry picked from commit 3f78c94) * [Bug][Data Source] Move data source manageable feature flag to DSM plugin (#7440) * Move data source manageable feature flag to DSM plugin Signed-off-by: yubonluo <yubonluo@amazon.com> * Changeset file for PR #7440 created/updated --------- Signed-off-by: yubonluo <yubonluo@amazon.com> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com> (cherry picked from commit dfcd2e1) * [Discover-next] data set picker (#7426) * update using query manager Signed-off-by: Kawika Avilla <kavilla414@gmail.com> * aggs not working Signed-off-by: Kawika Avilla <kavilla414@gmail.com> * almost there Signed-off-by: Kawika Avilla <kavilla414@gmail.com> * stablish Signed-off-by: Kawika Avilla <kavilla414@gmail.com> * thanks ashwin Signed-off-by: Kawika Avilla <kavilla414@gmail.com> * update ref name Signed-off-by: Kawika Avilla <kavilla414@gmail.com> * fix timefields Signed-off-by: Kawika Avilla <kavilla414@gmail.com> * clean up some console logs Signed-off-by: Kawika Avilla <kavilla414@gmail.com> * go safer route of setting language Signed-off-by: Kawika Avilla <kavilla414@gmail.com> * its working again Signed-off-by: Kawika Avilla <kavilla414@gmail.com> * fix names Signed-off-by: Kawika Avilla <kavilla414@gmail.com> * restore code editor and indices Signed-off-by: Kawika Avilla <kavilla414@gmail.com> * sql df Signed-off-by: Kawika Avilla <kavilla414@gmail.com> * fix external datasources again Signed-off-by: Kawika Avilla <kavilla414@gmail.com> --------- Signed-off-by: Kawika Avilla <kavilla414@gmail.com> (cherry picked from commit 6a079d3) * Fix add filter for numeric scripted field (#7022) * Fix add filter for scripted field Signed-off-by: abbyhu2000 <abigailhu2000@gmail.com> * add unit test Signed-off-by: abbyhu2000 <abigailhu2000@gmail.com> * change from nested ternary to if else Signed-off-by: abbyhu2000 <abigailhu2000@gmail.com> * add more unit tests Signed-off-by: abbyhu2000 <abigailhu2000@gmail.com> * change unit tests Signed-off-by: abbyhu2000 <abigailhu2000@gmail.com> * fix typo Signed-off-by: abbyhu2000 <abigailhu2000@gmail.com> --------- Signed-off-by: abbyhu2000 <abigailhu2000@gmail.com> (cherry picked from commit 7c1f8fc) * [Bug][Workspace] Add permission validation at workspace detail page (#7435) * [Bug][Workspace] Add permission validate at workspace detail page Signed-off-by: yubonluo <yubonluo@amazon.com> * Changeset file for PR #7435 created/updated --------- Signed-off-by: yubonluo <yubonluo@amazon.com> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com> (cherry picked from commit 45d34b5) * Revert "[Discover-next] data set picker (#7426)" (#7479) This reverts commit 6a079d3. --------- Co-authored-by: Miki <miki@amazon.com> Co-authored-by: Eric Wei <menwe@amazon.com> Co-authored-by: SuZhou-Joe <suzhou@amazon.com> Co-authored-by: yuboluo <yubonluo@amazon.com> Co-authored-by: Kawika Avilla <kavilla414@gmail.com> Co-authored-by: Qingyang(Abby) Hu <abigailhu2000@gmail.com> Co-authored-by: Sean Li <lnse@amazon.com> (cherry picked from commit 0515fb2) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
opensearch-project · Jul 25, 2024 · 4a3bf35 · 4a3bf35
1 parent 95b2ef6
commit 4a3bf35
Show file tree

Hide file tree

Showing 76 changed files with 44,032 additions and 115 deletions.
diff --git a/.eslintignore b/.eslintignore
@@ -30,3 +30,7 @@ target
 /packages/osd-test/src/functional_test_runner/lib/config/__tests__/fixtures/
 /packages/osd-ui-framework/dist
 /packages/osd-ui-shared-deps/flot_charts
+
+# antlr overrides
+/src/plugins/data/public/antlr/opensearch_sql/.generated/*
+/src/plugins/data/public/antlr/opensearch_sql/grammar/**/*
diff --git a/.eslintrc.js b/.eslintrc.js
@@ -751,5 +751,14 @@ module.exports = {
         'no-undef': 'off',
       },
     },
+    {
+      files: [
+        'src/plugins/data/public/antlr/opensearch_sql/.generated/*',
+        'src/plugins/data/public/antlr/opensearch_sql/grammar/**/*',
+      ],
+      rules: {
+        'filenames/match-regex': 'off',
+      },
+    },
   ],
 };
diff --git a/changelogs/fragments/7336.yml b/changelogs/fragments/7336.yml
@@ -0,0 +1,2 @@
+feat:
+- [Auto Suggest] OpenSearch SQL autosuggest with ANTLR ([#7336](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/7336))
diff --git a/changelogs/fragments/7363.yml b/changelogs/fragments/7363.yml
@@ -0,0 +1,2 @@
+feat:
+- Bump OUI to 1.8.0 ([#7363](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/7363))
diff --git a/config/opensearch_dashboards.yml b/config/opensearch_dashboards.yml
@@ -362,4 +362,4 @@
 # Set the backend roles in groups or users, whoever has the backend roles or exactly match the user ids defined in this config will be regard as dashboard admin.
 # Dashboard admin will have the access to all the workspaces(workspace.enabled: true) and objects inside OpenSearch Dashboards.
 # opensearchDashboards.dashboardAdmin.groups: ["dashboard_admin"]
-# opensearchDashboards.dashboardAdmin.users: ["dashboard_admin"]
+# opensearchDashboards.dashboardAdmin.users: ["dashboard_admin"]
diff --git a/package.json b/package.json
@@ -77,7 +77,12 @@
     "osd:bootstrap": "scripts/use_node scripts/build_ts_refs && scripts/use_node scripts/register_git_hook",
     "spec_to_console": "scripts/use_node scripts/spec_to_console",
     "pkg-version": "scripts/use_node -e \"console.log(require('./package.json').version)\"",
-    "release_note:generate": "scripts/use_node scripts/generate_release_note"
+    "release_note:generate": "scripts/use_node scripts/generate_release_note",
+    "cypress:run-without-security": "env TZ=America/Los_Angeles NO_COLOR=1 cypress run --headless --env SECURITY_ENABLED=false",
+    "cypress:run-with-security": "env TZ=America/Los_Angeles NO_COLOR=1 cypress run --headless --env SECURITY_ENABLED=true,openSearchUrl=https://localhost:9200,WAIT_FOR_LOADER_BUFFER_MS=500",
+    "osd:ciGroup10": "echo \"dashboard_sanity_test_spec.js\"",
+    "osd:ciGroup11": "echo \"apps/vis_builder/*.js\"",
+    "generate:opensearchsqlantlr": "./node_modules/antlr4ng-cli/index.js -Dlanguage=TypeScript -o ./src/plugins/data/public/antlr/opensearch_sql/.generated -visitor -no-listener -Xexact-output-dir ./src/plugins/data/public/antlr/opensearch_sql/grammar/OpenSearchSQLLexer.g4 ./src/plugins/data/public/antlr/opensearch_sql/grammar/OpenSearchSQLParser.g4"
   },
   "repository": {
     "type": "git",
@@ -109,7 +114,7 @@
     "**/qs": "^6.11.0",
     "**/semver": "^7.5.3",
     "**/set-value": "^4.1.0",
-    "**/tar":"^6.2.1",
+    "**/tar": "^6.2.1",
     "**/topo/hoek": "npm:@amoo-miki/hoek@6.1.3",
     "**/trim": "^0.0.3",
     "**/typescript": "4.0.2",
@@ -140,7 +145,7 @@
   "dependencies": {
     "@aws-crypto/client-node": "^3.1.1",
     "@elastic/datemath": "5.0.3",
-    "@elastic/eui": "npm:@opensearch-project/oui@1.7.0",
+    "@elastic/eui": "npm:@opensearch-project/oui@1.8.0",
     "@elastic/good": "^9.0.1-kibana3",
     "@elastic/numeral": "npm:@amoo-miki/numeral@2.6.0",
     "@elastic/request-crypto": "2.0.0",
@@ -177,6 +182,8 @@
     "JSONStream": "1.3.5",
     "abortcontroller-polyfill": "^1.4.0",
     "ajv": "^8.11.0",
+    "antlr4-c3": "^3.4.1",
+    "antlr4ng": "^3.0.4",
     "bluebird": "3.5.5",
     "chalk": "^4.1.0",
     "chokidar": "^3.4.2",
@@ -242,11 +249,13 @@
     "type-detect": "^4.0.8",
     "uuid": "3.3.2",
     "whatwg-fetch": "^3.0.0",
-    "yauzl": "^2.10.0"
+    "yauzl": "^2.10.0",
+    "@opensearch-dashboards-test/opensearch-dashboards-test-library": "https://github.com/opensearch-project/opensearch-dashboards-test-library/archive/refs/tags/1.0.6.tar.gz"
   },
   "devDependencies": {
     "@babel/core": "^7.22.9",
     "@babel/parser": "^7.22.9",
+    "@babel/plugin-transform-class-static-block": "^7.24.4",
     "@babel/register": "^7.22.9",
     "@babel/types": "^7.22.9",
     "@elastic/apm-rum": "^5.6.1",
@@ -361,6 +370,7 @@
     "@types/zen-observable": "^0.8.0",
     "@typescript-eslint/eslint-plugin": "^3.10.0",
     "@typescript-eslint/parser": "^3.10.0",
+    "antlr4ng-cli": "^2.0.0",
     "archiver": "^5.3.0",
     "axe-core": "^4.0.2",
     "babel-eslint": "^10.0.3",
@@ -461,6 +471,7 @@
     "regenerate": "^1.4.0",
     "reselect": "^4.0.0",
     "resize-observer-polyfill": "^1.5.1",
+    "rimraf": "^5.0.7",
     "selenium-webdriver": "^4.0.0-alpha.7",
     "simple-git": "^3.16.0",
     "sinon": "^7.4.2",

diff --git a/packages/osd-babel-preset/webpack_preset.js b/packages/osd-babel-preset/webpack_preset.js
@@ -55,6 +55,12 @@ module.exports = () => {
           fileName: false,
         },
       ],
+      [
+        require.resolve('@babel/plugin-transform-class-static-block'),
+        {
+          fileName: false,
+        },
+      ],
     ],
     env: {
       production: {

diff --git a/packages/osd-monaco/src/xjson/lexer_rules/index.ts b/packages/osd-monaco/src/xjson/lexer_rules/index.ts
@@ -33,6 +33,7 @@ import * as monaco from 'monaco-editor/esm/vs/editor/editor.api';
 import * as xJson from './xjson';
 import * as opensearchql from './opensearchql';
 import * as painless from './painless';
+import * as opensearchsql from './opensearchsql';
 
 export const registerLexerRules = (m: typeof monaco) => {
   m.languages.register({ id: xJson.ID });
@@ -41,4 +42,6 @@ export const registerLexerRules = (m: typeof monaco) => {
   m.languages.setMonarchTokensProvider(painless.ID, painless.lexerRules);
   m.languages.register({ id: opensearchql.ID });
   m.languages.setMonarchTokensProvider(opensearchql.ID, opensearchql.lexerRules);
+  m.languages.register({ id: opensearchsql.ID });
+  m.languages.setMonarchTokensProvider(opensearchsql.ID, opensearchsql.lexerRules);
 };
diff --git a/packages/osd-monaco/src/xjson/lexer_rules/opensearchsql.ts b/packages/osd-monaco/src/xjson/lexer_rules/opensearchsql.ts
@@ -0,0 +1,157 @@
+/*
+ * Copyright OpenSearch Contributors
+ * SPDX-License-Identifier: Apache-2.0
+ */
+
+import { monaco } from '../../monaco';
+
+export const ID = 'SQL';
+
+const keywords = [
+  'ALL',
+  'AND',
+  'AS',
+  'ASC',
+  'BOOLEAN',
+  'BETWEEN',
+  'BY',
+  'CASE',
+  'CAST',
+  'CROSS',
+  'COLUMNS',
+  'DATETIME',
+  'DELETE',
+  'DESC',
+  'DESCRIBE',
+  'DISTINCT',
+  'DOUBLE',
+  'ELSE',
+  'EXISTS',
+  'FALSE',
+  'FLOAT',
+  'FIRST',
+  'FROM',
+  'GROUP',
+  'HAVING',
+  'IN',
+  'INNER',
+  'INT',
+  'INTEGER',
+  'IS',
+  'JOIN',
+  'LAST',
+  'LEFT',
+  'LIKE',
+  'LIMIT',
+  'LONG',
+  'MATCH',
+  'NATURAL',
+  'NOT',
+  'NULL',
+  'NULLS',
+  'ON',
+  'OR',
+  'ORDER',
+  'OUTER',
+  'OVER',
+  'PARTITION',
+  'REGEXP',
+  'RIGHT',
+  'SELECT',
+  'SHOW',
+  'STRING',
+  'THEN',
+  'TRUE',
+  'UNION',
+  'USING',
+  'WHEN',
+  'WHERE',
+  'EXCEPT',
+];
+
+const functions = [
+  'AVG',
+  'COUNT',
+  'MAX',
+  'MIN',
+  'SUM',
+  'VAR_POP',
+  'VAR_SAMP',
+  'VARIANCE',
+  'STD',
+  'STDDEV',
+  'STDDEV_POP',
+  'STDDEV_SAMP',
+  'SUBSTRING',
+  'TRIM',
+];
+
+const operators = [
+  '=',
+  '>',
+  '<',
+  '!',
+  '~',
+  '\\|',
+  '&',
+  '\\^',
+  '\\*',
+  '/',
+  '%',
+  '\\+',
+  '-',
+  'DIV',
+  'MOD',
+];
+
+const brackets = [
+  { open: '(', close: ')', token: 'delimiter.parenthesis' },
+  { open: '[', close: ']', token: 'delimiter.square' },
+];
+
+export const lexerRules = {
+  defaultToken: 'invalid',
+  ignoreCase: true,
+  tokenPostfix: '',
+  keywords,
+  functions,
+  operators,
+  brackets,
+  tokenizer: {
+    root: [
+      [
+        /[a-zA-Z_$][a-zA-Z0-9_$]*/,
+        {
+          cases: {
+            '@keywords': 'keyword',
+            '@functions': 'function',
+            '@default': 'identifier',
+          },
+        },
+      ],
+      { include: '@whitespace' },
+      [/[()]/, '@brackets'],
+      [new RegExp(operators.join('|')), 'operator'],
+      [/[0-9]+(\.[0-9]+)?/, 'number'],
+      [/'([^'\\]|\\.)*$/, 'string.invalid'], // non-terminated string
+      [/'/, 'string', '@string'],
+      [/"/, 'string', '@string'],
+    ],
+    whitespace: [
+      [/[ \t\r\n]+/, 'white'],
+      [/\/\*/, 'comment', '@comment'],
+      [/--.*$/, 'comment'],
+    ],
+    string: [
+      [/[^'\\]+/, 'string'],
+      [/\\./, 'string.escape'],
+      [/'/, 'string', '@pop'],
+      [/"/, 'string', '@pop'],
+    ],
+    comment: [
+      [/[^/*]+/, 'comment'],
+      [/\*\//, 'comment', '@pop'],
+      [/[\/*]/, 'comment'],
+    ],
+  },
+} as monaco.languages.IMonarchLanguage;
diff --git a/packages/osd-optimizer/src/worker/webpack.config.ts b/packages/osd-optimizer/src/worker/webpack.config.ts
@@ -256,6 +256,16 @@ export function getWebpackConfig(bundle: Bundle, bundleRefs: BundleRefs, worker:
             loader: 'raw-loader',
           },
         },
+        {
+          test: /\.cjs$/,
+          include: /node_modules/,
+          use: {
+            loader: 'babel-loader',
+            options: {
+              presets: [BABEL_PRESET_PATH],
+            },
+          },
+        },
       ],
     },
 

diff --git a/packages/osd-ui-framework/package.json b/packages/osd-ui-framework/package.json
@@ -23,7 +23,7 @@
     "enzyme-adapter-react-16": "^1.9.1"
   },
   "devDependencies": {
-    "@elastic/eui": "npm:@opensearch-project/oui@1.7.0",
+    "@elastic/eui": "npm:@opensearch-project/oui@1.8.0",
     "@osd/babel-preset": "1.0.0",
     "@osd/optimizer": "1.0.0",
     "comment-stripper": "^0.0.4",

diff --git a/packages/osd-ui-shared-deps/package.json b/packages/osd-ui-shared-deps/package.json
@@ -10,7 +10,7 @@
   },
   "dependencies": {
     "@elastic/charts": "31.1.0",
-    "@elastic/eui": "npm:@opensearch-project/oui@1.7.0",
+    "@elastic/eui": "npm:@opensearch-project/oui@1.8.0",
     "@elastic/numeral": "npm:@amoo-miki/numeral@2.6.0",
     "@opensearch/datemath": "5.0.3",
     "@osd/i18n": "1.0.0",

diff --git a/src/core/public/application/types.ts b/src/core/public/application/types.ts
@@ -539,6 +539,8 @@ export interface AppMountParameters<HistoryLocationState = unknown> {
    * Optional datasource id to pass while mounting app
    */
   dataSourceId?: string;
+
+  optionalRef?: Record<string, React.RefObject<HTMLDivElement>>;
 }
 
 /**