Multi datasource support #311

sumukhswamy · 2024-04-23T01:54:43Z

Description

mds support for neo
cache changes
#303

Issues Resolved

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: sumukhswamy <sumukhhs@amazon.com>

BionIT · 2024-04-23T16:13:58Z

common/utils/fetch_datasources.ts

+            }
+
+            connectorGroups[connector].push(name);
+            if (name === urlDataSource) {


what is urlDataSource? Can we give a better name?

public/components/QueryLanguageSwitch/ClusterTabs.tsx

BionIT · 2024-04-23T16:15:47Z

public/components/QueryLanguageSwitch/ClusterTabs.tsx

+  constructor(props: SwitchProps) {
+    super(props);
+    this.state = {
+      cluster: 'Indexes',


do we use this anywhere?

The main reason I have a state here is to distinguish between indexes tab and data connections, which I check in
https://github.com/opensearch-project/dashboards-query-workbench/pull/311/files#diff-3ff08cae50890804a95e00ef9f0f2a560c5ef12a4075e322612f7008bcdaa978R23

public/components/SQLPage/SQLPage.tsx

public/components/SQLPage/sql_catalog_tree/s3_tree.tsx

BionIT · 2024-04-23T16:19:06Z

Can we add test details with screenshots or recording?

joshuali925

didn't finish, will review later

common/utils/fetch_datasources.ts

Signed-off-by: sumukhswamy <sumukhhs@amazon.com>

Co-authored-by: Joshua Li <joshuali925@gmail.com> Signed-off-by: Sumukh Swamy <sumukhhs@amazon.com>

sumukhswamy · 2024-04-25T23:52:59Z

Screen.Recording.2024-04-25.at.4.50.21.PM.mov

common/utils/fetch_datasources.ts

Signed-off-by: sumukhswamy <sumukhhs@amazon.com>

public/components/Main/main.tsx

public/application.tsx

public/components/SQLPage/sql_catalog_tree/s3_tree_helpers.tsx

server/services/QueryService.ts

Signed-off-by: sumukhswamy <sumukhhs@amazon.com>

ps48 · 2024-04-30T22:00:25Z

public/components/SQLPage/sql_catalog_tree/catalog_tree.tsx

+      ):(
+        <EuiFlexItem grow={false}>
+          <EuiEmptyPrompt
+            icon={<EuiIcon type="database" size="m" />}
+            iconColor="subdued"
+            titleSize="xs"
+            body={<p>Select a Data Source Connection to fetch Databases</p>}
+          />
+        </EuiFlexItem>


Nested ternary operators are difficult to understand, maybe can be split in variables like done inside the tree code.

Signed-off-by: sumukhswamy <sumukhhs@amazon.com>

ps48 · 2024-04-30T22:06:56Z

@sumukhswamy Can you please create an issue for fast follow on the linter issues? We can fix these after this PR is merged.

Signed-off-by: sumukhswamy <sumukhhs@amazon.com>

opensearch-trigger-bot · 2024-05-01T00:05:52Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/dashboards-query-workbench/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/dashboards-query-workbench/backport-2.x
# Create a new branch
git switch --create backport/backport-311-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 bb6116d4110500f89c62c03a3472013c311b2be2
# Push it to GitHub
git push --set-upstream origin backport/backport-311-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/dashboards-query-workbench/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-311-to-2.x.

opensearch-trigger-bot · 2024-05-01T00:05:53Z

The backport to 2.14 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/dashboards-query-workbench/backport-2.14 2.14
# Navigate to the new working tree
pushd ../.worktrees/dashboards-query-workbench/backport-2.14
# Create a new branch
git switch --create backport/backport-311-to-2.14
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 bb6116d4110500f89c62c03a3472013c311b2be2
# Push it to GitHub
git push --set-upstream origin backport/backport-311-to-2.14
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/dashboards-query-workbench/backport-2.14

Then, create a pull request where the base branch is 2.14 and the compare/head branch is backport/backport-311-to-2.14.

* added mds changes Signed-off-by: sumukhswamy <sumukhhs@amazon.com> * added mds changes Signed-off-by: sumukhswamy <sumukhhs@amazon.com> * added more changes Signed-off-by: sumukhswamy <sumukhhs@amazon.com> * upadted the group buttons and calls with mds Signed-off-by: sumukhswamy <sumukhhs@amazon.com> * added changes for and cache management Signed-off-by: sumukhswamy <sumukhhs@amazon.com> * added changes for PPL page, changes for fetching queries Signed-off-by: sumukhswamy <sumukhhs@amazon.com> * main test added Signed-off-by: sumukhswamy <sumukhhs@amazon.com> * added datsource dependency Signed-off-by: sumukhswamy <sumukhhs@amazon.com> * updated the tests for workbench with snapshots Signed-off-by: sumukhswamy <sumukhhs@amazon.com> * Update common/utils/fetch_datasources.ts Co-authored-by: Joshua Li <joshuali925@gmail.com> Signed-off-by: Sumukh Swamy <sumukhhs@amazon.com> * updated lint checker in async workbench Signed-off-by: sumukhswamy <sumukhhs@amazon.com> * updated with new comments Signed-off-by: sumukhswamy <sumukhhs@amazon.com> * added UX changes Signed-off-by: sumukhswamy <sumukhhs@amazon.com> * added changed for pr comments Signed-off-by: sumukhswamy <sumukhhs@amazon.com> * added changes for linter Signed-off-by: sumukhswamy <sumukhhs@amazon.com> * added changes for snapshop Signed-off-by: sumukhswamy <sumukhhs@amazon.com> * added snapshot for main Signed-off-by: sumukhswamy <sumukhhs@amazon.com> * updated tests for main Signed-off-by: sumukhswamy <sumukhhs@amazon.com> --------- Signed-off-by: sumukhswamy <sumukhhs@amazon.com> Signed-off-by: Sumukh Swamy <sumukhhs@amazon.com> Co-authored-by: Joshua Li <joshuali925@gmail.com>

* added mds changes * added mds changes * added more changes * upadted the group buttons and calls with mds * added changes for and cache management * added changes for PPL page, changes for fetching queries * main test added * added datsource dependency * updated the tests for workbench with snapshots * Update common/utils/fetch_datasources.ts * updated lint checker in async workbench * updated with new comments * added UX changes * added changed for pr comments * added changes for linter * added changes for snapshop * added snapshot for main * updated tests for main --------- Signed-off-by: sumukhswamy <sumukhhs@amazon.com> Signed-off-by: Sumukh Swamy <sumukhhs@amazon.com> Co-authored-by: Joshua Li <joshuali925@gmail.com>

sumukhswamy added 5 commits April 4, 2024 13:01

added mds changes

1a98286

Signed-off-by: sumukhswamy <sumukhhs@amazon.com>

added mds changes

a1e7567

Signed-off-by: sumukhswamy <sumukhhs@amazon.com>

added more changes

596e1c5

Signed-off-by: sumukhswamy <sumukhhs@amazon.com>

upadted the group buttons and calls with mds

aa0c837

Signed-off-by: sumukhswamy <sumukhhs@amazon.com>

added changes for and cache management

4150721

Signed-off-by: sumukhswamy <sumukhhs@amazon.com>

sumukhswamy requested review from ps48, kavithacm, derek-ho, joshuali925, dai-chen, YANG-DB, rupal-bq, mengweieric, vamsi-amazon, Swiddis, penghuo, seankao-az, Yury-Fridlyand, anirudha and paulstn as code owners April 23, 2024 01:54

BionIT reviewed Apr 23, 2024

View reviewed changes

joshuali925 reviewed Apr 23, 2024

View reviewed changes

common/utils/fetch_datasources.ts Outdated Show resolved Hide resolved

common/utils/fetch_datasources.ts Outdated Show resolved Hide resolved

sumukhswamy and others added 6 commits April 24, 2024 18:19

added changes for PPL page, changes for fetching queries

fad9c25

Signed-off-by: sumukhswamy <sumukhhs@amazon.com>

main test added

737c959

Signed-off-by: sumukhswamy <sumukhhs@amazon.com>

added datsource dependency

3896e60

Signed-off-by: sumukhswamy <sumukhhs@amazon.com>

updated the tests for workbench with snapshots

3b5137d

Signed-off-by: sumukhswamy <sumukhhs@amazon.com>

Update common/utils/fetch_datasources.ts

5a08ea2

Co-authored-by: Joshua Li <joshuali925@gmail.com> Signed-off-by: Sumukh Swamy <sumukhhs@amazon.com>

Merge branch 'main' into multi-datasource

75e01b3

zhyuanqi reviewed Apr 26, 2024

View reviewed changes

common/utils/fetch_datasources.ts Outdated Show resolved Hide resolved

common/utils/fetch_datasources.ts Show resolved Hide resolved

updated with new comments

8de3cce

Signed-off-by: sumukhswamy <sumukhhs@amazon.com>

sumukhswamy force-pushed the multi-datasource branch from 132f6b4 to 8de3cce Compare April 27, 2024 00:38