Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MDS] Data Connection details page with MDS support #7323

Merged
merged 98 commits into from
Jul 22, 2024

Conversation

RyanL1997
Copy link
Contributor

@RyanL1997 RyanL1997 commented Jul 19, 2024

Description

[MDS] Data Connection details page with MDS support #7323

Issues Resolved

Demo

I attached a demo video in this msg, Since the cache loading part takes so long, I will give you some key checkpoint of this video: https://drive.google.com/file/d/1NWs5t8e3PmF8bmm86LpfYM4skrq8dGXJ/view?usp=sharing (Due to the size of github uploading, I have to upload it here.)

  • This demo is recoded under a simulation of neo environment (multi-datasource enable), and this video is mainly focusing on sending out flint related request/operations to a remote cluster.
  • 0:00 ~ 1:11: Configure a remote cluster with flint setup with some installed flint data sources
  • 1:31 ~ 4:22: Start pulling cache into session storage (This is taking a bit long since it is the first loading)
  • 4:30 ~ 4:36: Show the acceleration flyout with a single acceleration in the associated object table
  • 4:37 ~ 4:45: Show the associated object flyout with multiple accelerations in associated object table with table column details
  • 4:48 ~ 5:00: Switch to acceleration table tab and show the acceleration creation flow in different places
  • 5:00 ~ 5:50: Start configuring a acceleration (covering index) creation. -> submit the creation request -> success -> table updating
  • 5:57 ~ 6:07: Show the acceleration details flyout with the cover index we just created
  • 6:08 ~ 8:30: In the acceleration details flyout, submitting a delete operation for the covering index we just created. -> success -> table updating
  • 8:30 ~ 8:51: Checking the updated status for that covering index as 'deleted' in both associated object table and acceleration table

Integration flow

Since the original integration plugin in dashboaards observability does not support MDS, so the integration flow for direct query data source will only be enabled when both MDS is disabled and dashboards observability is installed.

With Integration flow - observability dashboard not installed OR MDS is enabled

Screenshot 2024-07-22 at 3 24 12 AM

With Integration flow

Screenshot 2024-07-22 at 3 28 53 AM

Testing the changes

Changelog

  • feat: [MDS] Data Connection details page with MDS support

Check List

  • All tests pass
    • yarn test:jest
    • yarn test:jest_integration
  • New functionality includes testing.
  • New functionality has been documented.
  • Update CHANGELOG.md
  • Commits are signed per the DCO using --signoff

Copy link
Contributor

❌ Empty Changelog Section

The Changelog section in your PR description is empty. Please add a valid changelog entry or entries. If you did add a changelog entry, check to make sure that it was not accidentally included inside the comment block in the Changelog section.

@RyanL1997 RyanL1997 changed the title [MDS] Observability datasource migration [MDS] Observability datasource migration part 2 Jul 19, 2024
Copy link
Contributor

❌ Empty Changelog Section

The Changelog section in your PR description is empty. Please add a valid changelog entry or entries. If you did add a changelog entry, check to make sure that it was not accidentally included inside the comment block in the Changelog section.

Copy link

codecov bot commented Jul 19, 2024

Codecov Report

Attention: Patch coverage is 66.18141% with 604 lines in your changes missing coverage. Please review.

Project coverage is 67.61%. Comparing base (3526f96) to head (c2d2c14).

Files Patch % Lines
...iated_object_management/associated_objects_tab.tsx 48.90% 44 Missing and 26 partials ⚠️
...anagement/framework/catalog_cache/cache_loader.tsx 64.17% 40 Missing and 8 partials ⚠️
..._components/acceleration_creation/create/utils.tsx 34.61% 34 Missing ⚠️
..._source_management/framework/utils/use_polling.tsx 49.12% 27 Missing and 2 partials ⚠️
...cceleration_creation/selectors/source_selector.tsx 60.86% 19 Missing and 8 partials ⚠️
...eration_management/acceleration_details_flyout.tsx 70.96% 18 Missing and 9 partials ⚠️
...ted_object_management/associated_objects_table.tsx 60.93% 17 Missing and 8 partials ⚠️
...nts/acceleration_management/acceleration_table.tsx 69.23% 15 Missing and 9 partials ⚠️
...anagement/framework/catalog_cache/cache_manager.ts 75.28% 14 Missing and 8 partials ⚠️
.../visual_editors/skipping_index/generate_fields.tsx 37.14% 19 Missing and 3 partials ⚠️
... and 30 more
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #7323      +/-   ##
==========================================
- Coverage   67.75%   67.61%   -0.14%     
==========================================
  Files        3521     3583      +62     
  Lines       69781    72172    +2391     
  Branches    11390    11810     +420     
==========================================
+ Hits        47277    48799    +1522     
- Misses      19694    20388     +694     
- Partials     2810     2985     +175     
Flag Coverage Δ
Linux_1 33.15% <ø> (ø)
Linux_2 55.50% <ø> (ø)
Linux_3 44.50% <66.18%> (+1.19%) ⬆️
Linux_4 34.07% <14.30%> (-0.74%) ⬇️
Windows_1 33.17% <ø> (ø)
Windows_2 55.45% <ø> (ø)
Windows_3 44.52% <66.18%> (+1.20%) ⬆️
Windows_4 34.07% <14.30%> (-0.74%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

❌ Empty Changelog Section

The Changelog section in your PR description is empty. Please add a valid changelog entry or entries. If you did add a changelog entry, check to make sure that it was not accidentally included inside the comment block in the Changelog section.

1 similar comment
Copy link
Contributor

❌ Empty Changelog Section

The Changelog section in your PR description is empty. Please add a valid changelog entry or entries. If you did add a changelog entry, check to make sure that it was not accidentally included inside the comment block in the Changelog section.

@RyanL1997 RyanL1997 changed the title [MDS] Observability datasource migration part 2 [MDS] Data Connection details page with MDS support Jul 20, 2024
Copy link
Contributor

❌ Empty Changelog Section

The Changelog section in your PR description is empty. Please add a valid changelog entry or entries. If you did add a changelog entry, check to make sure that it was not accidentally included inside the comment block in the Changelog section.

Signed-off-by: Ryan Liang <jiallian@amazon.com>
Signed-off-by: Ryan Liang <jiallian@amazon.com>
Signed-off-by: Ryan Liang <jiallian@amazon.com>
Signed-off-by: Ryan Liang <jiallian@amazon.com>
Signed-off-by: Ryan Liang <jiallian@amazon.com>
Signed-off-by: Ryan Liang <jiallian@amazon.com>
Signed-off-by: Ryan Liang <jiallian@amazon.com>
Signed-off-by: Ryan Liang <jiallian@amazon.com>
Signed-off-by: Ryan Liang <jiallian@amazon.com>
Signed-off-by: Ryan Liang <jiallian@amazon.com>
Signed-off-by: Ryan Liang <jiallian@amazon.com>
Signed-off-by: Ryan Liang <jiallian@amazon.com>
Signed-off-by: Ryan Liang <jiallian@amazon.com>
Signed-off-by: Ryan Liang <jiallian@amazon.com>
@mengweieric
Copy link
Collaborator

checked that failing functional test CI does not relate to the changes from this PR

@mengweieric mengweieric merged commit 34f1b73 into opensearch-project:main Jul 22, 2024
65 of 67 checks passed
@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch-Dashboards/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch-Dashboards/backport-2.x
# Create a new branch
git switch --create backport/backport-7323-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 34f1b73a2aa95bbddcdb4605a7f5b69ec8e8c1d8
# Push it to GitHub
git push --set-upstream origin backport/backport-7323-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch-Dashboards/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-7323-to-2.x.

@mengweieric
Copy link
Collaborator

will merge this backport PR soon and retry backport

RyanL1997 added a commit to RyanL1997/OpenSearch-Dashboards that referenced this pull request Jul 23, 2024
…ct#7323)

* Data connection details draft - 0

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Data connection details draft 1 - Add datasource overview section

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add access control tab draft 0 - add the tab of ac

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add access control tab draft 1 - add the ac as a tab

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Cache manager draft 0 - setup for with a testable acceleration table

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Cache manager draft 1 - fix the acceleration name

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Cache manager draft 2 - fix the acceleration status color

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Cache manager draft 3 - Add action menu and action overlay

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Acceleration draft 0 - Implement acceleration actions

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Acceleration draft 1 - Fix the table title

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Acceleration draft 2 - still having issue on the settings api call

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Acceleration draft 3 - fix the fetching apis and still having issue on the notiffication of flyout operation

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Acceleration draft 4 - fix the flyout operation

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix add mds for acceleration flow draft 0 - need to pass mds id into detail page call

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add mds for acceleration flow draft 1 - add feature flag check and pass in id for fetch select datasource

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add mds for acceleration flow draft 2 - add mds flow to all the acceleration table action flow

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add mds for acceleration flow draft 3 - remove unused variable in acceleration table class

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix the route for datasource detail

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add mds for acceleration flow draft 4 - add mds support for acceleration operations table triggered flow

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add mds for acceleration flow draft 5 - add mds support for acceleration operation flyout trigger flow

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add acceleration creation flow draft 0 - directly migrate the entire acc create flow

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add acceleration creation flow draft 1 - relocate the flyout render into plugin ts

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add acceleration creation flow draft 2 - fix the query work bench check

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add acceleration creation flow draft 3 - fix the acceleration creation flow when mds is disabled 0

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add acceleration creation flow draft 4 - unrelated to the creation flow but add the cache hooks for all the meta info

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add acceleration creation flow draft 5 - mds support with acceleration creation flow

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add associated objects flow draft 0 - works without adding mds support

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Remove some comments 0

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add associated objects flow draft 1 - fix the navi to app

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add associated objects flow draft 2 - fix the navi to app and add mds support 0

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test 0 - add unit test for ao utils

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test 1 - setup utils dir for connection detail and add unit tests

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test 2 - Add uniti test for acc flyout modules 0

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Enable card on connection details for query in discover

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add the mds support for acc flyout details fetch functions

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test 3 - Add unit test for plugin ts flyout setup

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test 4 - Add unit test for some of the acceleration components

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test 5 - clean up the operation unit test a bit

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Implement the acceleration flyout forwarding with mds support

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add discover forwarding action

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test 6 - add more test cases on accelerations comoponents

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* IMPORTANT - fix mds flows for cashloader when mds enabled

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* IMPORTANT - fix mds flows for direct query when mds enabled

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix acc details tab tests

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix everything in mds refresh

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* P00000000 - revert the change of cacheloader and correct the cacheloader and cachemnger usage

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Uncomment security role check

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix unit tests snapshot

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test 7 - add unit tests for acceleration creation main flow

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add the server setup

Co-authored-by: Shenoy Pratik <sgguruda@amazon.com>
Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add the server part of dsm without renaming

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* rename 0 - base has been renamed test yes

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* rename 1 - class has been renamed test yes

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix the test issue

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix the details flyout

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test 8 - add more test for acc creation flow

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Changeset file for PR opensearch-project#7323 created/updated

* Enable the query workbench forwarding

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Resolve conflict router

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test 7 - fix the acceleration creation testing

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test 8 - Add tests for caching components

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Test update

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix the acc creation button in ao tab

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Relocate the framework test dir

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add more tests on framework

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test - use polling

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test - acc table

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test - acc tab

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Rename catlog into catalog

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add more unit tests

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add test for ao table

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix the test for refreshing time

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix the snapshot

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix the timezone issue in test again

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test - add more tests for acc creation related components

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Rename dir - 1

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Rename dir - 2 acceleration_management

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Rename dir - 3 associated_objects_management

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Rename dir - 4 connection_detail

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* fix time zone issue again?

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* integration - local cluster setup without flyout

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* integration - local cluster setup plus flyout

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix the feature flag status undefined issue when the navigation feature is enabled

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix tests and add more error handling

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Integration - add render conditions and fix tests

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Integration - add unit tests p1

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Integration - add unit tests p2

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* test cov

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* test cov

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* test cov +

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* test cov +

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* test cov +

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Resolve comments

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Resolve comments - 2

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Resolve comments - 3

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Resolve comments - 4

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Resolve comments - 5

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Bond the obs check for query card

Signed-off-by: Ryan Liang <jiallian@amazon.com>

---------

Signed-off-by: Ryan Liang <jiallian@amazon.com>
Co-authored-by: Shenoy Pratik <sgguruda@amazon.com>
Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com>
BionIT pushed a commit that referenced this pull request Jul 23, 2024
) (#7397)

* [MDS] Data Connection details page with MDS support (#7323)

* Data connection details draft - 0

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Data connection details draft 1 - Add datasource overview section

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add access control tab draft 0 - add the tab of ac

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add access control tab draft 1 - add the ac as a tab

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Cache manager draft 0 - setup for with a testable acceleration table

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Cache manager draft 1 - fix the acceleration name

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Cache manager draft 2 - fix the acceleration status color

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Cache manager draft 3 - Add action menu and action overlay

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Acceleration draft 0 - Implement acceleration actions

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Acceleration draft 1 - Fix the table title

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Acceleration draft 2 - still having issue on the settings api call

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Acceleration draft 3 - fix the fetching apis and still having issue on the notiffication of flyout operation

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Acceleration draft 4 - fix the flyout operation

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix add mds for acceleration flow draft 0 - need to pass mds id into detail page call

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add mds for acceleration flow draft 1 - add feature flag check and pass in id for fetch select datasource

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add mds for acceleration flow draft 2 - add mds flow to all the acceleration table action flow

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add mds for acceleration flow draft 3 - remove unused variable in acceleration table class

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix the route for datasource detail

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add mds for acceleration flow draft 4 - add mds support for acceleration operations table triggered flow

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add mds for acceleration flow draft 5 - add mds support for acceleration operation flyout trigger flow

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add acceleration creation flow draft 0 - directly migrate the entire acc create flow

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add acceleration creation flow draft 1 - relocate the flyout render into plugin ts

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add acceleration creation flow draft 2 - fix the query work bench check

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add acceleration creation flow draft 3 - fix the acceleration creation flow when mds is disabled 0

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add acceleration creation flow draft 4 - unrelated to the creation flow but add the cache hooks for all the meta info

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add acceleration creation flow draft 5 - mds support with acceleration creation flow

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add associated objects flow draft 0 - works without adding mds support

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Remove some comments 0

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add associated objects flow draft 1 - fix the navi to app

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add associated objects flow draft 2 - fix the navi to app and add mds support 0

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test 0 - add unit test for ao utils

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test 1 - setup utils dir for connection detail and add unit tests

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test 2 - Add uniti test for acc flyout modules 0

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Enable card on connection details for query in discover

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add the mds support for acc flyout details fetch functions

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test 3 - Add unit test for plugin ts flyout setup

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test 4 - Add unit test for some of the acceleration components

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test 5 - clean up the operation unit test a bit

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Implement the acceleration flyout forwarding with mds support

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add discover forwarding action

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test 6 - add more test cases on accelerations comoponents

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* IMPORTANT - fix mds flows for cashloader when mds enabled

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* IMPORTANT - fix mds flows for direct query when mds enabled

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix acc details tab tests

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix everything in mds refresh

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* P00000000 - revert the change of cacheloader and correct the cacheloader and cachemnger usage

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Uncomment security role check

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix unit tests snapshot

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test 7 - add unit tests for acceleration creation main flow

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add the server setup

Co-authored-by: Shenoy Pratik <sgguruda@amazon.com>
Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add the server part of dsm without renaming

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* rename 0 - base has been renamed test yes

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* rename 1 - class has been renamed test yes

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix the test issue

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix the details flyout

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test 8 - add more test for acc creation flow

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Changeset file for PR #7323 created/updated

* Enable the query workbench forwarding

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Resolve conflict router

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test 7 - fix the acceleration creation testing

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test 8 - Add tests for caching components

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Test update

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix the acc creation button in ao tab

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Relocate the framework test dir

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add more tests on framework

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test - use polling

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test - acc table

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test - acc tab

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Rename catlog into catalog

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add more unit tests

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add test for ao table

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix the test for refreshing time

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix the snapshot

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix the timezone issue in test again

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test - add more tests for acc creation related components

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Rename dir - 1

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Rename dir - 2 acceleration_management

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Rename dir - 3 associated_objects_management

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Rename dir - 4 connection_detail

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* fix time zone issue again?

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* integration - local cluster setup without flyout

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* integration - local cluster setup plus flyout

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix the feature flag status undefined issue when the navigation feature is enabled

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix tests and add more error handling

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Integration - add render conditions and fix tests

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Integration - add unit tests p1

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Integration - add unit tests p2

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* test cov

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* test cov

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* test cov +

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* test cov +

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* test cov +

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Resolve comments

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Resolve comments - 2

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Resolve comments - 3

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Resolve comments - 4

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Resolve comments - 5

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Bond the obs check for query card

Signed-off-by: Ryan Liang <jiallian@amazon.com>

---------

Signed-off-by: Ryan Liang <jiallian@amazon.com>
Co-authored-by: Shenoy Pratik <sgguruda@amazon.com>
Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com>

* Fix wrong cherry pick

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Change the integration name in installed integration table from flint s3 into s3 glue

Signed-off-by: Ryan Liang <jiallian@amazon.com>

---------

Signed-off-by: Ryan Liang <jiallian@amazon.com>
Co-authored-by: Shenoy Pratik <sgguruda@amazon.com>
Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com>
opensearch-trigger-bot bot pushed a commit that referenced this pull request Jul 23, 2024
) (#7397)

* [MDS] Data Connection details page with MDS support (#7323)

* Data connection details draft - 0

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Data connection details draft 1 - Add datasource overview section

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add access control tab draft 0 - add the tab of ac

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add access control tab draft 1 - add the ac as a tab

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Cache manager draft 0 - setup for with a testable acceleration table

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Cache manager draft 1 - fix the acceleration name

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Cache manager draft 2 - fix the acceleration status color

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Cache manager draft 3 - Add action menu and action overlay

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Acceleration draft 0 - Implement acceleration actions

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Acceleration draft 1 - Fix the table title

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Acceleration draft 2 - still having issue on the settings api call

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Acceleration draft 3 - fix the fetching apis and still having issue on the notiffication of flyout operation

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Acceleration draft 4 - fix the flyout operation

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix add mds for acceleration flow draft 0 - need to pass mds id into detail page call

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add mds for acceleration flow draft 1 - add feature flag check and pass in id for fetch select datasource

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add mds for acceleration flow draft 2 - add mds flow to all the acceleration table action flow

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add mds for acceleration flow draft 3 - remove unused variable in acceleration table class

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix the route for datasource detail

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add mds for acceleration flow draft 4 - add mds support for acceleration operations table triggered flow

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add mds for acceleration flow draft 5 - add mds support for acceleration operation flyout trigger flow

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add acceleration creation flow draft 0 - directly migrate the entire acc create flow

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add acceleration creation flow draft 1 - relocate the flyout render into plugin ts

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add acceleration creation flow draft 2 - fix the query work bench check

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add acceleration creation flow draft 3 - fix the acceleration creation flow when mds is disabled 0

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add acceleration creation flow draft 4 - unrelated to the creation flow but add the cache hooks for all the meta info

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add acceleration creation flow draft 5 - mds support with acceleration creation flow

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add associated objects flow draft 0 - works without adding mds support

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Remove some comments 0

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add associated objects flow draft 1 - fix the navi to app

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add associated objects flow draft 2 - fix the navi to app and add mds support 0

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test 0 - add unit test for ao utils

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test 1 - setup utils dir for connection detail and add unit tests

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test 2 - Add uniti test for acc flyout modules 0

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Enable card on connection details for query in discover

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add the mds support for acc flyout details fetch functions

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test 3 - Add unit test for plugin ts flyout setup

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test 4 - Add unit test for some of the acceleration components

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test 5 - clean up the operation unit test a bit

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Implement the acceleration flyout forwarding with mds support

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add discover forwarding action

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test 6 - add more test cases on accelerations comoponents

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* IMPORTANT - fix mds flows for cashloader when mds enabled

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* IMPORTANT - fix mds flows for direct query when mds enabled

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix acc details tab tests

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix everything in mds refresh

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* P00000000 - revert the change of cacheloader and correct the cacheloader and cachemnger usage

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Uncomment security role check

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix unit tests snapshot

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test 7 - add unit tests for acceleration creation main flow

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add the server setup

Co-authored-by: Shenoy Pratik <sgguruda@amazon.com>
Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add the server part of dsm without renaming

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* rename 0 - base has been renamed test yes

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* rename 1 - class has been renamed test yes

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix the test issue

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix the details flyout

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test 8 - add more test for acc creation flow

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Changeset file for PR #7323 created/updated

* Enable the query workbench forwarding

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Resolve conflict router

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test 7 - fix the acceleration creation testing

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test 8 - Add tests for caching components

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Test update

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix the acc creation button in ao tab

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Relocate the framework test dir

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add more tests on framework

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test - use polling

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test - acc table

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test - acc tab

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Rename catlog into catalog

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add more unit tests

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add test for ao table

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix the test for refreshing time

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix the snapshot

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix the timezone issue in test again

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* unit test - add more tests for acc creation related components

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Rename dir - 1

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Rename dir - 2 acceleration_management

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Rename dir - 3 associated_objects_management

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Rename dir - 4 connection_detail

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* fix time zone issue again?

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* integration - local cluster setup without flyout

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* integration - local cluster setup plus flyout

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix the feature flag status undefined issue when the navigation feature is enabled

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix tests and add more error handling

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Integration - add render conditions and fix tests

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Integration - add unit tests p1

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Integration - add unit tests p2

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* test cov

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* test cov

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* test cov +

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* test cov +

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* test cov +

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Resolve comments

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Resolve comments - 2

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Resolve comments - 3

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Resolve comments - 4

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Resolve comments - 5

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Bond the obs check for query card

Signed-off-by: Ryan Liang <jiallian@amazon.com>

---------

Signed-off-by: Ryan Liang <jiallian@amazon.com>
Co-authored-by: Shenoy Pratik <sgguruda@amazon.com>
Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com>

* Fix wrong cherry pick

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Change the integration name in installed integration table from flint s3 into s3 glue

Signed-off-by: Ryan Liang <jiallian@amazon.com>

---------

Signed-off-by: Ryan Liang <jiallian@amazon.com>
Co-authored-by: Shenoy Pratik <sgguruda@amazon.com>
Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com>
(cherry picked from commit fac8667)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
mengweieric pushed a commit that referenced this pull request Jul 23, 2024
) (#7397) (#7407)

* [MDS] Data Connection details page with MDS support (#7323)

* Data connection details draft - 0



* Data connection details draft 1 - Add datasource overview section



* Add access control tab draft 0 - add the tab of ac



* Add access control tab draft 1 - add the ac as a tab



* Cache manager draft 0 - setup for with a testable acceleration table



* Cache manager draft 1 - fix the acceleration name



* Cache manager draft 2 - fix the acceleration status color



* Cache manager draft 3 - Add action menu and action overlay



* Acceleration draft 0 - Implement acceleration actions



* Acceleration draft 1 - Fix the table title



* Acceleration draft 2 - still having issue on the settings api call



* Acceleration draft 3 - fix the fetching apis and still having issue on the notiffication of flyout operation



* Acceleration draft 4 - fix the flyout operation



* Fix add mds for acceleration flow draft 0 - need to pass mds id into detail page call



* Add mds for acceleration flow draft 1 - add feature flag check and pass in id for fetch select datasource



* Add mds for acceleration flow draft 2 - add mds flow to all the acceleration table action flow



* Add mds for acceleration flow draft 3 - remove unused variable in acceleration table class



* Fix the route for datasource detail



* Add mds for acceleration flow draft 4 - add mds support for acceleration operations table triggered flow



* Add mds for acceleration flow draft 5 - add mds support for acceleration operation flyout trigger flow



* Add acceleration creation flow draft 0 - directly migrate the entire acc create flow



* Add acceleration creation flow draft 1 - relocate the flyout render into plugin ts



* Add acceleration creation flow draft 2 - fix the query work bench check



* Add acceleration creation flow draft 3 - fix the acceleration creation flow when mds is disabled 0



* Add acceleration creation flow draft 4 - unrelated to the creation flow but add the cache hooks for all the meta info



* Add acceleration creation flow draft 5 - mds support with acceleration creation flow



* Add associated objects flow draft 0 - works without adding mds support



* Remove some comments 0



* Add associated objects flow draft 1 - fix the navi to app



* Add associated objects flow draft 2 - fix the navi to app and add mds support 0



* unit test 0 - add unit test for ao utils



* unit test 1 - setup utils dir for connection detail and add unit tests



* unit test 2 - Add uniti test for acc flyout modules 0



* Enable card on connection details for query in discover



* Add the mds support for acc flyout details fetch functions



* unit test 3 - Add unit test for plugin ts flyout setup



* unit test 4 - Add unit test for some of the acceleration components



* unit test 5 - clean up the operation unit test a bit



* Implement the acceleration flyout forwarding with mds support



* Add discover forwarding action



* unit test 6 - add more test cases on accelerations comoponents



* IMPORTANT - fix mds flows for cashloader when mds enabled



* IMPORTANT - fix mds flows for direct query when mds enabled



* Fix acc details tab tests



* Fix everything in mds refresh



* P00000000 - revert the change of cacheloader and correct the cacheloader and cachemnger usage



* Uncomment security role check



* Fix unit tests snapshot



* unit test 7 - add unit tests for acceleration creation main flow



* Add the server setup




* Add the server part of dsm without renaming



* rename 0 - base has been renamed test yes



* rename 1 - class has been renamed test yes



* Fix the test issue



* Fix the details flyout



* unit test 8 - add more test for acc creation flow



* Changeset file for PR #7323 created/updated

* Enable the query workbench forwarding



* Resolve conflict router



* unit test 7 - fix the acceleration creation testing



* unit test 8 - Add tests for caching components



* Test update



* Fix the acc creation button in ao tab



* Relocate the framework test dir



* Add more tests on framework



* unit test - use polling



* unit test - acc table



* unit test - acc tab



* Rename catlog into catalog



* Add more unit tests



* Add test for ao table



* Fix the test for refreshing time



* Fix the snapshot



* Fix the timezone issue in test again



* unit test - add more tests for acc creation related components



* Rename dir - 1



* Rename dir - 2 acceleration_management



* Rename dir - 3 associated_objects_management



* Rename dir - 4 connection_detail



* fix time zone issue again?



* integration - local cluster setup without flyout



* integration - local cluster setup plus flyout



* Fix the feature flag status undefined issue when the navigation feature is enabled



* Fix tests and add more error handling



* Integration - add render conditions and fix tests



* Integration - add unit tests p1



* Integration - add unit tests p2



* test cov



* test cov



* test cov +



* test cov +



* test cov +



* Resolve comments



* Resolve comments - 2



* Resolve comments - 3



* Resolve comments - 4



* Resolve comments - 5



* Bond the obs check for query card



---------





* Fix wrong cherry pick



* Change the integration name in installed integration table from flint s3 into s3 glue



---------




(cherry picked from commit fac8667)

Signed-off-by: Ryan Liang <jiallian@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Shenoy Pratik <sgguruda@amazon.com>
Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants