Update integrations to allow custom checkpoint locations #1501

Swiddis · 2024-03-08T17:38:26Z

Description

Per #1468, integrations has an improperly hardcoded checkpoint location. This PR adds the ability for users to input their own location.

Before:

After:

Issues Resolved

Closes #1468

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

…-install

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

codecov · 2024-03-08T17:51:27Z

Codecov Report

Attention: Patch coverage is 21.27660% with 37 lines in your changes are missing coverage. Please review.

Project coverage is 56.64%. Comparing base (4fdd4e0) to head (5736312).

Files	Patch %	Lines
...ents/integrations/components/setup_integration.tsx	21.27%	37 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1501      +/-   ##
==========================================
+ Coverage   56.58%   56.64%   +0.05%     
==========================================
  Files         331      331              
  Lines       12138    12145       +7     
  Branches     3119     3116       -3     
==========================================
+ Hits         6868     6879      +11     
+ Misses       5216     5213       -3     
+ Partials       54       53       -1

Flag	Coverage Δ
dashboards-observability	`56.64% <21.27%> (+0.05%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

ps48 · 2024-03-09T17:53:22Z

public/components/integrations/components/setup_integration.tsx

            error={["Must be a URL starting with 's3://'."]}
          >
            <EuiFieldText
              value={config.connectionLocation}
              onChange={(event) => updateConfig({ connectionLocation: event.target.value })}
              placeholder="s3://"
-              isInvalid={isBlurred && !config.connectionLocation.startsWith('s3://')}
+              isInvalid={isBucketBlurred && !config.connectionLocation.startsWith('s3://')}


do we need to do a Regex check for location and bucket checkpoint instead of a prefix check?

dashboards-observability/common/constants/data_sources.ts

Line 39 in cea2e85

export const ACCELERATION_S3_URL_REGEX = /^(s3|s3a):\/\/[a-zA-Z0-9.\-]+/;

Might be overkill? I think UI-wise we already communicate that we need a URL and the protocol check is enough that a user would have to be trying to break it. Not sure if we should try and play the "Write a URL validator" game ourselves unless we have a library specifically for it.

* Update integration type for new format Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Update integration type validator for new type Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Fix type errors in integration_reader Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Fix return types in adaptor Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Update integrations constants Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Fix local repository and local repository tests Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Fix remaining integrations unit tests Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Add integration format migration Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Fix frontend usage of integration assets Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Update integration install query iteration Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Add checkpoint field to setup form Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Update checkpoint text based on UI feedback Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Add new checkpoint usage to setup flow Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Split out integrations setup methods Signed-off-by: Simeon Widdis <sawiddis@amazon.com> --------- Signed-off-by: Simeon Widdis <sawiddis@amazon.com> (cherry picked from commit dec4ff2) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Update integration type for new format * Update integration type validator for new type * Fix type errors in integration_reader * Fix return types in adaptor * Update integrations constants * Fix local repository and local repository tests * Fix remaining integrations unit tests * Add integration format migration * Fix frontend usage of integration assets * Update integration install query iteration * Add checkpoint field to setup form * Update checkpoint text based on UI feedback * Add new checkpoint usage to setup flow * Split out integrations setup methods --------- (cherry picked from commit dec4ff2) Signed-off-by: Simeon Widdis <sawiddis@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…project#1501) (opensearch-project#1511) * Update integration type for new format * Update integration type validator for new type * Fix type errors in integration_reader * Fix return types in adaptor * Update integrations constants * Fix local repository and local repository tests * Fix remaining integrations unit tests * Add integration format migration * Fix frontend usage of integration assets * Update integration install query iteration * Add checkpoint field to setup form * Update checkpoint text based on UI feedback * Add new checkpoint usage to setup flow * Split out integrations setup methods --------- (cherry picked from commit dec4ff2) Signed-off-by: Simeon Widdis <sawiddis@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> (cherry picked from commit 3191614)

Swiddis added 14 commits March 6, 2024 14:40

Update integration type for new format

9516abd

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Update integration type validator for new type

c7ae842

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Fix type errors in integration_reader

d54f9f6

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Fix return types in adaptor

0678304

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Update integrations constants

5efccef

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Fix local repository and local repository tests

5d741ad

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Fix remaining integrations unit tests

338cf3d

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Add integration format migration

b9f025e

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Merge remote-tracking branch 'upstream/main' into feature/conditional…

3bbd6b6

…-install

Fix frontend usage of integration assets

82d3eab

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Update integration install query iteration

fc22b95

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Add checkpoint field to setup form

176ca4a

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Update checkpoint text based on UI feedback

7068477

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Add new checkpoint usage to setup flow

f29009d

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Swiddis added bug Something isn't working integrations Used to denote items related to the Integrations project backport 2.x labels Mar 8, 2024

Swiddis added 2 commits March 8, 2024 10:17

Merge remote-tracking branch 'upstream/main' into feature/checkpoint

86b9fca

Split out integrations setup methods

ae4c5c7

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Swiddis force-pushed the feature/checkpoint branch from ce784cc to ae4c5c7 Compare March 8, 2024 18:38

Swiddis marked this pull request as ready for review March 8, 2024 18:43

Swiddis requested review from ps48, kavithacm, derek-ho, joshuali925, dai-chen, YANG-DB, rupal-bq and mengweieric as code owners March 8, 2024 18:43

Swiddis requested review from vamsimanohar, penghuo, seankao-az, anirudha, paulstn and sumukhswamy as code owners March 8, 2024 18:43

Merge branch 'main' into feature/checkpoint

01d984d

ps48 reviewed Mar 9, 2024

View reviewed changes

ps48 approved these changes Mar 9, 2024

View reviewed changes

seankao-az approved these changes Mar 9, 2024

View reviewed changes

Merge branch 'main' into feature/checkpoint

5736312

Swiddis merged commit dec4ff2 into opensearch-project:main Mar 9, 2024
15 of 20 checks passed

Swiddis deleted the feature/checkpoint branch March 9, 2024 19:53

opensearch-trigger-bot bot mentioned this pull request Mar 9, 2024

[Backport 2.x] Update integrations to allow custom checkpoint locations #1511

Merged

Swiddis mentioned this pull request Mar 11, 2024

[FEATURE] Copy applicable labels from original PR when backporting. VachaShah/backport#12

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update integrations to allow custom checkpoint locations #1501

Update integrations to allow custom checkpoint locations #1501

Swiddis commented Mar 8, 2024 •

edited

Loading

codecov bot commented Mar 8, 2024 •

edited

Loading

ps48 Mar 9, 2024

Swiddis Mar 9, 2024 •

edited

Loading

Update integrations to allow custom checkpoint locations #1501

Update integrations to allow custom checkpoint locations #1501

Conversation

Swiddis commented Mar 8, 2024 • edited Loading

Description

Issues Resolved

Check List

codecov bot commented Mar 8, 2024 • edited Loading

Codecov Report

ps48 Mar 9, 2024

Choose a reason for hiding this comment

Swiddis Mar 9, 2024 • edited Loading

Choose a reason for hiding this comment

Swiddis commented Mar 8, 2024 •

edited

Loading

codecov bot commented Mar 8, 2024 •

edited

Loading

Swiddis Mar 9, 2024 •

edited

Loading