[Remote Store] Add rest endpoint for remote store restore #3576

sachinpkale · 2022-06-14T03:12:04Z

Signed-off-by: Sachin Kale kalsac@amazon.com

Description

This change adds rest API endpoint to restore data from remote store. It only provides skeleton of the rest endpoint along with dummy implementation of restore logic. Actual implementation of the restore will come in the subsequent PRs.

Request

POST /_remotestore/_restore

{
    "indices": [
        "my-index-1",
        "my-index-2"
    ]
}

Issues Resolved

[Remote Store] API support to restore data from remote store #3145

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

opensearch-ci-bot · 2022-06-14T03:44:16Z

❌ Gradle Check failure 8c199ea657c2f4df69ed339bab6e8ee39bee9e2d
Log 5960

Reports 5960

opensearch-ci-bot · 2022-06-14T03:44:29Z

❌ Gradle Check failure 04b975379dc0d644f138d94f076271ce7f16afd9
Log 5961

Reports 5961

satyajitg28 · 2022-06-14T04:31:04Z

.../java/org/opensearch/action/admin/cluster/remote_store/restore/RestoreRemoteStoreAction.java

+ * compatible open source license.
+ */
+
+package org.opensearch.action.admin.cluster.remote_store.restore;


Will remotestore be a better naming convention for the package compared to remote_store ?

Agree, will get it changed.

opensearch-ci-bot · 2022-06-14T04:47:55Z

❌ Gradle Check failure 9883317bb5fc403fddc7641aff169079805a4ff1
Log 5965

Reports 5965

sachinpkale · 2022-06-14T04:55:37Z

Test failures. Converting to draft for now.

opensearch-ci-bot · 2022-06-14T09:46:46Z

❌ Gradle Check failure e5f53ff8c1ba43041b4a544a57f50cf032dfd135
Log 5974

Reports 5974

opensearch-ci-bot · 2022-06-14T10:33:42Z

❌ Gradle Check failure 2618e2af785085fe0b7204857b7caafa94878b21
Log 5976

Reports 5976

sachinpkale · 2022-06-14T12:27:55Z

Moving to draft until I fix all the tests.

opensearch-ci-bot · 2022-06-14T14:51:13Z

❌ Gradle Check failure d2d815d8f7ad293e606c69acfa5cb1620f93c3ac
Log 5983

Reports 5983

sachinpkale · 2022-06-15T02:59:18Z

client/rest-high-level/src/test/java/org/opensearch/client/RestHighLevelClientTests.java

@@ -885,7 +885,8 @@ public void testApiNamingConventions() throws Exception {
            "nodes.hot_threads",
            "nodes.usage",
            "nodes.reload_secure_settings",
-            "search_shards", };
+            "search_shards",
+            "remote_store.restore", };


New API is added as high level rest client is not added for _remotestore APIs. I am not sure about guideline on writing high level rest client for new endpoints. If it is required, will add a tracking issue to add the same.

I am guessing that will be needed since it's a new API. And also https://github.com/opensearch-project/opensearch-java and https://github.com/opensearch-project/opensearch-api-specification. I'd appreciate a write up on "adding new RESTful APIs" somewhere in the developer guide, and all the subsequent changes that are required today for the next person.

Tracking issues: #3602 and opensearch-project/opensearch-java#168

opensearch-ci-bot · 2022-06-16T08:24:59Z

❌ Gradle Check failure 9334ca368b16f44963fe5977d051c6efca15aa66
Log 6055

Reports 6055

opensearch-ci-bot · 2022-06-16T08:44:16Z

❌ Gradle Check failure 4cda8e2
Log 6056

Reports 6056

opensearch-ci-bot · 2022-06-16T10:06:21Z

❌ Gradle Check failure 37d07a0
Log 6057

Reports 6057

sachinpkale · 2022-06-16T12:28:29Z

Following 3 tests are failing:

./gradlew ':server:internalClusterTest' --tests "org.opensearch.cluster.routing.PrimaryAllocationIT.testPrimaryReplicaResyncFailed" -Dtests.seed=B09620CBB34D45F7 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=en-PH -Dtests.timezone=W-SU -Druntime.java=17
./gradlew ':server:internalClusterTest' --tests "org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testHighWatermarkNotExceeded" -Dtests.seed=B09620CBB34D45F7 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=es-BO -Dtests.timezone=Indian/Cocos -Druntime.java=17
./gradlew ':server:internalClusterTest' --tests "org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testRestoreSnapshotAllocationDoesNotExceedWatermark" -Dtests.seed=B09620CBB34D45F7 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=es-BO -Dtests.timezone=Indian/Cocos -Druntime.java=17

I was able to run them successfully on local.

Bukhtawar · 2022-06-16T12:29:53Z

start gradle check

opensearch-ci-bot · 2022-06-16T13:00:05Z

❌ Gradle Check failure 37d07a0
Log 6059

Reports 6059

Signed-off-by: Sachin Kale <kalsac@amazon.com>

…Response Signed-off-by: Sachin Kale <kalsac@amazon.com>

github-actions · 2022-08-05T02:00:01Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/1473/
CommitID: 5288f3e

…-project#3576) * Add rest endpoint for remote store restore Signed-off-by: Sachin Kale <kalsac@amazon.com>

…4380) * [Remote Store] Upload segments to remote store post refresh (#3460) * Add RemoteDirectory interface to copy segment files to/from remote store Signed-off-by: Sachin Kale <kalsac@amazon.com> Co-authored-by: Sachin Kale <kalsac@amazon.com> * Add index level setting for remote store Signed-off-by: Sachin Kale <kalsac@amazon.com> Co-authored-by: Sachin Kale <kalsac@amazon.com> * Add RemoteDirectoryFactory and use RemoteDirectory instance in RefreshListener Co-authored-by: Sachin Kale <kalsac@amazon.com> Signed-off-by: Sachin Kale <kalsac@amazon.com> * Upload segment to remote store post refresh Signed-off-by: Sachin Kale <kalsac@amazon.com> Co-authored-by: Sachin Kale <kalsac@amazon.com> Signed-off-by: Sachin Kale <kalsac@amazon.com> * [Remote Store] Inject remote store in IndexShard instead of RemoteStoreRefreshListener (#3703) * Inject remote store in IndexShard instead of RemoteStoreRefreshListener Signed-off-by: Sachin Kale <kalsac@amazon.com> * Pass supplier of RepositoriesService to RemoteDirectoryFactory Signed-off-by: Sachin Kale <kalsac@amazon.com> * Create isRemoteStoreEnabled function for IndexShard Signed-off-by: Sachin Kale <kalsac@amazon.com> * Explicitly close remoteStore on indexShard close Signed-off-by: Sachin Kale <kalsac@amazon.com> * Change RemoteDirectory.close to a no-op Signed-off-by: Sachin Kale <kalsac@amazon.com> Co-authored-by: Sachin Kale <kalsac@amazon.com> * [Remote Store] Add remote store restore API implementation (#3642) * Add remote restore API implementation Signed-off-by: Sachin Kale <kalsac@amazon.com> * [Remote Store] Add support to add nested settings for remote store (#4060) * Add support to add nested settings for remote store Signed-off-by: Sachin Kale <kalsac@amazon.com> * [Remote Store] Add rest endpoint for remote store restore (#3576) * Add rest endpoint for remote store restore Signed-off-by: Sachin Kale <kalsac@amazon.com> * [Remote Store] Add validator that forces segment replication type before enabling remote store (#4175) * Add validator that forces segment replication type before enabling remote store Signed-off-by: Sachin Kale <kalsac@amazon.com> * [Remote Store] Change remote_store setting validation message to make it more clear (#4199) * Change remote_store setting validation message to make it more clear Signed-off-by: Sachin Kale <kalsac@amazon.com> * [Remote Store] Add RemoteSegmentStoreDirectory to interact with remote segment store (#4020) * Add RemoteSegmentStoreDirectory to interact with remote segment store Signed-off-by: Sachin Kale <kalsac@amazon.com> * Use RemoteSegmentStoreDirectory instead of RemoteDirectory (#4240) * Use RemoteSegmentStoreDirectory instead of RemoteDirectory Signed-off-by: Sachin Kale <kalsac@amazon.com> Signed-off-by: Sachin Kale <kalsac@amazon.com> Co-authored-by: Sachin Kale <kalsac@amazon.com>

@sachinpkale

I have nominated and maintainers have agreed to invite Sachin Kale (@sachinpkale) to be a co-maintainer. Sachin has kindly accepted. Sachin has led the design and implementation of the remote backed storage feature in OpenSearch. This feature was introduced as experimental in OpenSearch 2.3 and is planned for general availability in 2.9. Some significant issues and PRs authored by Sachin for this effort are as follows: Feature proposal: opensearch-project#1968 Upload segments to remote store post refresh: opensearch-project#3460 Add rest endpoint for remote store restore: opensearch-project#3576 Add RemoteSegmentStoreDirectory to interact with remote segment store: opensearch-project#4020 In total, Sachin has authored 57 PRs going back to May 2022. He also frequently reviews contributions from others and has reviewed nearly 100 PRs in the same time frame. Signed-off-by: Andrew Ross <andrross@amazon.com>

@sachinpkale

I have nominated and maintainers have agreed to invite Sachin Kale (@sachinpkale) to be a co-maintainer. Sachin has kindly accepted. Sachin has led the design and implementation of the remote backed storage feature in OpenSearch. This feature was introduced as experimental in OpenSearch 2.3 and is planned for general availability in 2.9. Some significant issues and PRs authored by Sachin for this effort are as follows: Feature proposal: opensearch-project#1968 Upload segments to remote store post refresh: opensearch-project#3460 Add rest endpoint for remote store restore: opensearch-project#3576 Add RemoteSegmentStoreDirectory to interact with remote segment store: opensearch-project#4020 In total, Sachin has authored 57 PRs going back to May 2022. He also frequently reviews contributions from others and has reviewed nearly 100 PRs in the same time frame. Signed-off-by: Andrew Ross <andrross@amazon.com>

@sachinpkale

I have nominated and maintainers have agreed to invite Sachin Kale (@sachinpkale) to be a co-maintainer. Sachin has kindly accepted. Sachin has led the design and implementation of the remote backed storage feature in OpenSearch. This feature was introduced as experimental in OpenSearch 2.3 and is planned for general availability in 2.9. Some significant issues and PRs authored by Sachin for this effort are as follows: Feature proposal: opensearch-project#1968 Upload segments to remote store post refresh: opensearch-project#3460 Add rest endpoint for remote store restore: opensearch-project#3576 Add RemoteSegmentStoreDirectory to interact with remote segment store: opensearch-project#4020 In total, Sachin has authored 57 PRs going back to May 2022. He also frequently reviews contributions from others and has reviewed nearly 100 PRs in the same time frame. Signed-off-by: Andrew Ross <andrross@amazon.com>

@sachinpkale

I have nominated and maintainers have agreed to invite Sachin Kale (@sachinpkale) to be a co-maintainer. Sachin has kindly accepted. Sachin has led the design and implementation of the remote backed storage feature in OpenSearch. This feature was introduced as experimental in OpenSearch 2.3 and is planned for general availability in 2.9. Some significant issues and PRs authored by Sachin for this effort are as follows: Feature proposal: #1968 Upload segments to remote store post refresh: #3460 Add rest endpoint for remote store restore: #3576 Add RemoteSegmentStoreDirectory to interact with remote segment store: #4020 In total, Sachin has authored 57 PRs going back to May 2022. He also frequently reviews contributions from others and has reviewed nearly 100 PRs in the same time frame. Signed-off-by: Andrew Ross <andrross@amazon.com>

@sachinpkale

I have nominated and maintainers have agreed to invite Sachin Kale (@sachinpkale) to be a co-maintainer. Sachin has kindly accepted. Sachin has led the design and implementation of the remote backed storage feature in OpenSearch. This feature was introduced as experimental in OpenSearch 2.3 and is planned for general availability in 2.9. Some significant issues and PRs authored by Sachin for this effort are as follows: Feature proposal: opensearch-project#1968 Upload segments to remote store post refresh: opensearch-project#3460 Add rest endpoint for remote store restore: opensearch-project#3576 Add RemoteSegmentStoreDirectory to interact with remote segment store: opensearch-project#4020 In total, Sachin has authored 57 PRs going back to May 2022. He also frequently reviews contributions from others and has reviewed nearly 100 PRs in the same time frame. Signed-off-by: Andrew Ross <andrross@amazon.com>

@sachinpkale

I have nominated and maintainers have agreed to invite Sachin Kale (@sachinpkale) to be a co-maintainer. Sachin has kindly accepted. Sachin has led the design and implementation of the remote backed storage feature in OpenSearch. This feature was introduced as experimental in OpenSearch 2.3 and is planned for general availability in 2.9. Some significant issues and PRs authored by Sachin for this effort are as follows: Feature proposal: #1968 Upload segments to remote store post refresh: #3460 Add rest endpoint for remote store restore: #3576 Add RemoteSegmentStoreDirectory to interact with remote segment store: #4020 In total, Sachin has authored 57 PRs going back to May 2022. He also frequently reviews contributions from others and has reviewed nearly 100 PRs in the same time frame. Signed-off-by: Andrew Ross <andrross@amazon.com>

@sachinpkale

opensearch-project#8047) I have nominated and maintainers have agreed to invite Sachin Kale (@sachinpkale) to be a co-maintainer. Sachin has kindly accepted. Sachin has led the design and implementation of the remote backed storage feature in OpenSearch. This feature was introduced as experimental in OpenSearch 2.3 and is planned for general availability in 2.9. Some significant issues and PRs authored by Sachin for this effort are as follows: Feature proposal: opensearch-project#1968 Upload segments to remote store post refresh: opensearch-project#3460 Add rest endpoint for remote store restore: opensearch-project#3576 Add RemoteSegmentStoreDirectory to interact with remote segment store: opensearch-project#4020 In total, Sachin has authored 57 PRs going back to May 2022. He also frequently reviews contributions from others and has reviewed nearly 100 PRs in the same time frame. Signed-off-by: Andrew Ross <andrross@amazon.com>

@sachinpkale

I have nominated and maintainers have agreed to invite Sachin Kale (@sachinpkale) to be a co-maintainer. Sachin has kindly accepted. Sachin has led the design and implementation of the remote backed storage feature in OpenSearch. This feature was introduced as experimental in OpenSearch 2.3 and is planned for general availability in 2.9. Some significant issues and PRs authored by Sachin for this effort are as follows: Feature proposal: opensearch-project#1968 Upload segments to remote store post refresh: opensearch-project#3460 Add rest endpoint for remote store restore: opensearch-project#3576 Add RemoteSegmentStoreDirectory to interact with remote segment store: opensearch-project#4020 In total, Sachin has authored 57 PRs going back to May 2022. He also frequently reviews contributions from others and has reviewed nearly 100 PRs in the same time frame. Signed-off-by: Andrew Ross <andrross@amazon.com> Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

@sachinpkale

I have nominated and maintainers have agreed to invite Sachin Kale (@sachinpkale) to be a co-maintainer. Sachin has kindly accepted. Sachin has led the design and implementation of the remote backed storage feature in OpenSearch. This feature was introduced as experimental in OpenSearch 2.3 and is planned for general availability in 2.9. Some significant issues and PRs authored by Sachin for this effort are as follows: Feature proposal: opensearch-project#1968 Upload segments to remote store post refresh: opensearch-project#3460 Add rest endpoint for remote store restore: opensearch-project#3576 Add RemoteSegmentStoreDirectory to interact with remote segment store: opensearch-project#4020 In total, Sachin has authored 57 PRs going back to May 2022. He also frequently reviews contributions from others and has reviewed nearly 100 PRs in the same time frame. Signed-off-by: Andrew Ross <andrross@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

sachinpkale requested review from a team and reta as code owners June 14, 2022 03:12

sachinpkale force-pushed the main branch from 8c199ea to 04b9753 Compare June 14, 2022 03:24

sachinpkale mentioned this pull request Jun 14, 2022

[Remote Store] API support to restore data from remote store #3145

Closed

sachinpkale force-pushed the main branch from 04b9753 to 9883317 Compare June 14, 2022 04:24

satyajitg28 reviewed Jun 14, 2022

View reviewed changes

sachinpkale marked this pull request as draft June 14, 2022 04:55

sachinpkale marked this pull request as ready for review June 14, 2022 08:55

sachinpkale marked this pull request as draft June 14, 2022 12:27

sachinpkale marked this pull request as ready for review June 15, 2022 02:10

sachinpkale commented Jun 15, 2022

View reviewed changes

This was referenced Jun 16, 2022

[Remote Store] Add wrapper for RestHighLevelClient that provides methods for accessing the Remote Store API. #3602

Open

[FEATURE] Add java client support for remote store APIs opensearch-project/opensearch-java#168

Open

sachinpkale force-pushed the main branch 3 times, most recently from 9334ca3 to 4cda8e2 Compare June 16, 2022 08:12

Sachin Kale added 4 commits August 5, 2022 00:05

Remove deprecated parameter from new API

82cc525

Signed-off-by: Sachin Kale <kalsac@amazon.com>

Add _remotestore/_restore rest handle registration behind feature flag

1d68da8

Signed-off-by: Sachin Kale <kalsac@amazon.com>

Make new classes final

dc13f01

Signed-off-by: Sachin Kale <kalsac@amazon.com>

Use RestoreClusterStateListener with generic type that extends Action…

5288f3e

…Response Signed-off-by: Sachin Kale <kalsac@amazon.com>

sachinpkale force-pushed the main branch from d6fa757 to 5288f3e Compare August 5, 2022 01:34

Bukhtawar approved these changes Aug 5, 2022

View reviewed changes

Bukhtawar merged commit 42058aa into opensearch-project:main Aug 5, 2022

dreamer-89 pushed a commit to dreamer-89/OpenSearch that referenced this pull request Aug 12, 2022

[Remote Store] Add rest endpoint for remote store restore (opensearch…

7d3f79e

…-project#3576) * Add rest endpoint for remote store restore Signed-off-by: Sachin Kale <kalsac@amazon.com>

sachinpkale added a commit to sachinpkale/OpenSearch that referenced this pull request Sep 1, 2022

[Remote Store] Add rest endpoint for remote store restore (opensearch…

d5c4878

…-project#3576) * Add rest endpoint for remote store restore Signed-off-by: Sachin Kale <kalsac@amazon.com>

sachinpkale mentioned this pull request Sep 1, 2022

[Backport 2.x] [Remote Store] Backport remote segment store changes #4380

Merged

6 tasks

sachinpkale added a commit to sachinpkale/OpenSearch that referenced this pull request Sep 2, 2022

[Remote Store] Add rest endpoint for remote store restore (opensearch…

e051d48

…-project#3576) * Add rest endpoint for remote store restore Signed-off-by: Sachin Kale <kalsac@amazon.com>

sachinpkale added a commit to sachinpkale/OpenSearch that referenced this pull request Sep 2, 2022

[Remote Store] Add rest endpoint for remote store restore (opensearch…

3d50a74

…-project#3576) * Add rest endpoint for remote store restore Signed-off-by: Sachin Kale <kalsac@amazon.com>

wbeckler mentioned this pull request Nov 30, 2022

Defining "Experimental"/"Feature Flags"/? opensearch-project/.github#108

Open

andrross mentioned this pull request Jun 12, 2023

Add @sachinpkale as an OpenSearch maintainer #8019

Merged

6 tasks

andrross mentioned this pull request Jun 13, 2023

[Backport 2.x] Add @sachinpkale as an OpenSearch maintainer (#8019) #8047

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Remote Store] Add rest endpoint for remote store restore #3576

[Remote Store] Add rest endpoint for remote store restore #3576

sachinpkale commented Jun 14, 2022 •

edited

Loading

opensearch-ci-bot commented Jun 14, 2022

opensearch-ci-bot commented Jun 14, 2022

satyajitg28 Jun 14, 2022

sachinpkale Jun 14, 2022

opensearch-ci-bot commented Jun 14, 2022

sachinpkale commented Jun 14, 2022

opensearch-ci-bot commented Jun 14, 2022

opensearch-ci-bot commented Jun 14, 2022

sachinpkale commented Jun 14, 2022

opensearch-ci-bot commented Jun 14, 2022

sachinpkale Jun 15, 2022

dblock Jun 15, 2022

sachinpkale Jun 16, 2022

opensearch-ci-bot commented Jun 16, 2022

opensearch-ci-bot commented Jun 16, 2022

opensearch-ci-bot commented Jun 16, 2022

sachinpkale commented Jun 16, 2022

Bukhtawar commented Jun 16, 2022

opensearch-ci-bot commented Jun 16, 2022

github-actions bot commented Aug 5, 2022

[Remote Store] Add rest endpoint for remote store restore #3576

[Remote Store] Add rest endpoint for remote store restore #3576

Conversation

sachinpkale commented Jun 14, 2022 • edited Loading

Description

Request

Issues Resolved

Check List

opensearch-ci-bot commented Jun 14, 2022

opensearch-ci-bot commented Jun 14, 2022

satyajitg28 Jun 14, 2022

Choose a reason for hiding this comment

sachinpkale Jun 14, 2022

Choose a reason for hiding this comment

opensearch-ci-bot commented Jun 14, 2022

sachinpkale commented Jun 14, 2022

opensearch-ci-bot commented Jun 14, 2022

opensearch-ci-bot commented Jun 14, 2022

sachinpkale commented Jun 14, 2022

opensearch-ci-bot commented Jun 14, 2022

sachinpkale Jun 15, 2022

Choose a reason for hiding this comment

dblock Jun 15, 2022

Choose a reason for hiding this comment

sachinpkale Jun 16, 2022

Choose a reason for hiding this comment

opensearch-ci-bot commented Jun 16, 2022

opensearch-ci-bot commented Jun 16, 2022

opensearch-ci-bot commented Jun 16, 2022

sachinpkale commented Jun 16, 2022

Bukhtawar commented Jun 16, 2022

opensearch-ci-bot commented Jun 16, 2022

github-actions bot commented Aug 5, 2022

Gradle Check (Jenkins) Run Completed with:

sachinpkale commented Jun 14, 2022 •

edited

Loading