[Remote Store] Add RemoteSegmentStoreDirectory to interact with remote segment store #4020

sachinpkale · 2022-07-27T04:17:30Z

Signed-off-by: Sachin Kale kalsac@amazon.com

Description

To avoid concurrency issues where two primary exists for a given shard at the same time and upload segment files which can overwrite each other, we decided to add a unique suffix (UUID) to each segment filename that is uploaded to remote segment store.
But when we restore these segment files, we need restore them as per their original name. Also, need a mechanism to understand which segment file is a part of particular commit checkpoint. For this, we also upload a metadata file per checkpoint (refresh/commit). This metadata file contains map of original segment name to uploaded segment name.
The above logic is implemented in RemoteSegmentStoreDirectory which composes two instances of remote directories (one for segment and another for metadata) and still provides a directory interface for a caller. This way, caller would invoke directory methods of RemoteSegmentStoreDirectory in the same way as FSDirectory.
Two instances of RemoteDirectory that are part of RemoteSegmentStoreDirectory:
- remoteDataDirectory: <Cluster UUID>/<Index UUID>/<Shard ID>/segments/data
- remoteMetadataDirectory: <Cluster UUID>/<Index UUID>/<Shard ID>/segments/metadata
Sample Files under each directory path:
- remoteMetadataDirectory
  - refresh_mapping__1__z__lKDiNIIBrs0AUNsRcOa3
  - commit_mapping__1__z__lKDiNIIBrs0AUNsRcOa3
  - refresh_mapping__1__y__g6DeNIIBrs0AUNsRQubk
  - commit_mapping__1__y__g6DeNIIBrs0AUNsRQubk
- remoteDataDirectory
  - _10v.cfe__yXvPNIIBrs0AUNsRUXfQ
  - _10v.cfs__uHvPNIIBrs0AUNsRUVA2
  - _10v.si__h3vPNIIBrs0AUNsRSgJ-
  - _10w.cfe__UnrPNIIBrs0AUNsRR43c
  - _10w.cfs__ZXrPNIIBrs0AUNsRSbQU
  - _10w.si__KnrPNIIBrs0AUNsRRD-K

Issues Resolved

[Remote Segment Store] Failure handling #3906

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2022-07-27T04:22:15Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/1025/
CommitID: 26025178955f231df71b8be27ae7a3a7362d20ea

github-actions · 2022-07-27T04:35:27Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/1026/
CommitID: 505acf9e05e552aac6948a62a061ad22fae7e386

github-actions · 2022-07-27T06:53:34Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/1034/
CommitID: fc25eaf72d6d06edcb47b70d9e2fcf380dc78bf5

sachinpkale · 2022-07-28T08:57:46Z

@andrross @Bukhtawar Please review.

github-actions · 2022-07-28T09:22:59Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/1064/
CommitID: 6e053b4

codecov-commenter · 2022-07-28T09:23:57Z

Codecov Report

Merging #4020 (2470f4a) into main (a469a3c) will increase coverage by 0.04%.
The diff coverage is 86.59%.

@@             Coverage Diff              @@
##               main    #4020      +/-   ##
============================================
+ Coverage     70.59%   70.64%   +0.04%     
- Complexity    57083    57118      +35     
============================================
  Files          4603     4605       +2     
  Lines        274551   274670     +119     
  Branches      40210    40223      +13     
============================================
+ Hits         193831   194037     +206     
+ Misses        64514    64378     -136     
- Partials      16206    16255      +49

Impacted Files	Coverage Δ
...ava/org/opensearch/client/RestHighLevelClient.java	`44.32% <ø> (-0.16%)`	⬇️
...gregations/metrics/GeoBoundsAggregatorFactory.java	`88.88% <ø> (ø)`
...search/aggregations/metrics/InternalGeoBounds.java	`66.66% <ø> (ø)`
...o/search/aggregations/metrics/ParsedGeoBounds.java	`88.00% <ø> (ø)`
.../main/java/org/opensearch/search/SearchModule.java	`96.27% <ø> (-0.03%)`	⬇️
...earch/search/aggregations/AggregationBuilders.java	`46.15% <ø> (+1.15%)`	⬆️
...regations/support/AggregationInspectionHelper.java	`53.65% <ø> (+1.27%)`	⬆️
...g/opensearch/test/InternalAggregationTestCase.java	`98.21% <ø> (-0.46%)`	⬇️
...a/org/opensearch/test/OpenSearchIntegTestCase.java	`57.37% <ø> (-0.11%)`	⬇️
...regations/metrics/AbstractGeoBoundsAggregator.java	`56.52% <56.52%> (ø)`
... and 477 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Bukhtawar · 2022-07-29T06:19:20Z

/cc : @ashking94 for help with review

ashking94 · 2022-07-29T06:50:34Z

server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java

+    private final RemoteDirectory remoteMetadataDirectory;
+
+    /**
+     * To prevent explosion of refresh metadata files, we replace refresh files for the given primary term and generation


Do you mean collision here?

No. Idea is to keep count of metadata files in check as we will be calling listAll on metadataDirectory in order to get the list of segments.
Each commit will be followed by many refreshes. We don't want to add one metadata file per refresh. We will just replace the previous one. This works as we still have unique filenames for commit and scales as commits are not as frequent as refreshes.

ashking94 · 2022-07-29T06:51:11Z

server/src/main/java/org/opensearch/index/store/RemoteDirectory.java

+     * Returns names of files with given prefix in this directory.
+     * @param filenamePrefix The prefix to match against file names in the directory
+     * @return A list of the matching filenames in the directory
+     * @throws IOException IOException if there were any failures in reading from the blob container


one IOException can be removed from java doc?

ashking94 · 2022-07-29T06:54:45Z

server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java

+        private final String checksum;
+
+        UploadedSegmentMetadata(String originalFilename, String uploadedFilename, String checksum) {
+            this.originalFilename = originalFilename;


Should we add assertions here to be sure that these are valid values?

UploadedSegmentMetadata is a POJO which contains 3 strings. IMO, It doesn't/shouldn't do any validation.

ashking94 · 2022-07-29T06:56:42Z

server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java

+
+        public static UploadedSegmentMetadata fromString(String uploadedFilename) {
+            String[] values = uploadedFilename.split(SEPARATOR);
+            return new UploadedSegmentMetadata(values[0], values[1], values[2]);


Should we handle exception use cases? Ideally a user should not be putting any other file than what the Opensearch remote store is putting in the concerned prefix. However, is there any behaviour if we see dubious files?

Same as above comment. UploadedSegmentMetadata should not validate this. The caller, on the other hand, should.

ashking94 · 2022-07-29T07:02:01Z

server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java

+ * another instance of {@code RemoteDirectory}.
+ * @opensearch.internal
+ */
+public final class RemoteSegmentStoreDirectory extends FilterDirectory {


Given there are 2 directories where the calls can be delegated, what value addition does FilterDirectory provides us here? Could we have extended Directory directly?

Extending directory would require most of the abstract methods to be re-implemented where we will be delegating it mostly to the data directory.

If you check RemoteDirectory class, most of the methods are throwing unsupported exception. Once we have the implementation for those methods and we think some pre-processing is required before delegating, we need to implement them in this class as well.

Quoting from FilterDirectory javadoc:

if you plan to write your own Directory implementation, you should consider extending directly Directory or BaseDirectory rather than try to reuse functionality of existing Directorys by extending this class.

Based on this, we should extend FilterDirectory only if we plan to use the Directory implementation. But in this case, we're not really using Directory implementation. Again, any benefit we're getting here by extending FilterDirectory?

By extending FilterDirectory, we don't have to implement all the methods that are present in the Directory. We delegate most of the methods directly to remoteDataDirectory. For the methods that need pre/post processing, they are implemented in this class. The first line of javadoc of FilterDirectory says:

Directory implementation that delegates calls to another directory.

This is exactly what we want.

ashking94 · 2022-07-29T07:42:22Z

server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java

+        if (latestRefreshMetadataFile.isPresent()) {
+            logger.info("Reading latest refresh metadata file {}", latestRefreshMetadataFile.get());
+            segmentMetadataMap.putAll(readMetadataFile(latestRefreshMetadataFile.get()));
+            this.refreshMetadataFileUniqueSuffix = MetadataFilenameUtils.getUuid(latestRefreshMetadataFile.get());


This method from the looks is only supposed to return the value that is eventually set to a class variable. Is it possible to set refreshMetadataFileUniqueSuffix outside this block and one level above as the segmentsUploadedToRemoteStore variable.

Also, the name of the method is readXXX. May be we can rename to something like readXXXAndUpdateYYY.

Agree, this method should not be assigning refreshMetadataFileUniqueSuffix, will get this changed. But the goal of this method is to read only.

In what use case would we need to update refreshMetadataFileUniqueSuffix as part for this method?

This was changed as part of latest commit.

ashking94 · 2022-07-29T07:55:35Z

server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java

+    }
+
+    /**
+     * Returns list of all the segment files uploaded to remote segment store till the last refresh checkpoint.


the method only returns the latest segment files from the last known commit (remotely) and the incremental refreshed segment files. Seems a bit misleading from the description that Returns list of all the segment files uploaded to remote segment store till the last refresh checkpoint.which sounds like all segment files ever.

Do we foresee where all we intend to use this method?

commit metadata contains all the segment files till that point.

This method will be used in restore flow where we get the list of segment files from remote store and use it to copy to local directory.

ashking94 · 2022-07-29T08:03:20Z

server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java

+    public long fileLength(String name) throws IOException {
+        String remoteFilename = getExistingRemoteFilename(name);
+        if (remoteFilename != null) {
+            return remoteDataDirectory.fileLength(remoteFilename);


remoteDataDirectory.fileLength throws NoSuchFileException(name). You might want to handle the exception here and not return the internal filename outside?

there are more occurrences where this would be applicable. Pls look into those as well.

But we are not exposing the internal filename, right? name is passed to this method.

ashking94 · 2022-07-29T08:09:11Z

server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java

+     * @throws IOException in case of I/O error
+     */
+    @Override
+    public IndexOutput createOutput(String name, IOContext context) throws IOException {


Looks like this is not really throwing an IOException nor any other exception. We can remove throws part where we are not really throwing exceptions.

remoteDataDirectory.createOutput throws IOException.

ashking94 · 2022-07-29T08:11:35Z

server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java

+        if (segmentsUploadedToRemoteStore.containsKey(localFilename)) {
+            return segmentsUploadedToRemoteStore.get(localFilename).uploadedFilename;
+        } else {
+            return null;


Could we have a fallback to remote store here in the else part if the local cache for whatsoever reason does not contain the name in the map?

I added this flow on purpose. This could result in a lot of load on the remote store itself.

sachinpkale · 2022-07-29T16:21:18Z

Update: I got to know that SegmentInfosSnapshot does not just contain incremental segment files since the last commit but contain list of all the live segment files for the given shard. This would simplify current approach where instead of keeping track of 2 separate metadata files (commit and refresh), we keep track of only one metadata file. This is just a thought as of now. I will make the changes after outlining all the details.

github-actions · 2022-08-01T04:59:39Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/1153/
CommitID: 813dc00

sachinpkale · 2022-08-01T05:57:20Z

Build failed with:

java.lang.AssertionError: Failure at [repository_s3/20_repository_permanent_credentials:152]: expected [2xx] status code but api [snapshot.create] returned [400 Bad Request] [{"error":{"root_cause":[{"type":"invalid_snapshot_name_exception","reason":"[repository_permanent:snapshot-one] Invalid snapshot name [snapshot-one], snapshot with the same name already exists","stack_trace":"InvalidSnapshotNameException[[repository_permanent:snapshot-one] Invalid snapshot name [snapshot-one], snapshot with the same name already exists]

This test is not related with the changes in this PR. Re-triggering the build.

github-actions · 2022-08-01T06:35:55Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/1164/
CommitID: 522905e8036ae7e0b721a931a3883080eadbd115

dreamer-89 · 2022-08-01T17:36:56Z

Created #4069 to track this.

github-actions · 2022-08-02T02:37:24Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/1235/
CommitID: 2efcc1067cbca71d68ae2e92419786fa75f49974

ankitkala · 2022-08-02T04:51:40Z

server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java

+ * another instance of {@code RemoteDirectory}.
+ * @opensearch.internal
+ */
+public final class RemoteSegmentStoreDirectory extends FilterDirectory {


Quoting from FilterDirectory javadoc:

if you plan to write your own Directory implementation, you should consider extending directly Directory or BaseDirectory rather than try to reuse functionality of existing Directorys by extending this class.

Based on this, we should extend FilterDirectory only if we plan to use the Directory implementation. But in this case, we're not really using Directory implementation. Again, any benefit we're getting here by extending FilterDirectory?

ankitkala · 2022-08-02T04:52:53Z

server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java

+     * Each segment file is uploaded with unique suffix.
+     * For example, _0.cfe in local filesystem will be uploaded to remote segment store as _0.cfe__gX7bNIIBrs0AUNsR2yEG
+     */
+    public static final String SEGMENT_NAME_UUID_SEPARATOR = "__";


Can the segment name or UUID also contain __ ? Just wondering if we can run into corner cases where the segment name translation can break.

UUID should not contain special character other than -. As per lucene file format explained here: https://lucene.apache.org/core/3_0_3/fileformats.html#File%20Naming, __ should not be included in the segment name. But I agree with the concern. Not sure which delimiter can be used which will guarantee that it will not be used in the segment name.

Got it. Should be fine then. First occurrence is always guaranteed to be the separator.

ankitkala · 2022-08-02T05:05:46Z

server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java

+     */
+    public void init() throws IOException {
+        this.refreshMetadataFileUniqueSuffix = UUIDs.base64UUID();
+        this.segmentsUploadedToRemoteStore = new ConcurrentHashMap<>(readLatestMetadataFile());


We haven't really uploaded any segments to remote store at this point right? or does this handle the cases where replica gets promoted to primary?

Yes, failover or node restart.

@sachinpkale it seems like there is no cleanup mechanism in place: if init() is called for initialized bu stale cache, the stale files have to be cleaned up (using previous metadataFileUniqueSuffix).

@reta We always read latest metadata file. So, even if there are more than 1 metadata files, it won't impact the correctness.
But you are right on cleanup part. We need cleanup of stale metadata files in order to keep performance of init() in check. I will be raising a follow-up PR which will contain clean-up logic. Implementation is part of another branch: sachinpkale@b0d95c9#diff-f57628264c235a133d2d6db458404e07a43f18026e40ed5e956536c03add9925R273

ankitkala · 2022-08-02T05:11:40Z

server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java

+        if (latestRefreshMetadataFile.isPresent()) {
+            logger.info("Reading latest refresh metadata file {}", latestRefreshMetadataFile.get());
+            segmentMetadataMap.putAll(readMetadataFile(latestRefreshMetadataFile.get()));
+            this.refreshMetadataFileUniqueSuffix = MetadataFilenameUtils.getUuid(latestRefreshMetadataFile.get());


In what use case would we need to update refreshMetadataFileUniqueSuffix as part for this method?

github-actions · 2022-08-02T05:19:54Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/1239/
CommitID: 1a4cbda5ca69ea58fdaa3bfe6e5448e7127b82ec

github-actions · 2022-08-02T11:43:45Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/1258/
CommitID: 1ef80346f1044a3b29c3ebeaf151ecad8baaa61e

github-actions · 2022-08-02T13:54:45Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/1263/
CommitID: f9c943065ec02409992d885b7d2af51febda1d9c

Bukhtawar · 2022-08-08T06:25:53Z

server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java

+    private final RemoteDirectory remoteDataDirectory;
+    /**
+     * remoteMetadataDirectory is used to store metadata files at path: cluster_UUID/index_UUID/shardId/segments/metadata
+     */
+    private final RemoteDirectory remoteMetadataDirectory;
+
+    /**
+     * To prevent explosion of refresh metadata files, we replace refresh files for the given primary term and generation
+     * This is achieved by uploading refresh metadata file with the same UUID suffix.
+     */
+    private String metadataFileUniqueSuffix;
+
+    /**
+     * Keeps track of local segment filename to uploaded filename along with other attributes like checksum.
+     * This map acts as a cache layer for uploaded segment filenames which helps avoid calling listAll() each time.
+     * It is important to initialize this map on creation of RemoteSegmentStoreDirectory and update it on each upload and delete.
+     */
+    private Map<String, UploadedSegmentMetadata> segmentsUploadedToRemoteStore;
+
+    private static final Logger logger = LogManager.getLogger(RemoteSegmentStoreDirectory.class);
+
+    public RemoteSegmentStoreDirectory(RemoteDirectory remoteDataDirectory, RemoteDirectory remoteMetadataDirectory) throws IOException {


The only concern I see with having two RemoteDirectory instances is they can fundamentally be two separate stores altogether with varying consistency properties. Do we really need them to be exposed separately. Can we not keep the metadata repository internal to this class, to avoid possible misuse of this class.

remoteMetadataDirectory will be internal to this class. Callers of RemoteSegmentStoreDirectory will not be aware of the existence of two different remote directories.

On that part of creating an instance of RemoteSegmentStoreDirectory, it will be handled by the factory. The implementation of factory method will be part of subsequent PR but you can find the code in this commit: https://github.com/sachinpkale/OpenSearch/blob/remote_segment_failure_handling/server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectoryFactory.java

Sure but the uploadMetadata is still public

public void uploadMetadata(Collection<String> segmentFiles, Directory storeDirectory, long primaryTerm, long generation)

Yes, I think you are suggesting to make uploadMetadata private. That is that plan but I haven't made that change as not sure on how to decide the trigger for metadata upload.
Today, it will be called by RemoteStoreRefereshListener as RemoteStoreRefreshListener knows when the refresh is happening.

@Bukhtawar Created a tracking issue for the same: #4171

Signed-off-by: Sachin Kale <kalsac@amazon.com>

github-actions · 2022-08-12T06:08:20Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/1685/
CommitID: dafb962f0564dcb29205039d867a82190c3de463

sachinpkale · 2022-08-12T06:44:35Z

Failing tests:

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testRestoreSnapshotAllocationDoesNotExceedWatermark" -Dtests.seed=DFD6B4D068693FA8 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=en-PH -Dtests.timezone=ROK -Druntime.java=17

Seems like a flaky test. Re-triggering build

Signed-off-by: Sachin Kale <kalsac@amazon.com>

github-actions · 2022-08-12T07:11:19Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/1689/
CommitID: 2470f4a

Bukhtawar · 2022-08-12T08:40:32Z

server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java

+    @Override
+    public void deleteFile(String name) throws IOException {
+        String remoteFilename = getExistingRemoteFilename(name);
+        if (remoteFilename != null) {
+            remoteDataDirectory.deleteFile(remoteFilename);
+            segmentsUploadedToRemoteStore.remove(name);
+        }
+    }


One rare race could be we delete file and segmentsUploadedToRemoteStore hasn't been updated and then a concurrent upload happens and we check for containsFile where the stale entry that the file is uploaded is present, will cause the new segment file to get skipped from being uploaded.
Can you confirm if there is a valid scenario that might hit this race?

I don't think of a scenario where this should happen due to following reasons:

Segment file with the same name is not created, even after segments are deleted as part of merge, new segment files will always be prefixed with next generation number.

Even if segment file with the same name is re-created (this contradicts with point number 1), we make sure each segment file is uploaded with the unique suffix.

…e segment store (opensearch-project#4020) * Add RemoteSegmentStoreDirectory to interact with remote segment store Signed-off-by: Sachin Kale <kalsac@amazon.com>

…4380) * [Remote Store] Upload segments to remote store post refresh (#3460) * Add RemoteDirectory interface to copy segment files to/from remote store Signed-off-by: Sachin Kale <kalsac@amazon.com> Co-authored-by: Sachin Kale <kalsac@amazon.com> * Add index level setting for remote store Signed-off-by: Sachin Kale <kalsac@amazon.com> Co-authored-by: Sachin Kale <kalsac@amazon.com> * Add RemoteDirectoryFactory and use RemoteDirectory instance in RefreshListener Co-authored-by: Sachin Kale <kalsac@amazon.com> Signed-off-by: Sachin Kale <kalsac@amazon.com> * Upload segment to remote store post refresh Signed-off-by: Sachin Kale <kalsac@amazon.com> Co-authored-by: Sachin Kale <kalsac@amazon.com> Signed-off-by: Sachin Kale <kalsac@amazon.com> * [Remote Store] Inject remote store in IndexShard instead of RemoteStoreRefreshListener (#3703) * Inject remote store in IndexShard instead of RemoteStoreRefreshListener Signed-off-by: Sachin Kale <kalsac@amazon.com> * Pass supplier of RepositoriesService to RemoteDirectoryFactory Signed-off-by: Sachin Kale <kalsac@amazon.com> * Create isRemoteStoreEnabled function for IndexShard Signed-off-by: Sachin Kale <kalsac@amazon.com> * Explicitly close remoteStore on indexShard close Signed-off-by: Sachin Kale <kalsac@amazon.com> * Change RemoteDirectory.close to a no-op Signed-off-by: Sachin Kale <kalsac@amazon.com> Co-authored-by: Sachin Kale <kalsac@amazon.com> * [Remote Store] Add remote store restore API implementation (#3642) * Add remote restore API implementation Signed-off-by: Sachin Kale <kalsac@amazon.com> * [Remote Store] Add support to add nested settings for remote store (#4060) * Add support to add nested settings for remote store Signed-off-by: Sachin Kale <kalsac@amazon.com> * [Remote Store] Add rest endpoint for remote store restore (#3576) * Add rest endpoint for remote store restore Signed-off-by: Sachin Kale <kalsac@amazon.com> * [Remote Store] Add validator that forces segment replication type before enabling remote store (#4175) * Add validator that forces segment replication type before enabling remote store Signed-off-by: Sachin Kale <kalsac@amazon.com> * [Remote Store] Change remote_store setting validation message to make it more clear (#4199) * Change remote_store setting validation message to make it more clear Signed-off-by: Sachin Kale <kalsac@amazon.com> * [Remote Store] Add RemoteSegmentStoreDirectory to interact with remote segment store (#4020) * Add RemoteSegmentStoreDirectory to interact with remote segment store Signed-off-by: Sachin Kale <kalsac@amazon.com> * Use RemoteSegmentStoreDirectory instead of RemoteDirectory (#4240) * Use RemoteSegmentStoreDirectory instead of RemoteDirectory Signed-off-by: Sachin Kale <kalsac@amazon.com> Signed-off-by: Sachin Kale <kalsac@amazon.com> Co-authored-by: Sachin Kale <kalsac@amazon.com>

@sachinpkale

I have nominated and maintainers have agreed to invite Sachin Kale (@sachinpkale) to be a co-maintainer. Sachin has kindly accepted. Sachin has led the design and implementation of the remote backed storage feature in OpenSearch. This feature was introduced as experimental in OpenSearch 2.3 and is planned for general availability in 2.9. Some significant issues and PRs authored by Sachin for this effort are as follows: Feature proposal: opensearch-project#1968 Upload segments to remote store post refresh: opensearch-project#3460 Add rest endpoint for remote store restore: opensearch-project#3576 Add RemoteSegmentStoreDirectory to interact with remote segment store: opensearch-project#4020 In total, Sachin has authored 57 PRs going back to May 2022. He also frequently reviews contributions from others and has reviewed nearly 100 PRs in the same time frame. Signed-off-by: Andrew Ross <andrross@amazon.com>

@sachinpkale

I have nominated and maintainers have agreed to invite Sachin Kale (@sachinpkale) to be a co-maintainer. Sachin has kindly accepted. Sachin has led the design and implementation of the remote backed storage feature in OpenSearch. This feature was introduced as experimental in OpenSearch 2.3 and is planned for general availability in 2.9. Some significant issues and PRs authored by Sachin for this effort are as follows: Feature proposal: opensearch-project#1968 Upload segments to remote store post refresh: opensearch-project#3460 Add rest endpoint for remote store restore: opensearch-project#3576 Add RemoteSegmentStoreDirectory to interact with remote segment store: opensearch-project#4020 In total, Sachin has authored 57 PRs going back to May 2022. He also frequently reviews contributions from others and has reviewed nearly 100 PRs in the same time frame. Signed-off-by: Andrew Ross <andrross@amazon.com>

@sachinpkale

I have nominated and maintainers have agreed to invite Sachin Kale (@sachinpkale) to be a co-maintainer. Sachin has kindly accepted. Sachin has led the design and implementation of the remote backed storage feature in OpenSearch. This feature was introduced as experimental in OpenSearch 2.3 and is planned for general availability in 2.9. Some significant issues and PRs authored by Sachin for this effort are as follows: Feature proposal: opensearch-project#1968 Upload segments to remote store post refresh: opensearch-project#3460 Add rest endpoint for remote store restore: opensearch-project#3576 Add RemoteSegmentStoreDirectory to interact with remote segment store: opensearch-project#4020 In total, Sachin has authored 57 PRs going back to May 2022. He also frequently reviews contributions from others and has reviewed nearly 100 PRs in the same time frame. Signed-off-by: Andrew Ross <andrross@amazon.com>

@sachinpkale

I have nominated and maintainers have agreed to invite Sachin Kale (@sachinpkale) to be a co-maintainer. Sachin has kindly accepted. Sachin has led the design and implementation of the remote backed storage feature in OpenSearch. This feature was introduced as experimental in OpenSearch 2.3 and is planned for general availability in 2.9. Some significant issues and PRs authored by Sachin for this effort are as follows: Feature proposal: #1968 Upload segments to remote store post refresh: #3460 Add rest endpoint for remote store restore: #3576 Add RemoteSegmentStoreDirectory to interact with remote segment store: #4020 In total, Sachin has authored 57 PRs going back to May 2022. He also frequently reviews contributions from others and has reviewed nearly 100 PRs in the same time frame. Signed-off-by: Andrew Ross <andrross@amazon.com>

@sachinpkale

I have nominated and maintainers have agreed to invite Sachin Kale (@sachinpkale) to be a co-maintainer. Sachin has kindly accepted. Sachin has led the design and implementation of the remote backed storage feature in OpenSearch. This feature was introduced as experimental in OpenSearch 2.3 and is planned for general availability in 2.9. Some significant issues and PRs authored by Sachin for this effort are as follows: Feature proposal: opensearch-project#1968 Upload segments to remote store post refresh: opensearch-project#3460 Add rest endpoint for remote store restore: opensearch-project#3576 Add RemoteSegmentStoreDirectory to interact with remote segment store: opensearch-project#4020 In total, Sachin has authored 57 PRs going back to May 2022. He also frequently reviews contributions from others and has reviewed nearly 100 PRs in the same time frame. Signed-off-by: Andrew Ross <andrross@amazon.com>

@sachinpkale

I have nominated and maintainers have agreed to invite Sachin Kale (@sachinpkale) to be a co-maintainer. Sachin has kindly accepted. Sachin has led the design and implementation of the remote backed storage feature in OpenSearch. This feature was introduced as experimental in OpenSearch 2.3 and is planned for general availability in 2.9. Some significant issues and PRs authored by Sachin for this effort are as follows: Feature proposal: #1968 Upload segments to remote store post refresh: #3460 Add rest endpoint for remote store restore: #3576 Add RemoteSegmentStoreDirectory to interact with remote segment store: #4020 In total, Sachin has authored 57 PRs going back to May 2022. He also frequently reviews contributions from others and has reviewed nearly 100 PRs in the same time frame. Signed-off-by: Andrew Ross <andrross@amazon.com>

@sachinpkale

opensearch-project#8047) I have nominated and maintainers have agreed to invite Sachin Kale (@sachinpkale) to be a co-maintainer. Sachin has kindly accepted. Sachin has led the design and implementation of the remote backed storage feature in OpenSearch. This feature was introduced as experimental in OpenSearch 2.3 and is planned for general availability in 2.9. Some significant issues and PRs authored by Sachin for this effort are as follows: Feature proposal: opensearch-project#1968 Upload segments to remote store post refresh: opensearch-project#3460 Add rest endpoint for remote store restore: opensearch-project#3576 Add RemoteSegmentStoreDirectory to interact with remote segment store: opensearch-project#4020 In total, Sachin has authored 57 PRs going back to May 2022. He also frequently reviews contributions from others and has reviewed nearly 100 PRs in the same time frame. Signed-off-by: Andrew Ross <andrross@amazon.com>

@sachinpkale

I have nominated and maintainers have agreed to invite Sachin Kale (@sachinpkale) to be a co-maintainer. Sachin has kindly accepted. Sachin has led the design and implementation of the remote backed storage feature in OpenSearch. This feature was introduced as experimental in OpenSearch 2.3 and is planned for general availability in 2.9. Some significant issues and PRs authored by Sachin for this effort are as follows: Feature proposal: opensearch-project#1968 Upload segments to remote store post refresh: opensearch-project#3460 Add rest endpoint for remote store restore: opensearch-project#3576 Add RemoteSegmentStoreDirectory to interact with remote segment store: opensearch-project#4020 In total, Sachin has authored 57 PRs going back to May 2022. He also frequently reviews contributions from others and has reviewed nearly 100 PRs in the same time frame. Signed-off-by: Andrew Ross <andrross@amazon.com> Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

@sachinpkale

I have nominated and maintainers have agreed to invite Sachin Kale (@sachinpkale) to be a co-maintainer. Sachin has kindly accepted. Sachin has led the design and implementation of the remote backed storage feature in OpenSearch. This feature was introduced as experimental in OpenSearch 2.3 and is planned for general availability in 2.9. Some significant issues and PRs authored by Sachin for this effort are as follows: Feature proposal: opensearch-project#1968 Upload segments to remote store post refresh: opensearch-project#3460 Add rest endpoint for remote store restore: opensearch-project#3576 Add RemoteSegmentStoreDirectory to interact with remote segment store: opensearch-project#4020 In total, Sachin has authored 57 PRs going back to May 2022. He also frequently reviews contributions from others and has reviewed nearly 100 PRs in the same time frame. Signed-off-by: Andrew Ross <andrross@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

sachinpkale requested review from a team and reta as code owners July 27, 2022 04:17

sachinpkale force-pushed the remote_segment_store_directory branch from 2602517 to 505acf9 Compare July 27, 2022 04:33

sachinpkale force-pushed the remote_segment_store_directory branch from 505acf9 to fc25eaf Compare July 27, 2022 06:51

sachinpkale force-pushed the remote_segment_store_directory branch from fc25eaf to 6e053b4 Compare July 28, 2022 08:57

ashking94 requested changes Jul 29, 2022

View reviewed changes

sachinpkale changed the title ~~Add RemoteSegmentStoreDirectory to interact with remote segment store~~ [Remote Store] Add RemoteSegmentStoreDirectory to interact with remote segment store Aug 1, 2022

sachinpkale force-pushed the remote_segment_store_directory branch from 813dc00 to 522905e Compare August 1, 2022 06:06

sachinpkale force-pushed the remote_segment_store_directory branch from 522905e to 2efcc10 Compare August 2, 2022 02:07

sachinpkale force-pushed the remote_segment_store_directory branch from 2efcc10 to 1a4cbda Compare August 2, 2022 05:12

ankitkala reviewed Aug 2, 2022

View reviewed changes

sachinpkale force-pushed the remote_segment_store_directory branch from 1a4cbda to 1ef8034 Compare August 2, 2022 11:20

sachinpkale force-pushed the remote_segment_store_directory branch from 1ef8034 to f9c9430 Compare August 2, 2022 13:29

Bukhtawar reviewed Aug 8, 2022

View reviewed changes

Sachin Kale added 2 commits August 12, 2022 11:12

Add RemoteSegmentStoreDirectory to interact with remote segment store

b115593

Signed-off-by: Sachin Kale <kalsac@amazon.com>

Use one type of metadata file

02bc44d

Signed-off-by: Sachin Kale <kalsac@amazon.com>

sachinpkale force-pushed the remote_segment_store_directory branch from ce356bf to dafb962 Compare August 12, 2022 05:43

sachinpkale mentioned this pull request Aug 12, 2022

[Remote Store] Track segment uploads as a part of an upload listener. #4198

Open

Wrap uploadMetadata implementation with synchronized

2470f4a

Signed-off-by: Sachin Kale <kalsac@amazon.com>

sachinpkale force-pushed the remote_segment_store_directory branch from dafb962 to 2470f4a Compare August 12, 2022 06:46

Bukhtawar reviewed Aug 12, 2022

View reviewed changes

Bukhtawar approved these changes Aug 12, 2022

View reviewed changes

Bukhtawar merged commit 5f2e66b into opensearch-project:main Aug 13, 2022

sachinpkale mentioned this pull request Aug 17, 2022

Use RemoteSegmentStoreDirectory instead of RemoteDirectory #4240

Merged

5 tasks

sachinpkale mentioned this pull request Sep 1, 2022

[Backport 2.x] [Remote Store] Backport remote segment store changes #4380

Merged

6 tasks

andrross mentioned this pull request Jun 12, 2023

Add @sachinpkale as an OpenSearch maintainer #8019

Merged

6 tasks

andrross mentioned this pull request Jun 13, 2023

[Backport 2.x] Add @sachinpkale as an OpenSearch maintainer (#8019) #8047

Merged

6 tasks

[Remote Store] Add RemoteSegmentStoreDirectory to interact with remote segment store #4020

[Remote Store] Add RemoteSegmentStoreDirectory to interact with remote segment store #4020

Conversation

sachinpkale commented Jul 27, 2022 • edited Loading

Description

Issues Resolved

Check List

github-actions bot commented Jul 27, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jul 27, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jul 27, 2022

Gradle Check (Jenkins) Run Completed with:

sachinpkale commented Jul 28, 2022

github-actions bot commented Jul 28, 2022

Gradle Check (Jenkins) Run Completed with:

codecov-commenter commented Jul 28, 2022 • edited Loading

Codecov Report

Bukhtawar commented Jul 29, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ashking94 Jul 29, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sachinpkale commented Jul 29, 2022

github-actions bot commented Aug 1, 2022

Gradle Check (Jenkins) Run Completed with:

sachinpkale commented Aug 1, 2022

github-actions bot commented Aug 1, 2022

Gradle Check (Jenkins) Run Completed with:

dreamer-89 commented Aug 1, 2022

github-actions bot commented Aug 2, 2022

Gradle Check (Jenkins) Run Completed with:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Aug 2, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 2, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 2, 2022

Gradle Check (Jenkins) Run Completed with:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sachinpkale Aug 8, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Aug 12, 2022

Gradle Check (Jenkins) Run Completed with:

sachinpkale commented Aug 12, 2022

github-actions bot commented Aug 12, 2022

Gradle Check (Jenkins) Run Completed with:

sachinpkale commented Jul 27, 2022 •

edited

Loading

codecov-commenter commented Jul 28, 2022 •

edited

Loading

ashking94 Jul 29, 2022 •

edited

Loading

sachinpkale Aug 8, 2022 •

edited

Loading

sachinpkale Aug 12, 2022 •

edited

Loading