-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Remote Store] Add multipart upload integration for translog and segment files #7119
Merged
gbbafna
merged 18 commits into
opensearch-project:main
from
raghuvanshraj:multi-part-upload-core-2-pr
Jul 11, 2023
Merged
[Remote Store] Add multipart upload integration for translog and segment files #7119
gbbafna
merged 18 commits into
opensearch-project:main
from
raghuvanshraj:multi-part-upload-core-2-pr
Jul 11, 2023
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Gradle Check (Jenkins) Run Completed with:
|
6 tasks
linuxpi
reviewed
Apr 21, 2023
server/src/main/java/org/opensearch/common/blobstore/BlobContainer.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/common/blobstore/stream/write/UploadResponse.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/shard/RemoteStoreRefreshListener.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/translog/transfer/BlobStoreTransferService.java
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/translog/transfer/BlobStoreTransferService.java
Outdated
Show resolved
Hide resolved
raghuvanshraj
force-pushed
the
multi-part-upload-core-2-pr
branch
from
May 2, 2023 09:58
716c219
to
33f3a69
Compare
Gradle Check (Jenkins) Run Completed with:
|
ashking94
reviewed
May 19, 2023
server/src/main/java/org/opensearch/common/blobstore/BlobContainer.java
Outdated
Show resolved
Hide resolved
ashking94
reviewed
May 19, 2023
server/src/main/java/org/opensearch/common/blobstore/stream/write/UploadResponse.java
Outdated
Show resolved
Hide resolved
ashking94
reviewed
May 19, 2023
server/src/main/java/org/opensearch/common/blobstore/stream/write/UploadResponse.java
Outdated
Show resolved
Hide resolved
ashking94
reviewed
May 19, 2023
gbbafna
reviewed
May 19, 2023
server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/shard/RemoteStoreRefreshListener.java
Outdated
Show resolved
Hide resolved
ashking94
reviewed
May 19, 2023
server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java
Outdated
Show resolved
Hide resolved
ashking94
reviewed
May 19, 2023
server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java
Outdated
Show resolved
Hide resolved
ashking94
reviewed
May 19, 2023
server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java
Outdated
Show resolved
Hide resolved
ashking94
requested changes
May 19, 2023
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
have done high level review, will review once more.
gbbafna
reviewed
May 23, 2023
server/src/main/java/org/opensearch/index/shard/RemoteStoreRefreshListener.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/translog/transfer/BlobStoreTransferService.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/translog/transfer/BlobStoreTransferService.java
Outdated
Show resolved
Hide resolved
server/src/test/java/org/opensearch/index/store/RemoteSegmentStoreDirectoryTests.java
Show resolved
Hide resolved
raghuvanshraj
force-pushed
the
multi-part-upload-core-2-pr
branch
from
May 24, 2023 15:52
33f3a69
to
4827a91
Compare
Gradle Check (Jenkins) Run Completed with:
|
6 tasks
raghuvanshraj
force-pushed
the
multi-part-upload-core-2-pr
branch
from
May 25, 2023 02:15
4827a91
to
74518bc
Compare
raghuvanshraj
changed the title
Add multipart upload integration for translog and segment files
[Remote Store] Add multipart upload integration for translog and segment files
May 25, 2023
Gradle Check (Jenkins) Run Completed with:
|
Failure from gradle CI logs:
This seems to be a flaky test as it is passing when I run it locally. I've opened a bug for the same: #7755 |
raghuvanshraj
force-pushed
the
multi-part-upload-core-2-pr
branch
from
May 29, 2023 01:30
74518bc
to
55a31ac
Compare
Gradle Check (Jenkins) Run Completed with:
|
raghuvanshraj
force-pushed
the
multi-part-upload-core-2-pr
branch
from
May 29, 2023 12:46
55a31ac
to
c48cc33
Compare
Gradle Check (Jenkins) Run Completed with:
|
gbbafna
reviewed
Jun 1, 2023
server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/store/exception/ChecksumCombinationException.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/translog/checked/TranslogCheckedContainer.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/translog/checked/TranslogCheckedContainer.java
Outdated
Show resolved
Hide resolved
raghuvanshraj
force-pushed
the
multi-part-upload-core-2-pr
branch
from
June 5, 2023 06:32
c48cc33
to
51ba892
Compare
Gradle Check (Jenkins) Run Completed with:
|
Gradle Check (Jenkins) Run Completed with:
|
6 tasks
ashking94
added a commit
to ashking94/OpenSearch
that referenced
this pull request
Jul 11, 2023
…ect#7119 Signed-off-by: Ashish Singh <ssashish@amazon.com>
raghuvanshraj
added
backport 2.x
Backport to 2.x branch
and removed
backport 2.x
Backport to 2.x branch
labels
Jul 11, 2023
sachinpkale
pushed a commit
that referenced
this pull request
Jul 12, 2023
ashking94
pushed a commit
to ashking94/OpenSearch
that referenced
this pull request
Jul 12, 2023
…ent files (opensearch-project#7119) Signed-off-by: Raghuvansh Raj <raghraaj@amazon.com>
ashking94
added a commit
to ashking94/OpenSearch
that referenced
this pull request
Jul 12, 2023
…-project#7119 (opensearch-project#8632) --------- Signed-off-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: Sachin Kale <kalsac@amazon.com> Co-authored-by: Sachin Kale <kalsac@amazon.com>
6 tasks
ashking94
added a commit
to ashking94/OpenSearch
that referenced
this pull request
Jul 12, 2023
…-project#7119 (opensearch-project#8632) --------- Signed-off-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: Sachin Kale <kalsac@amazon.com> Co-authored-by: Sachin Kale <kalsac@amazon.com> Signed-off-by: Ashish Singh <ssashish@amazon.com>
ashking94
pushed a commit
to ashking94/OpenSearch
that referenced
this pull request
Jul 12, 2023
…ent files (opensearch-project#7119) Signed-off-by: Raghuvansh Raj <raghraaj@amazon.com> Signed-off-by: Ashish Singh <ssashish@amazon.com>
ashking94
added a commit
to ashking94/OpenSearch
that referenced
this pull request
Jul 12, 2023
…-project#7119 (opensearch-project#8632) --------- Signed-off-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: Sachin Kale <kalsac@amazon.com> Co-authored-by: Sachin Kale <kalsac@amazon.com> Signed-off-by: Ashish Singh <ssashish@amazon.com>
ashking94
pushed a commit
to ashking94/OpenSearch
that referenced
this pull request
Jul 12, 2023
…ent files (opensearch-project#7119) Signed-off-by: Raghuvansh Raj <raghraaj@amazon.com> Signed-off-by: Ashish Singh <ssashish@amazon.com>
ashking94
added a commit
to ashking94/OpenSearch
that referenced
this pull request
Jul 12, 2023
…-project#7119 (opensearch-project#8632) --------- Signed-off-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: Sachin Kale <kalsac@amazon.com> Co-authored-by: Sachin Kale <kalsac@amazon.com> Signed-off-by: Ashish Singh <ssashish@amazon.com>
vikasvb90
pushed a commit
to raghuvanshraj/OpenSearch
that referenced
this pull request
Jul 12, 2023
…ent files (opensearch-project#7119) Signed-off-by: Raghuvansh Raj <raghraaj@amazon.com>
vikasvb90
pushed a commit
to raghuvanshraj/OpenSearch
that referenced
this pull request
Jul 12, 2023
…-project#7119 (opensearch-project#8632) --------- Signed-off-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: Sachin Kale <kalsac@amazon.com> Co-authored-by: Sachin Kale <kalsac@amazon.com>
raghuvanshraj
added a commit
to raghuvanshraj/OpenSearch
that referenced
this pull request
Jul 12, 2023
…ent files (opensearch-project#7119) Signed-off-by: Raghuvansh Raj <raghraaj@amazon.com>
raghuvanshraj
pushed a commit
to raghuvanshraj/OpenSearch
that referenced
this pull request
Jul 12, 2023
…-project#7119 (opensearch-project#8632) --------- Signed-off-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: Sachin Kale <kalsac@amazon.com> Co-authored-by: Sachin Kale <kalsac@amazon.com>
sachinpkale
pushed a commit
that referenced
this pull request
Jul 12, 2023
* [Remote Store] Add multipart upload integration for translog and segment files (#7119) Signed-off-by: Raghuvansh Raj <raghraaj@amazon.com> Signed-off-by: Ashish Singh <ssashish@amazon.com> * Fix remote segments sync retry regression introduced in PR #7119 (#8632) --------- Signed-off-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: Sachin Kale <kalsac@amazon.com> Co-authored-by: Sachin Kale <kalsac@amazon.com> Signed-off-by: Ashish Singh <ssashish@amazon.com> * Empty-Commit Signed-off-by: Ashish Singh <ssashish@amazon.com> --------- Signed-off-by: Raghuvansh Raj <raghraaj@amazon.com> Signed-off-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: Sachin Kale <kalsac@amazon.com> Co-authored-by: Raghuvansh Raj <raghraaj@amazon.com> Co-authored-by: Sachin Kale <kalsac@amazon.com>
dzane17
pushed a commit
to dzane17/OpenSearch
that referenced
this pull request
Jul 12, 2023
…ent files (opensearch-project#7119) Signed-off-by: Raghuvansh Raj <raghraaj@amazon.com>
dzane17
pushed a commit
to dzane17/OpenSearch
that referenced
this pull request
Jul 12, 2023
…-project#7119 (opensearch-project#8632) --------- Signed-off-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: Sachin Kale <kalsac@amazon.com> Co-authored-by: Sachin Kale <kalsac@amazon.com>
buddharajusahil
pushed a commit
to buddharajusahil/OpenSearch
that referenced
this pull request
Jul 18, 2023
…ent files (opensearch-project#7119) Signed-off-by: Raghuvansh Raj <raghraaj@amazon.com> Signed-off-by: sahil buddharaju <sahilbud@amazon.com>
buddharajusahil
pushed a commit
to buddharajusahil/OpenSearch
that referenced
this pull request
Jul 18, 2023
…-project#7119 (opensearch-project#8632) --------- Signed-off-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: Sachin Kale <kalsac@amazon.com> Co-authored-by: Sachin Kale <kalsac@amazon.com> Signed-off-by: sahil buddharaju <sahilbud@amazon.com>
baba-devv
pushed a commit
to baba-devv/OpenSearch
that referenced
this pull request
Jul 29, 2023
…ent files (opensearch-project#7119) Signed-off-by: Raghuvansh Raj <raghraaj@amazon.com>
baba-devv
pushed a commit
to baba-devv/OpenSearch
that referenced
this pull request
Jul 29, 2023
…-project#7119 (opensearch-project#8632) --------- Signed-off-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: Sachin Kale <kalsac@amazon.com> Co-authored-by: Sachin Kale <kalsac@amazon.com>
shiv0408
pushed a commit
to Gaurav614/OpenSearch
that referenced
this pull request
Apr 25, 2024
…ent files (opensearch-project#7119) Signed-off-by: Raghuvansh Raj <raghraaj@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>
shiv0408
pushed a commit
to Gaurav614/OpenSearch
that referenced
this pull request
Apr 25, 2024
…-project#7119 (opensearch-project#8632) --------- Signed-off-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: Sachin Kale <kalsac@amazon.com> Co-authored-by: Sachin Kale <kalsac@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Credits: @vikasvb90 and @itiyamas for the design and core implementations of the feature.
Description
Adds multipart parallel uploads in segment and translog upload path for remote store utilizing utilities and models added in #7000.
Issues Resolved
Part of #6703
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.