New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Remote Routing Table] Initial commit for index routing table manifest #13577

Merged

sachinpkale merged 25 commits into opensearch-project:main from himshikha:remote_routing_input_stream

Jun 6, 2024

Contributor

himshikha commented May 7, 2024 •

edited

Loading

Description

This change adds remote store objects for Routing Table upload. This also adds indices to be uploaded as part of cluster metadata manifest to track indices changed as part of one cluster state update.

Related Issues

Resolves (#13960)

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)
Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Bukhtawar and others added 6 commits

May 7, 2024 16:18


          Initial commit for index routing table manifest

a66aaaf

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>


          Changes for IndexRoutingTableHeader

ad480ee

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>


          Revert unintentional changes for IndexRoutingTableHeader

2ad70c4

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>


          Revert unintentional changes for IndexRoutingTableHeader

acc172e

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>


          Changes for IndexRoutingTableInputStream

f65b102

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>


          Fixing IndexRoutingTableInputStream and moving checksum to end to file

441f520

Signed-off-by: Himshikha Gupta <himshikh@amazon.com>

Contributor

github-actions bot commented May 7, 2024

❌ Gradle check result for 441f520: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Bukhtawar reviewed

View reviewed changes

...r/src/main/java/org/opensearch/gateway/remote/routingtable/IndexRoutingTableInputStream.java Outdated Show resolved Hide resolved

andrross reviewed

View reviewed changes

server/src/main/java/org/opensearch/common/io/stream/BufferedChecksumStreamInput.java Outdated Show resolved Hide resolved

This was referenced May 8, 2024

Routing table input stream #13602

Closed

Remote routing input stream read #13603

Closed


          Add read flow for IndexRoutingTable

ee70dca

Signed-off-by: Arpit Bandejiya <abandeji@amazon.com>

Contributor

github-actions bot commented May 8, 2024

❌ Gradle check result for ee70dca: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Contributor

github-actions bot commented May 13, 2024

❌ Gradle check result for 9efd85b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?


          Moving routing table version from IndexRouting stream to manifest

7aeecc8

Signed-off-by: Himshikha Gupta <himshikh@amazon.com>

himshikha force-pushed the remote_routing_input_stream branch from 9efd85b to 7aeecc8 Compare

May 14, 2024 06:04

Contributor

github-actions bot commented May 14, 2024

❌ Gradle check result for 7aeecc8: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?


          Refactor reader and add failure test

7b2bc79

Signed-off-by: Arpit Bandejiya <abandeji@amazon.com>

Contributor

github-actions bot commented May 14, 2024

❌ Gradle check result for 7b2bc79: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?


          Fix GatewayMetaStatePersistedStateTests

88d266f

Signed-off-by: Arpit Bandejiya <abandeji@amazon.com>

Contributor

github-actions bot commented May 14, 2024

❌ Gradle check result for 88d266f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?


          Moving codec to version 2 for compatibility with manifest parser

4c97869

Signed-off-by: Himshikha Gupta <himshikh@amazon.com>

himshikha force-pushed the remote_routing_input_stream branch from 1ef13ae to 4c97869 Compare

May 14, 2024 10:56

Contributor

github-actions bot commented May 14, 2024

❌ Gradle check result for 1ef13ae: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Contributor

github-actions bot commented May 14, 2024

❌ Gradle check result for 4c97869: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Himshikha Gupta added 2 commits

May 28, 2024 16:07


          Removing buffer logic

418d9fd

Signed-off-by: Himshikha Gupta <himshikh@amazon.com>


          Merge branch 'main' into remote_routing_input_stream

0400f40

Contributor

github-actions bot commented May 28, 2024

❌ Gradle check result for 0400f40: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?


          Move BufferedChecksum streams to libs/core

7f33ba2

Signed-off-by: Himshikha Gupta <himshikh@amazon.com>

Contributor

github-actions bot commented May 29, 2024

❌ Gradle check result for 7f33ba2: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

sachinpkale reviewed

View reviewed changes

server/src/main/java/org/opensearch/gateway/remote/ClusterMetadataManifest.java Outdated Show resolved Hide resolved

server/src/main/java/org/opensearch/gateway/remote/RemoteClusterStateService.java Show resolved Hide resolved

Contributor

github-actions bot commented Jun 4, 2024

❌ Gradle check result for 8ecf865: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Bukhtawar approved these changes

View reviewed changes

Collaborator

Bukhtawar left a comment

Overall LGTM


          addressing PR comments

bbebce3

Signed-off-by: Himshikha Gupta <himshikh@amazon.com>

himshikha force-pushed the remote_routing_input_stream branch from 8ecf865 to bbebce3 Compare

June 5, 2024 18:14

Contributor

github-actions bot commented Jun 5, 2024

❌ Gradle check result for bbebce3: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

sachinpkale added the skip-changelog label

Himshikha Gupta added 3 commits

June 6, 2024 11:49


          Add package-info

24fe063

Signed-off-by: Himshikha Gupta <himshikh@amazon.com>


          Merge branch 'main' into remote_routing_input_stream

c8ebbc0


          Fixing javadoc failure

01019da

Signed-off-by: Himshikha Gupta <himshikh@amazon.com>

Contributor

github-actions bot commented Jun 6, 2024

❌ Gradle check result for 24fe063: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Contributor

github-actions bot commented Jun 6, 2024

✅ Gradle check result for 01019da: SUCCESS

sachinpkale approved these changes

View reviewed changes

sachinpkale merged commit 7faae25 into opensearch-project:main

28 checks passed

sachinpkale added the backport 2.x label

opensearch-trigger-bot bot pushed a commit that referenced this pull request


          [Remote Routing Table] Initial commit for index routing table manifest (

76d971b

#13577)

* Initial commit for index routing table manifest

Co-authored-by: Bukhtawar Khan <bukhtawa@amazon.com>
Co-authored-by: Himshikha Gupta <himshikh@amazon.com>
Co-authored-by: Arpit Bandejiya <abandeji@amazon.com>
(cherry picked from commit 7faae25)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

opensearch-trigger-bot bot mentioned this pull request

[Backport 2.x] [Remote Routing Table] Initial commit for index routing table manifest #14034

Closed

himshikha mentioned this pull request

[Backport 2.x] [Remote Routing Table] Initial commit for index routing table manifest #14066

Merged

3 tasks

prudhvigodithi mentioned this pull request

Add additional details on Gradle Check failures autocut issues #13950

Closed

Bukhtawar added a commit that referenced this pull request


          [Backport 2.x] [Remote Routing Table] Initial commit for index routin…

0b98803

…g table manifest (#14066)

* [Remote Routing Table] Remote Index routing table manifest (#13577)

Co-authored-by: Bukhtawar Khan <bukhtawa@amazon.com>
Co-authored-by: Himshikha Gupta <himshikh@amazon.com>

parv0201 pushed a commit to parv0201/OpenSearch that referenced this pull request


          [Remote Routing Table] Initial commit for index routing table manifest (

69107ed

opensearch-project#13577)

* Initial commit for index routing table manifest

Co-authored-by: Bukhtawar Khan <bukhtawa@amazon.com>
Co-authored-by: Himshikha Gupta <himshikh@amazon.com>
Co-authored-by: Arpit Bandejiya <abandeji@amazon.com>

This was referenced Jun 13, 2024

[AUTOCUT] Gradle Check Flaky Test Report for IndicesRequestCacheIT #14288

Open

[AUTOCUT] Gradle Check Flaky Test Report for AwarenessAttributeDecommissionIT #14290

Open

[AUTOCUT] Gradle Check Flaky Test Report for RemoteSplitIndexIT #14296

Open

[AUTOCUT] Gradle Check Flaky Test Report for S3BlobStoreRepositoryTests #14299

Open

[AUTOCUT] Gradle Check Flaky Test Report for AzureBlobContainerRetriesTests #14318

Open

finnegancarroll mentioned this pull request

Remove composeBuild fixture tasks when docker support not found #14357

Merged

3 tasks

prudhvigodithi mentioned this pull request

[AUTOCUT] Gradle Check Flaky Test Report for IndicesRequestCacheIT prudhvigodithi/OpenSearch#27

Closed

kkewwei pushed a commit to kkewwei/OpenSearch that referenced this pull request


          [Backport 2.x] [Remote Routing Table] Initial commit for index routin…

c641bc4

…g table manifest (opensearch-project#14066)

* [Remote Routing Table] Remote Index routing table manifest (opensearch-project#13577)

Co-authored-by: Bukhtawar Khan <bukhtawa@amazon.com>
Co-authored-by: Himshikha Gupta <himshikh@amazon.com>
Signed-off-by: kkewwei <kkewwei@163.com>

wdongyu pushed a commit to wdongyu/OpenSearch that referenced this pull request


          [Remote Routing Table] Initial commit for index routing table manifest (

a37d8da

opensearch-project#13577)

* Initial commit for index routing table manifest

Co-authored-by: Bukhtawar Khan <bukhtawa@amazon.com>
Co-authored-by: Himshikha Gupta <himshikh@amazon.com>
Co-authored-by: Arpit Bandejiya <abandeji@amazon.com>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

andrross andrross left review comments

Bukhtawar Bukhtawar approved these changes

sachinpkale sachinpkale approved these changes

anasalkouz Awaiting requested review from anasalkouz anasalkouz is a code owner

CEHENKLE Awaiting requested review from CEHENKLE CEHENKLE is a code owner

dblock Awaiting requested review from dblock dblock is a code owner

dbwiddis Awaiting requested review from dbwiddis dbwiddis is a code owner

dreamer-89 Awaiting requested review from dreamer-89

gbbafna Awaiting requested review from gbbafna gbbafna is a code owner

kotwanikunal Awaiting requested review from kotwanikunal kotwanikunal is a code owner

mch2 Awaiting requested review from mch2 mch2 is a code owner

msfroh Awaiting requested review from msfroh msfroh is a code owner

nknize Awaiting requested review from nknize nknize is a code owner

owaiskazi19 Awaiting requested review from owaiskazi19 owaiskazi19 is a code owner

reta Awaiting requested review from reta reta is a code owner

Rishikesh1159 Awaiting requested review from Rishikesh1159 Rishikesh1159 is a code owner

saratvemulapalli Awaiting requested review from saratvemulapalli saratvemulapalli is a code owner

shwetathareja Awaiting requested review from shwetathareja shwetathareja is a code owner

sohami Awaiting requested review from sohami sohami is a code owner

tlfeng Awaiting requested review from tlfeng

VachaShah Awaiting requested review from VachaShah VachaShah is a code owner

Labels

backport 2.x skip-changelog