Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add restriction to have a single repository with shallow snapshot v2 setting #15901

Merged
merged 2 commits into from
Sep 12, 2024

Conversation

sachinpkale
Copy link
Member

@sachinpkale sachinpkale commented Sep 11, 2024

Description

  • Shallow snapshot V2 setting is introduced to enable creating snapshot which uses pinned timestamp functionality of remote store.
  • As pinned timestamps are created at cluster level, if multiple repositories enable this feature and create snapshots, clean-up of data from remote store on snapshot deletion becomes tricky and we may end up leaving zombie data in the remote store.
  • To avoid this, in this PR, we have added a restriction to have only 1 repository with shallow snapshot V2 enabled.
  • In order to enable shallow snapshot V2 setting on another repository, we need to disable this setting on the previously enabled repository as well as delete any snapshots that are created with the previous repo.
  • We will ease up this restriction in the upcoming releases.

Related Issues

Check List

  • Functionality includes testing.
  • API changes companion pull request created, if applicable.
  • Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Copy link
Contributor

✅ Gradle check result for 1526597: SUCCESS

…setting

Signed-off-by: Sachin Kale <sachinpkale@gmail.com>
Copy link
Contributor

❕ Gradle check result for c57c9c2: UNSTABLE

  • TEST FAILURES:
      1 org.opensearch.remotestore.RemoteStoreStatsIT.testNonZeroPrimaryStatsOnNewlyCreatedIndexWithZeroDocs
      1 org.opensearch.remotestore.RemoteStoreStatsIT.testDownloadStatsCorrectnessSinglePrimarySingleReplica

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Copy link
Contributor

✅ Gradle check result for 0d86005: SUCCESS

Copy link

codecov bot commented Sep 11, 2024

Codecov Report

Attention: Patch coverage is 86.36364% with 6 lines in your changes missing coverage. Please review.

Project coverage is 72.01%. Comparing base (92d7fe8) to head (de752dc).
Report is 6 commits behind head on main.

Files with missing lines Patch % Lines
...g/opensearch/repositories/RepositoriesService.java 80.76% 4 Missing and 1 partial ⚠️
...remotestore/RemoteStorePinnedTimestampService.java 94.44% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main   #15901      +/-   ##
============================================
+ Coverage     71.90%   72.01%   +0.10%     
- Complexity    64216    64311      +95     
============================================
  Files          5272     5272              
  Lines        300597   300636      +39     
  Branches      43440    43449       +9     
============================================
+ Hits         216151   216499     +348     
+ Misses        66680    66422     -258     
+ Partials      17766    17715      -51     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…_TIMESTAMP_DELIMITER

Signed-off-by: Sachin Kale <sachinpkale@gmail.com>
Copy link
Contributor

✅ Gradle check result for de752dc: SUCCESS

@sachinpkale sachinpkale merged commit 330b249 into opensearch-project:main Sep 12, 2024
34 of 39 checks passed
@sachinpkale sachinpkale added backport 2.x Backport to 2.x branch backport 2.17 labels Sep 12, 2024
opensearch-trigger-bot bot pushed a commit that referenced this pull request Sep 12, 2024
…setting (#15901)

* Add restriction to have a single repository with shallow snapshot v2 setting

Signed-off-by: Sachin Kale <sachinpkale@gmail.com>

* Do not allow shallow snapshot v2 repo name to contain SNAPSHOT_PINNED_TIMESTAMP_DELIMITER

Signed-off-by: Sachin Kale <sachinpkale@gmail.com>

---------

Signed-off-by: Sachin Kale <sachinpkale@gmail.com>
(cherry picked from commit 330b249)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
opensearch-trigger-bot bot pushed a commit that referenced this pull request Sep 12, 2024
…setting (#15901)

* Add restriction to have a single repository with shallow snapshot v2 setting

Signed-off-by: Sachin Kale <sachinpkale@gmail.com>

* Do not allow shallow snapshot v2 repo name to contain SNAPSHOT_PINNED_TIMESTAMP_DELIMITER

Signed-off-by: Sachin Kale <sachinpkale@gmail.com>

---------

Signed-off-by: Sachin Kale <sachinpkale@gmail.com>
(cherry picked from commit 330b249)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
sachinpkale added a commit that referenced this pull request Sep 12, 2024
…low snapshot v2 setting (#15911)

* Add restriction to have a single repository with shallow snapshot v2 setting (#15901)

* Add restriction to have a single repository with shallow snapshot v2 setting

Signed-off-by: Sachin Kale <sachinpkale@gmail.com>

* Do not allow shallow snapshot v2 repo name to contain SNAPSHOT_PINNED_TIMESTAMP_DELIMITER

Signed-off-by: Sachin Kale <sachinpkale@gmail.com>

---------

Signed-off-by: Sachin Kale <sachinpkale@gmail.com>
(cherry picked from commit 330b249)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Add a method to avoid breaking changes

Signed-off-by: Sachin Kale <sachinpkale@gmail.com>

---------

Signed-off-by: Sachin Kale <sachinpkale@gmail.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Sachin Kale <sachinpkale@gmail.com>
sachinpkale added a commit that referenced this pull request Sep 12, 2024
…ow snapshot v2 setting (#15912)

* Add restriction to have a single repository with shallow snapshot v2 setting (#15901)

* Add restriction to have a single repository with shallow snapshot v2 setting

Signed-off-by: Sachin Kale <sachinpkale@gmail.com>

* Do not allow shallow snapshot v2 repo name to contain SNAPSHOT_PINNED_TIMESTAMP_DELIMITER

Signed-off-by: Sachin Kale <sachinpkale@gmail.com>

---------

Signed-off-by: Sachin Kale <sachinpkale@gmail.com>
(cherry picked from commit 330b249)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Add a method to avoid breaking changes

Signed-off-by: Sachin Kale <sachinpkale@gmail.com>

---------

Signed-off-by: Sachin Kale <sachinpkale@gmail.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Sachin Kale <sachinpkale@gmail.com>
dk2k pushed a commit to dk2k/OpenSearch that referenced this pull request Oct 16, 2024
…setting (opensearch-project#15901)

* Add restriction to have a single repository with shallow snapshot v2 setting

Signed-off-by: Sachin Kale <sachinpkale@gmail.com>

* Do not allow shallow snapshot v2 repo name to contain SNAPSHOT_PINNED_TIMESTAMP_DELIMITER

Signed-off-by: Sachin Kale <sachinpkale@gmail.com>

---------

Signed-off-by: Sachin Kale <sachinpkale@gmail.com>
dk2k pushed a commit to dk2k/OpenSearch that referenced this pull request Oct 17, 2024
…setting (opensearch-project#15901)

* Add restriction to have a single repository with shallow snapshot v2 setting

Signed-off-by: Sachin Kale <sachinpkale@gmail.com>

* Do not allow shallow snapshot v2 repo name to contain SNAPSHOT_PINNED_TIMESTAMP_DELIMITER

Signed-off-by: Sachin Kale <sachinpkale@gmail.com>

---------

Signed-off-by: Sachin Kale <sachinpkale@gmail.com>
dk2k pushed a commit to dk2k/OpenSearch that referenced this pull request Oct 21, 2024
…setting (opensearch-project#15901)

* Add restriction to have a single repository with shallow snapshot v2 setting

Signed-off-by: Sachin Kale <sachinpkale@gmail.com>

* Do not allow shallow snapshot v2 repo name to contain SNAPSHOT_PINNED_TIMESTAMP_DELIMITER

Signed-off-by: Sachin Kale <sachinpkale@gmail.com>

---------

Signed-off-by: Sachin Kale <sachinpkale@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants