[Remote Translog] Trimming based on remote segment upload and cleaning older tlog files #5662

gbbafna · 2022-12-30T14:16:09Z

Signed-off-by: Gaurav Bafna gbbafna@amazon.com

Description

After any segment uploaded to remote store , we will update minSeqNoRequired for Translog. This is implemented only for RemoteFSTranslog .

minSeqNoRequired will be used to preserve generations to be trimmed till it is uploaded to remote store.

Once uploaded , post trimming, remote translog will be cleaned up as well along with local tlog files.

Issues Resolved

#5567

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2022-12-30T14:24:28Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/8627/
CommitID: a626f3e
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-02T09:19:40Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/8696/
CommitID: a8e3ba0
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-02T09:20:17Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/8695/
CommitID: f8694d6
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-02T09:47:17Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/8697/
CommitID: e1f2c43

codecov-commenter · 2023-01-02T09:48:56Z

Codecov Report

Merging #5662 (0037a24) into main (12782f7) will increase coverage by 0.02%.
The diff coverage is 71.05%.

@@             Coverage Diff              @@
##               main    #5662      +/-   ##
============================================
+ Coverage     71.02%   71.04%   +0.02%     
- Complexity    58708    58727      +19     
============================================
  Files          4766     4766              
  Lines        280049   280087      +38     
  Branches      40434    40436       +2     
============================================
+ Hits         198892   198989      +97     
+ Misses        64945    64786     -159     
- Partials      16212    16312     +100

Impacted Files	Coverage Δ
...n/java/org/opensearch/index/engine/NoOpEngine.java	`63.01% <0.00%> (-0.88%)`	⬇️
...opensearch/index/translog/NoOpTranslogManager.java	`64.51% <0.00%> (-2.16%)`	⬇️
.../org/opensearch/index/translog/TranslogWriter.java	`73.51% <ø> (-0.40%)`	⬇️
...rg/opensearch/index/translog/RemoteFsTranslog.java	`72.26% <52.63%> (-1.47%)`	⬇️
...search/index/shard/RemoteStoreRefreshListener.java	`79.81% <100.00%> (+0.57%)`	⬆️
...search/index/translog/InternalTranslogManager.java	`71.42% <100.00%> (+0.41%)`	⬆️
...n/java/org/opensearch/index/translog/Translog.java	`79.75% <100.00%> (+0.02%)`	⬆️
...ex/translog/transfer/BlobStoreTransferService.java	`86.66% <100.00%> (+0.95%)`	⬆️
...h/index/translog/transfer/FileTransferTracker.java	`88.23% <100.00%> (+3.86%)`	⬆️
...dex/translog/transfer/TranslogTransferManager.java	`82.97% <100.00%> (+1.36%)`	⬆️
... and 489 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

ashking94 · 2023-01-09T11:15:26Z

server/src/main/java/org/opensearch/index/translog/RemoteFsTranslog.java

@@ -47,6 +47,8 @@ public class RemoteFsTranslog extends Translog {
    private final FileTransferTracker fileTransferTracker;
    private volatile long maxRemoteTranslogGenerationUploaded;

+    private volatile long minSeqNoToKeep;


The default value of 0 should be good when minSeqNoToKeep has not been set ever?

Could there be a case when a replica-primary promotion has happened and this has not been initialised or primary-primary relocation (peer recovery) in which case this can be problematic?

In worst case, the default value of 0 would prevent further truncations. There would be some assertion failures , which would be disabled in prod.

github-actions · 2023-01-09T11:28:15Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9066/
CommitID: 463c931
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Bukhtawar · 2023-01-09T17:15:13Z

server/src/main/java/org/opensearch/index/translog/RemoteFsTranslog.java

+    protected void setMinSeqNoToKeep(long seqNo) {
+        if (seqNo < this.minSeqNoToKeep) {
+            throw new IllegalArgumentException(
+                "min seq number required can't go backwards: " + "current [" + this.minSeqNoToKeep + "] new [" + seqNo + "]"
+            );
+        }
+        this.minSeqNoToKeep = seqNo;


For my understanding min should be able to go lower right?

No, once it is set to x +10, then we should not be able to set it to x ,as 0..x operations could have been deleted by then . We can set it to x+20 after that .

server/src/main/java/org/opensearch/index/translog/RemoteFsTranslog.java

Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>

github-actions · 2023-01-10T10:17:56Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9158/
CommitID: 3186594
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>

github-actions · 2023-01-10T12:18:05Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9171/
CommitID: ee4f238
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>

github-actions · 2023-01-10T13:35:01Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9179/
CommitID: b6d72e3
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

gbbafna · 2023-01-10T14:29:12Z

Created #5791 as per @Bukhtawar's suggestion .

…g older tlog files (opensearch-project#5662) * RemoteFSTranslog Trimming and GC Logic Signed-off-by: Gaurav Bafna <gbbafna@amazon.com> Signed-off-by: Sachin Kale <kalsac@amazon.com>

…g older tlog files (#5662) (#5793) * RemoteFSTranslog Trimming and GC Logic Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>

…g older tlog files (opensearch-project#5662) (opensearch-project#5793) * RemoteFSTranslog Trimming and GC Logic Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>

…g older tlog files (#5662) (#5792) * RemoteFSTranslog Trimming and GC Logic Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>

gbbafna requested review from reta, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, setiah, kartg, kotwanikunal, mch2, nknize, owaiskazi19, adnapibar, Rishikesh1159, ryanbogan, saratvemulapalli, shwetathareja, dreamer-89, tlfeng, VachaShah and xuezhou25 as code owners December 30, 2022 14:16

gbbafna marked this pull request as draft December 30, 2022 14:16

gbbafna added the skip-changelog label Jan 2, 2023

gbbafna force-pushed the remote-txlog-minGen branch 2 times, most recently from a8e3ba0 to e1f2c43 Compare January 2, 2023 09:14

ashking94 reviewed Jan 9, 2023

View reviewed changes

Bukhtawar reviewed Jan 9, 2023

View reviewed changes

server/src/main/java/org/opensearch/index/translog/RemoteFsTranslog.java Outdated Show resolved Hide resolved

gbbafna added 6 commits January 10, 2023 15:12

RemoteFSTranslog Trimming and GC Logic

54c2989

Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>

Removing files from FileTransferTracker as well

c12708d

Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>

Updating minSeqNo only in case of succesful segment upload

b6ee694

Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>

Addressing PR comments

904c556

Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>

PR comments

6a68c12

Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>

Fixing log

3186594

Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>

gbbafna force-pushed the remote-txlog-minGen branch from 463c931 to 3186594 Compare January 10, 2023 10:08

Fix issues in merge

ee4f238

Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>

sachinpkale mentioned this pull request Jan 10, 2023

[Remote Translog] Reduce number of operations in testConcurrentWriteViewsAndSnapshot #5789

Merged

6 tasks

Fix spotless errors

b6d72e3

Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>

Bukhtawar approved these changes Jan 10, 2023

View reviewed changes

gbbafna merged commit ff8a3af into opensearch-project:main Jan 10, 2023

sachinpkale mentioned this pull request Jan 10, 2023

[Backport 2.x] [Remote Translog] Trimming based on remote segment upload and cleanin… #5792

Merged

6 tasks

sachinpkale mentioned this pull request Jan 10, 2023

[Backport 2.5][Remote Translog] Trimming based on remote segment upload and cleanin… #5793

Merged

6 tasks

gbbafna pushed a commit that referenced this pull request Jan 10, 2023

[Remote Translog] Trimming based on remote segment upload and cleanin…

dcafb10

…g older tlog files (#5662) (#5793) * RemoteFSTranslog Trimming and GC Logic Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>

gbbafna mentioned this pull request Jan 10, 2023

[Backport 2.x] Remote Translog Trimming based on remote segment upload and cleanin… #5796

Closed

6 tasks

gbbafna pushed a commit that referenced this pull request Jan 10, 2023

[Remote Translog] Trimming based on remote segment upload and cleanin…

36a4727

…g older tlog files (#5662) (#5792) * RemoteFSTranslog Trimming and GC Logic Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>

kotwanikunal pushed a commit that referenced this pull request Jan 25, 2023

[Remote Translog] Trimming based on remote segment upload and cleanin…

aa0802b

…g older tlog files (#5662) (#5792) * RemoteFSTranslog Trimming and GC Logic Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Remote Translog] Trimming based on remote segment upload and cleaning older tlog files #5662

[Remote Translog] Trimming based on remote segment upload and cleaning older tlog files #5662

gbbafna commented Dec 30, 2022 •

edited

Loading

github-actions bot commented Dec 30, 2022

github-actions bot commented Jan 2, 2023

github-actions bot commented Jan 2, 2023

github-actions bot commented Jan 2, 2023

codecov-commenter commented Jan 2, 2023 •

edited

Loading

ashking94 Jan 9, 2023

ashking94 Jan 9, 2023

gbbafna Jan 10, 2023

github-actions bot commented Jan 9, 2023

Bukhtawar Jan 9, 2023

gbbafna Jan 10, 2023

github-actions bot commented Jan 10, 2023

github-actions bot commented Jan 10, 2023

github-actions bot commented Jan 10, 2023

gbbafna commented Jan 10, 2023

[Remote Translog] Trimming based on remote segment upload and cleaning older tlog files #5662

[Remote Translog] Trimming based on remote segment upload and cleaning older tlog files #5662

Conversation

gbbafna commented Dec 30, 2022 • edited Loading

Description

Issues Resolved

Check List

github-actions bot commented Dec 30, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 2, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 2, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 2, 2023

Gradle Check (Jenkins) Run Completed with:

codecov-commenter commented Jan 2, 2023 • edited Loading

Codecov Report

ashking94 Jan 9, 2023

Choose a reason for hiding this comment

ashking94 Jan 9, 2023

Choose a reason for hiding this comment

gbbafna Jan 10, 2023

Choose a reason for hiding this comment

github-actions bot commented Jan 9, 2023

Gradle Check (Jenkins) Run Completed with:

Bukhtawar Jan 9, 2023

Choose a reason for hiding this comment

gbbafna Jan 10, 2023

Choose a reason for hiding this comment

github-actions bot commented Jan 10, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 10, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 10, 2023

Gradle Check (Jenkins) Run Completed with:

gbbafna commented Jan 10, 2023

gbbafna commented Dec 30, 2022 •

edited

Loading

codecov-commenter commented Jan 2, 2023 •

edited

Loading