[Remote Store] Fix flaky test RemoteStoreIT.testRemoteSegmentStoreRestore #6252

sachinpkale · 2023-02-09T02:19:57Z

Signed-off-by: Sachin Kale kalsac@amazon.com

Description

In RemoteStoreIT.testRemoteSegmentStoreRestore test, if we index X documents and primary goes down without refresh, as we are using only remote segment store, restore operation creates as empty index.
Indexing new doc on the restored index is treated as the first doc and response contains seqNo as 0 (for the first doc).
The seqNo initialization to 0 instead of -1 was failing the test which is fixed with this change.

Issues Resolved

[BUG] Flaky test org.opensearch.remotestore.RemoteStoreIT.testRemoteStoreRestoreFromRemoteSegmentStore #6248

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

sachinpkale · 2023-02-09T02:23:47Z

server/src/internalClusterTest/java/org/opensearch/remotestore/RemoteStoreIT.java

-    public void testRemoteStoreRestoreFromRemoteSegmentStore() throws IOException {
-        internalCluster().startNodes(3);
+    public void testRemoteSegmentStoreRestore() throws IOException {
+        internalCluster().startDataOnlyNodes(3);


This is not a fix to flakiness that was observed in #6248.

It is just to harden the test so that primary is not the cluster manager node as well.

sachinpkale · 2023-02-09T02:24:42Z

server/src/internalClusterTest/java/org/opensearch/search/SearchWeightedRoutingIT.java

     */
-    public void testStrictWeightedRouting() throws Exception {
+    public void testStrictWeightedRouting() {


These un-documented @throws created issues while running the tests on local. Not related to the fix of the flaky test.

@sachinpkale Which JDK are you using locally? I've seen inconsistencies with some non-LTS versions with regard to javadoc enforcement so I'd recommend using 11 or 17 if that is the cause here.

@andrross the javadoc enforcement should not apply over test code, do you see checks enforced somewhere?

@reta Just from this comment: #5995 (comment) I have no idea why JDK 14 seems to want to enforce javadoc rules against tests, but that's what Tianli's comment suggests is happening.

Oh I see, thanks @andrross, I think we agreed to discard JDK-14 issues (and unsupported JDKs in general)

github-actions · 2023-02-09T02:49:12Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/10897/
CommitID: 5975509
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

sachinpkale · 2023-02-09T03:10:37Z

Failing due to flaky test: #6255

Pushing empty commit to re-trigger the build.

github-actions · 2023-02-09T03:41:29Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/10901/
CommitID: 1e32c73
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-02-09T04:56:40Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/10905/
CommitID: 1e32c73
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

sachinpkale · 2023-02-09T06:37:39Z

server/src/internalClusterTest/java/org/opensearch/remotestore/RemoteStoreIT.java

-        long maxSeqNo = 0;
-        long maxSeqNoRefreshedOrFlushed = 0;
+        long maxSeqNo = -1;
+        long maxSeqNoRefreshedOrFlushed = -1;


This ^ is the fix for flakiness of the test.

Signed-off-by: Sachin Kale <kalsac@amazon.com>

github-actions · 2023-02-09T07:16:19Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/10913/
CommitID: 0b5b1cb

codecov-commenter · 2023-02-09T07:18:28Z

Codecov Report

Merging #6252 (0b5b1cb) into main (1f4cdd2) will increase coverage by 0.04%.
The diff coverage is 38.99%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@             Coverage Diff              @@
##               main    #6252      +/-   ##
============================================
+ Coverage     70.72%   70.76%   +0.04%     
- Complexity    58791    58873      +82     
============================================
  Files          4782     4789       +7     
  Lines        281484   281794     +310     
  Branches      40642    40671      +29     
============================================
+ Hits         199072   199412     +340     
- Misses        65984    66021      +37     
+ Partials      16428    16361      -67

Impacted Files	Coverage Δ
...es/replication/SegmentReplicationStatsRequest.java	`0.00% <0.00%> (ø)`
...ication/SegmentReplicationStatsRequestBuilder.java	`0.00% <0.00%> (ø)`
.../org/opensearch/client/support/AbstractClient.java	`32.00% <0.00%> (-0.31%)`	⬇️
...java/org/opensearch/index/shard/StoreRecovery.java	`65.74% <0.00%> (ø)`
.../indices/replication/SegmentReplicationSource.java	`100.00% <ø> (ø)`
...cation/TransportSegmentReplicationStatsAction.java	`11.11% <11.11%> (ø)`
...s/replication/SegmentReplicationStatsResponse.java	`13.33% <13.33%> (ø)`
...h/indices/replication/SegmentReplicationState.java	`46.96% <24.65%> (-31.98%)`	⬇️
...s/replication/SegmentReplicationTargetService.java	`50.00% <41.66%> (-0.72%)`	⬇️
...ces/replication/PrimaryShardReplicationSource.java	`96.29% <50.00%> (-3.71%)`	⬇️
... and 512 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Signed-off-by: Sachin Kale <kalsac@amazon.com> (cherry picked from commit 4f5fd9c) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…6265) (cherry picked from commit 4f5fd9c) Signed-off-by: Sachin Kale <kalsac@amazon.com>

sachinpkale added the skip-changelog label Feb 9, 2023

sachinpkale mentioned this pull request Feb 9, 2023

[BUG] Flaky test org.opensearch.remotestore.RemoteStoreIT.testRemoteStoreRestoreFromRemoteSegmentStore #6248

Closed

sachinpkale commented Feb 9, 2023

View reviewed changes

sachinpkale force-pushed the remote-store-test-fix branch from 5975509 to 1e32c73 Compare February 9, 2023 03:12

sachinpkale commented Feb 9, 2023

View reviewed changes

Fix flaky test RemoteStoreIT.testRemoteSegmentStoreRestore

0b5b1cb

Signed-off-by: Sachin Kale <kalsac@amazon.com>

sachinpkale force-pushed the remote-store-test-fix branch from 1e32c73 to 0b5b1cb Compare February 9, 2023 06:46

gbbafna approved these changes Feb 9, 2023

View reviewed changes

gbbafna merged commit 4f5fd9c into opensearch-project:main Feb 9, 2023

sachinpkale added the backport 2.x Backport to 2.x branch label Feb 9, 2023

opensearch-trigger-bot bot mentioned this pull request Feb 9, 2023

[Backport 2.x] [Remote Store] Fix flaky test RemoteStoreIT.testRemoteSegmentStoreRestore #6265

Merged

gbbafna pushed a commit that referenced this pull request Feb 9, 2023

Fix flaky test RemoteStoreIT.testRemoteSegmentStoreRestore (#6252) (#…

c0411f3

…6265) (cherry picked from commit 4f5fd9c) Signed-off-by: Sachin Kale <kalsac@amazon.com>

dreamer-89 mentioned this pull request Feb 9, 2023

[BUG] RemoteStoreIT.testRemoteSegmentStoreRestore test failure #6273

Closed

DarshitChanpura mentioned this pull request Apr 18, 2023

[BUG] RemoteStoreIT.testRemoteTranslogRestore flaky test failure #6188

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Remote Store] Fix flaky test RemoteStoreIT.testRemoteSegmentStoreRestore #6252

[Remote Store] Fix flaky test RemoteStoreIT.testRemoteSegmentStoreRestore #6252

sachinpkale commented Feb 9, 2023 •

edited

Loading

sachinpkale Feb 9, 2023

sachinpkale Feb 9, 2023

andrross Feb 9, 2023

reta Feb 9, 2023

andrross Feb 9, 2023

reta Feb 9, 2023 •

edited

Loading

github-actions bot commented Feb 9, 2023

sachinpkale commented Feb 9, 2023

github-actions bot commented Feb 9, 2023

github-actions bot commented Feb 9, 2023

sachinpkale Feb 9, 2023

github-actions bot commented Feb 9, 2023

codecov-commenter commented Feb 9, 2023

[Remote Store] Fix flaky test RemoteStoreIT.testRemoteSegmentStoreRestore #6252

[Remote Store] Fix flaky test RemoteStoreIT.testRemoteSegmentStoreRestore #6252

Conversation

sachinpkale commented Feb 9, 2023 • edited Loading

Description

Issues Resolved

Check List

sachinpkale Feb 9, 2023

Choose a reason for hiding this comment

sachinpkale Feb 9, 2023

Choose a reason for hiding this comment

andrross Feb 9, 2023

Choose a reason for hiding this comment

reta Feb 9, 2023

Choose a reason for hiding this comment

andrross Feb 9, 2023

Choose a reason for hiding this comment

reta Feb 9, 2023 • edited Loading

Choose a reason for hiding this comment

github-actions bot commented Feb 9, 2023

Gradle Check (Jenkins) Run Completed with:

sachinpkale commented Feb 9, 2023

github-actions bot commented Feb 9, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Feb 9, 2023

Gradle Check (Jenkins) Run Completed with:

sachinpkale Feb 9, 2023

Choose a reason for hiding this comment

github-actions bot commented Feb 9, 2023

Gradle Check (Jenkins) Run Completed with:

codecov-commenter commented Feb 9, 2023

Codecov Report

sachinpkale commented Feb 9, 2023 •

edited

Loading

reta Feb 9, 2023 •

edited

Loading