Removed timeout excessive logging in case of index is idle in replication #1114

mohitamg · 2023-09-04T04:41:43Z

Description

Relevant exceptions(5xx) are retried under the common module and during this time whole stack trace is logged under "WARN".
For certain operations, this can create excessive logging and creates noise.
Removed this log.warn message

Issues Resolved

Resolved #267

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…tion Signed-off-by: Mohit Kumar <mohitamg@amazon.com>

codecov · 2023-09-04T04:46:23Z

Codecov Report

Merging #1114 (54f7164) into main (b6d1b56) will increase coverage by 0.81%.
The diff coverage is 100.00%.

❗ Current head 54f7164 differs from pull request most recent head 72c05b3. Consider uploading reports for the commit 72c05b3 to get more accurate results

@@             Coverage Diff              @@
##               main    #1114      +/-   ##
============================================
+ Coverage     74.31%   75.13%   +0.81%     
  Complexity     1021     1021              
============================================
  Files           141      141              
  Lines          4762     4761       -1     
  Branches        521      521              
============================================
+ Hits           3539     3577      +38     
+ Misses          883      850      -33     
+ Partials        340      334       -6

Files Changed	Coverage Δ
...tlin/org/opensearch/replication/util/Extensions.kt	`63.33% <100.00%> (-0.41%)`	⬇️

... and 6 files with indirect coverage changes

ankitkala · 2023-09-04T04:56:35Z

src/main/kotlin/org/opensearch/replication/util/Extensions.kt

@@ -137,10 +137,6 @@ suspend fun <Req: ActionRequest, Resp: ActionResponse> Client.suspendExecuteWith
                throw ReplicationException(e, RestStatus.TOO_MANY_REQUESTS)
            }
        }
-        log.warn(
-            "Encountered a failure while executing in $req. Retrying in ${currentBackoff / 1000} seconds" +


Instead of deleting the complete log entry, should we explore keeping it and only remove the stack trace?
We can just print the error message maybe.

+1, this messages comes handy when troubleshooting failures.

removed stack trace, but tried getting exception via retryException.cause, retryException.message and retryException.toString() but all of them returned empty, hence the exception message

[2023-09-04T13:54:45,047][WARN ][o.o.r.t.s.ShardReplicationTask] [followCluster-1] [follower-01][0] Encountered a failure while executing changes. Retrying in 10 seconds.OpenSearchTimeoutException can be ignored!! [2023-09-04T13:56:55,095][WARN ][o.o.r.t.s.ShardReplicationTask] [followCluster-1] [follower-01][0] Encountered a failure while executing changes. Retrying in 20 seconds.OpenSearchTimeoutException can be ignored!! [2023-09-04T13:59:15,166][WARN ][o.o.r.t.s.ShardReplicationTask] [followCluster-1] [follower-01][0] Encountered a failure while executing changes. Retrying in 40 seconds.OpenSearchTimeoutException can be ignored!! [2023-09-04T14:01:55,209][WARN ][o.o.r.t.s.ShardReplicationTask] [followCluster-1] [follower-01][0] Encountered a failure while executing changes. Retrying in 80 seconds.OpenSearchTimeoutException can be ignored!! [2023-09-04T14:05:15,259][INFO ][o.o.r.t.s.ShardReplicationTask] [followCluster-1] [follower-01][0] opensearch[followCluster-1][replication_follower][T#3] @coroutine#7: Timed out waiting for new changes. Current seqNo: 1. OpenSearchTimeoutException[1m]

by the way , last log print is already printing out OpenSearchTimeoutException, so its fine

Let's keep it as Encountered a failure(can be ignored) while getting changes: OpenSearchTimeoutException. Retrying in ...

[2023-09-04T14:57:33,124][WARN ][o.o.r.t.s.ShardReplicationTask] [followCluster-1] [follower-01][0] Encountered a failure(can be ignored) while getting changes: OpenSearchTimeoutException. Retrying in 10 seconds. <===========--> 90% EXECUTING [5m 35s] [2023-09-04T14:59:43,157][WARN ][o.o.r.t.s.ShardReplicationTask] [followCluster-1] [follower-01][0] Encountered a failure(can be ignored) while getting changes: OpenSearchTimeoutException. Retrying in 20 seconds. <===========--> 90% EXECUTING [5m 40s] <===========--> 90% EXECUTING [7m 55s] [2023-09-04T15:02:03,216][WARN ][o.o.r.t.s.ShardReplicationTask] [followCluster-1] [follower-01][0] Encountered a failure(can be ignored) while getting changes: OpenSearchTimeoutException. Retrying in 40 seconds. [2023-09-04T15:04:43,262][WARN ][o.o.r.t.s.ShardReplicationTask] [followCluster-1] [follower-01][0] Encountered a failure(can be ignored) while getting changes: OpenSearchTimeoutException. Retrying in 80 seconds. [2023-09-04T15:08:03,284][INFO ][o.o.r.t.s.ShardReplicationTask] [followCluster-1] [follower-01][0] opensearch[followCluster-1][replication_follower][T#2] @coroutine#7: Timed out waiting for new changes. Current seqNo: 1. OpenSearchTimeoutException[1m]

…e in replication Signed-off-by: Mohit Kumar <mohitamg@amazon.com>

Signed-off-by: Mohit Kumar <mohitamg@amazon.com>

…tion (#1114) * Removed timeout excessive logging in case of index is idle in replication Signed-off-by: Mohit Kumar <mohitamg@amazon.com> * Removed timeout excessive logging in case of index is idle in replication Signed-off-by: Mohit Kumar <mohitamg@amazon.com> * Removed timeout excessive stack trace logging in case of index is idle in replication Signed-off-by: Mohit Kumar <mohitamg@amazon.com> * Removed timeout excessive stack trace logging in case of index is idle in replication Signed-off-by: Mohit Kumar <mohitamg@amazon.com> * Changed the log statement Signed-off-by: Mohit Kumar <mohitamg@amazon.com> --------- Signed-off-by: Mohit Kumar <mohitamg@amazon.com> (cherry picked from commit 426a2de)

…tion (#1114) (#1115) * Removed timeout excessive logging in case of index is idle in replication Signed-off-by: Mohit Kumar <mohitamg@amazon.com> * Removed timeout excessive logging in case of index is idle in replication Signed-off-by: Mohit Kumar <mohitamg@amazon.com> * Removed timeout excessive stack trace logging in case of index is idle in replication Signed-off-by: Mohit Kumar <mohitamg@amazon.com> * Removed timeout excessive stack trace logging in case of index is idle in replication Signed-off-by: Mohit Kumar <mohitamg@amazon.com> * Changed the log statement Signed-off-by: Mohit Kumar <mohitamg@amazon.com> --------- Signed-off-by: Mohit Kumar <mohitamg@amazon.com> (cherry picked from commit 426a2de) Co-authored-by: Mohit Kumar <113413713+mohitamg@users.noreply.github.com>

mohitamg added 2 commits September 4, 2023 09:59

Removed timeout excessive logging in case of index is idle in replica…

6e70e7a

…tion Signed-off-by: Mohit Kumar <mohitamg@amazon.com>

Removed timeout excessive logging in case of index is idle in replica…

a2d87d9

…tion Signed-off-by: Mohit Kumar <mohitamg@amazon.com>

mohitamg requested review from krishna-ggk, ankitkala, saikaranam-amazon, soosinha, gbbafna and monusingh-1 as code owners September 4, 2023 04:41

ankitkala reviewed Sep 4, 2023

View reviewed changes

mohitamg added 3 commits September 4, 2023 13:50

Removed timeout excessive stack trace logging in case of index is idl…

60b1d54

…e in replication Signed-off-by: Mohit Kumar <mohitamg@amazon.com>

Removed timeout excessive stack trace logging in case of index is idl…

db42dee

…e in replication Signed-off-by: Mohit Kumar <mohitamg@amazon.com>

Changed the log statement

72c05b3

Signed-off-by: Mohit Kumar <mohitamg@amazon.com>

ankitkala approved these changes Sep 4, 2023

View reviewed changes

monusingh-1 enabled auto-merge (squash) September 4, 2023 10:08

monusingh-1 approved these changes Sep 4, 2023

View reviewed changes

monusingh-1 merged commit 426a2de into opensearch-project:main Sep 4, 2023
10 of 11 checks passed

monusingh-1 added the backport 2.x label Sep 4, 2023

opensearch-trigger-bot bot mentioned this pull request Sep 4, 2023

[Backport 2.x] Removed timeout excessive logging in case of index is idle in replication #1115

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Removed timeout excessive logging in case of index is idle in replication #1114

Removed timeout excessive logging in case of index is idle in replication #1114

mohitamg commented Sep 4, 2023

codecov bot commented Sep 4, 2023 •

edited

Loading

ankitkala Sep 4, 2023

monusingh-1 Sep 4, 2023

mohitamg Sep 4, 2023

mohitamg Sep 4, 2023 •

edited

Loading

mohitamg Sep 4, 2023

ankitkala Sep 4, 2023

mohitamg Sep 4, 2023 •

edited

Loading

Removed timeout excessive logging in case of index is idle in replication #1114

Removed timeout excessive logging in case of index is idle in replication #1114

Conversation

mohitamg commented Sep 4, 2023

Description

Issues Resolved

Check List

codecov bot commented Sep 4, 2023 • edited Loading

Codecov Report

ankitkala Sep 4, 2023

Choose a reason for hiding this comment

monusingh-1 Sep 4, 2023

Choose a reason for hiding this comment

mohitamg Sep 4, 2023

Choose a reason for hiding this comment

mohitamg Sep 4, 2023 • edited Loading

Choose a reason for hiding this comment

mohitamg Sep 4, 2023

Choose a reason for hiding this comment

ankitkala Sep 4, 2023

Choose a reason for hiding this comment

mohitamg Sep 4, 2023 • edited Loading

Choose a reason for hiding this comment

codecov bot commented Sep 4, 2023 •

edited

Loading

mohitamg Sep 4, 2023 •

edited

Loading

mohitamg Sep 4, 2023 •

edited

Loading