Created new urgent priority threadpool for remote cluster state uploads #10685

shiv0408 · 2023-10-18T08:09:04Z

Description

Created a new Urgent WritePriority and created new S3 threadpool and client to upload the remote cluster states.

Testing

Spun up a new single node cluster with remote cluster state enabled and ran few operations which requires cluster state update.

We can see following threadpool stats in nodes/_stats response after that.

              "urgent_future_completion": {
                    "threads": 1,
                    "queue": 0,
                    "active": 0,
                    "rejected": 0,
                    "largest": 1,
                    "completed": 7
                },
                "urgent_stream_reader": {
                    "threads": 1,
                    "queue": 0,
                    "active": 0,
                    "rejected": 0,
                    "largest": 1,
                    "completed": 7
                },

Related Issues

Resolves #9816

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)
GitHub issue/PR created in OpenSearch documentation repo for the required public documentation changes (#[Issue/PR number])

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…or this priority which is being used to upload cluster state Signed-off-by: Shivansh Arora <shivansh.arora@protonmail.com>

Signed-off-by: Shivansh Arora <shivansh.arora@protonmail.com>

github-actions · 2023-10-18T08:12:18Z

Gradle Check (Jenkins) Run Completed with:

RESULT: ❌
URL:
CommitID: 00c29ed
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-10-18T08:52:18Z

Compatibility status:

Checks if related components are compatible with change a636b9b

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git]

Signed-off-by: Shivansh Arora <shivansh.arora@protonmail.com>

github-actions · 2023-10-18T10:14:26Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/28221/
CommitID: 967c4e1

codecov · 2023-10-18T10:15:19Z

Codecov Report

Merging #10685 (967c4e1) into main (7453daa) will decrease coverage by 0.02%.
Report is 2 commits behind head on main.
The diff coverage is 58.69%.

❗ Current head 967c4e1 differs from pull request most recent head a636b9b. Consider uploading reports for the commit a636b9b to get more accurate results

@@             Coverage Diff              @@
##               main   #10685      +/-   ##
============================================
- Coverage     71.22%   71.21%   -0.02%     
+ Complexity    58643    58507     -136     
============================================
  Files          4869     4853      -16     
  Lines        276492   275826     -666     
  Branches      40204    40142      -62     
============================================
- Hits         196944   196442     -502     
+ Misses        63040    62908     -132     
+ Partials      16508    16476      -32

Files	Coverage Δ
...search/repositories/s3/AmazonAsyncS3Reference.java	`84.61% <100.00%> (+1.28%)`	⬆️
.../repositories/s3/AmazonAsyncS3WithCredentials.java	`100.00% <100.00%> (ø)`
...org/opensearch/repositories/s3/S3AsyncService.java	`72.28% <100.00%> (+1.09%)`	⬆️
...va/org/opensearch/repositories/s3/S3BlobStore.java	`74.57% <100.00%> (+6.32%)`	⬆️
...a/org/opensearch/repositories/s3/S3Repository.java	`70.08% <100.00%> (+0.25%)`	⬆️
...h/common/blobstore/stream/write/WritePriority.java	`100.00% <100.00%> (ø)`
...epositories/blobstore/ChecksumBlobStoreFormat.java	`95.00% <ø> (-0.17%)`	⬇️
...rg/opensearch/repositories/s3/S3BlobContainer.java	`80.73% <60.00%> (+0.05%)`	⬆️
...ch/repositories/s3/async/AsyncTransferManager.java	`82.43% <71.42%> (+0.31%)`	⬆️
...earch/repositories/s3/async/AsyncPartsHandler.java	`85.18% <20.00%> (+1.85%)`	⬆️
... and 1 more

... and 535 files with indirect coverage changes

server/src/main/java/org/opensearch/repositories/blobstore/ChecksumBlobStoreFormat.java

Signed-off-by: Shivansh Arora <shivansh.arora@protonmail.com>

github-actions · 2023-10-22T20:02:45Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/28743/
CommitID: fef26e5
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Shivansh Arora <shivansh.arora@protonmail.com>

github-actions · 2023-10-22T20:28:41Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/28744/
CommitID: 1311929
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-10-23T08:15:05Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/28785/
CommitID: 1311929
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/repositories/blobstore/ChecksumBlobStoreFormat.java

Signed-off-by: Shivansh Arora <shivansh.arora@protonmail.com>

github-actions · 2023-10-23T09:29:24Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/28795/
CommitID: fc95922
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Shivansh Arora <shivansh.arora@protonmail.com>

github-actions · 2023-10-23T10:34:51Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/28804/
CommitID: a636b9b
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-10-23T11:48:22Z

Gradle Check (Jenkins) Run Completed with:

RESULT: TIMEOUT ❌
URL: https://build.ci.opensearch.org/job/gradle-check/28803/
CommitID: fc95922
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-10-23T15:29:52Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/28837/
CommitID: a636b9b
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

shwetathareja · 2023-10-23T15:56:40Z

Flaky test
#10154

github-actions · 2023-10-23T16:51:03Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/28851/
CommitID: a636b9b
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

shwetathareja · 2023-10-24T08:28:16Z

Snapshot test failing - org.opensearch.snapshots.DedicatedClusterSnapshotRestoreIT.testIndexDeletionDuringSnapshotCreationInQueue

#5031

…ds (#10685) * Created new Urgent WritePriority and a new threadpools and S3Client for this priority which is being used to upload cluster state Signed-off-by: Shivansh Arora <shivansh.arora@protonmail.com> (cherry picked from commit a2b5f0e) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ds (#10685) (#10879) * Created new Urgent WritePriority and a new threadpools and S3Client for this priority which is being used to upload cluster state (cherry picked from commit a2b5f0e) Signed-off-by: Shivansh Arora <shivansh.arora@protonmail.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ds (opensearch-project#10685) * Created new Urgent WritePriority and a new threadpools and S3Client for this priority which is being used to upload cluster state Signed-off-by: Shivansh Arora <shivansh.arora@protonmail.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

shiv0408 added 2 commits October 18, 2023 13:37

Created new Urgent WritePriority and a new threadpools and S3Client f…

844fdf9

…or this priority which is being used to upload cluster state Signed-off-by: Shivansh Arora <shivansh.arora@protonmail.com>

Fixed UTs

00c29ed

Signed-off-by: Shivansh Arora <shivansh.arora@protonmail.com>

github-actions bot added Cluster Manager enhancement Enhancement or improvement to existing feature or request v2.11.0 Issues and PRs related to version 2.11.0 labels Oct 18, 2023

Spotless change

967c4e1

Signed-off-by: Shivansh Arora <shivansh.arora@protonmail.com>

shiv0408 marked this pull request as ready for review October 19, 2023 03:50

shiv0408 requested review from abbashus, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, dbwiddis, dreamer-89, gbbafna, kartg, kotwanikunal, mch2, msfroh, nknize, owaiskazi19, peternied, reta, Rishikesh1159 and ryanbogan as code owners October 19, 2023 03:50

shwetathareja reviewed Oct 22, 2023

View reviewed changes

server/src/main/java/org/opensearch/repositories/blobstore/ChecksumBlobStoreFormat.java Outdated Show resolved Hide resolved

shiv0408 added 3 commits October 22, 2023 23:29

Added new wrapper methods for writeAsync to different WritePriorities

8dbd416

Signed-off-by: Shivansh Arora <shivansh.arora@protonmail.com>

Merge remote-tracking branch 'origin/main' into remote-state-threadpool

ead15a8

Added UT for urgentWriteAsync

fef26e5

Signed-off-by: Shivansh Arora <shivansh.arora@protonmail.com>

spotless changes

1311929

Signed-off-by: Shivansh Arora <shivansh.arora@protonmail.com>

linuxpi approved these changes Oct 23, 2023

View reviewed changes

shwetathareja reviewed Oct 23, 2023

View reviewed changes

server/src/main/java/org/opensearch/repositories/blobstore/ChecksumBlobStoreFormat.java Outdated Show resolved Hide resolved

Renamed urgent write method

fc95922

Signed-off-by: Shivansh Arora <shivansh.arora@protonmail.com>

shwetathareja approved these changes Oct 23, 2023

View reviewed changes

Merge remote-tracking branch 'origin' into remote-state-threadpool

a636b9b

Signed-off-by: Shivansh Arora <shivansh.arora@protonmail.com>

shwetathareja mentioned this pull request Oct 23, 2023

[BUG] Flaky org.opensearch.search.aggregations.metrics.CardinalityWithRequestBreakerIT test #10154

Closed

shwetathareja mentioned this pull request Oct 24, 2023

[BUG] org.opensearch.snapshots.DedicatedClusterSnapshotRestoreIT.testIndexDeletionDuringSnapshotCreationInQueue flaky #5031

Closed

shwetathareja merged commit a2b5f0e into opensearch-project:main Oct 24, 2023

shwetathareja added backport 2.x Backport to 2.x branch and removed v2.11.0 Issues and PRs related to version 2.11.0 labels Oct 24, 2023

opensearch-trigger-bot bot mentioned this pull request Oct 24, 2023

[Backport 2.x] Created new urgent priority threadpool for remote cluster state uploads #10879

Merged

shiv0408 deleted the remote-state-threadpool branch October 25, 2023 04:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Created new urgent priority threadpool for remote cluster state uploads #10685

Created new urgent priority threadpool for remote cluster state uploads #10685

shiv0408 commented Oct 18, 2023 •

edited

Loading

github-actions bot commented Oct 18, 2023

github-actions bot commented Oct 18, 2023 •

edited

Loading

github-actions bot commented Oct 18, 2023

codecov bot commented Oct 18, 2023 •

edited

Loading

github-actions bot commented Oct 22, 2023

github-actions bot commented Oct 22, 2023

github-actions bot commented Oct 23, 2023

github-actions bot commented Oct 23, 2023

github-actions bot commented Oct 23, 2023

github-actions bot commented Oct 23, 2023

github-actions bot commented Oct 23, 2023

shwetathareja commented Oct 23, 2023

github-actions bot commented Oct 23, 2023

shwetathareja commented Oct 24, 2023

Created new urgent priority threadpool for remote cluster state uploads #10685

Created new urgent priority threadpool for remote cluster state uploads #10685

Conversation

shiv0408 commented Oct 18, 2023 • edited Loading

Description

Testing

Related Issues

Check List

github-actions bot commented Oct 18, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 18, 2023 • edited Loading

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Oct 18, 2023

Gradle Check (Jenkins) Run Completed with:

codecov bot commented Oct 18, 2023 • edited Loading

Codecov Report

github-actions bot commented Oct 22, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 22, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 23, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 23, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 23, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 23, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 23, 2023

Gradle Check (Jenkins) Run Completed with:

shwetathareja commented Oct 23, 2023

github-actions bot commented Oct 23, 2023

Gradle Check (Jenkins) Run Completed with:

shwetathareja commented Oct 24, 2023

shiv0408 commented Oct 18, 2023 •

edited

Loading

github-actions bot commented Oct 18, 2023 •

edited

Loading

codecov bot commented Oct 18, 2023 •

edited

Loading