Add Async Batch Shard Settings #7056
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Pranshu-S <pranshushukla06@gmail.com>
Pranshu-S
requested review from
AMoo-Miki,
Naarcha-AWS,
dlvenable,
epugh,
hdhalter,
kolchfa-aws,
natebower,
stephen-crawford and
vagimeli
as code owners
|
Thanks, @Pranshu-S ! Are these changes being released in 2.14 or should they be backported to 2.13? |
This will be for 2.14. |
hdhalter
added
v2.14.0
release-notes
Naarcha-AWS
suggested changes
May 2, 2024
_install-and-configure/configuring-opensearch/cluster-settings.md
Outdated
Show resolved
Hide resolved
_install-and-configure/configuring-opensearch/cluster-settings.md
Outdated
Show resolved
Hide resolved
Signed-off-by: Pranshu-S <pranshushukla06@gmail.com>
| - `REPLICA_FIRST` – Replica shards are relocated first, before primary shards. This prioritization may help prevent a cluster's health status from going red when carrying out shard relocation in a mixed-version, segment-replication-enabled OpenSearch cluster. In this situation, primary shards relocated to OpenSearch nodes of a newer version could try to copy segment files to replica shards on an older version of OpenSearch, which would result in shard failure. Relocating replica shards first may help to avoid this in multi-version clusters. | ||
| - `NO_PREFERENCE` – The default behavior in which the order of shard relocation has no importance. | ||
|
|
||
| - `cluster.allocator.gateway.batch_size` (Integer): Limits the number of shards sent in one batch to data nodes for fetching metadata. Default is `2000`. |
There was a problem hiding this comment.
can we change it to -> Limits the number of shards sent in one batch to data nodes for fetching unassigned shard metadata
| - `NO_PREFERENCE` – The default behavior in which the order of shard relocation has no importance. | ||
|
|
||
| - `cluster.allocator.gateway.batch_size` (Integer): Limits the number of shards sent in one batch to data nodes for fetching metadata. Default is `2000`. | ||
| - `cluster.allocator.existing_shards_allocator.batch_enabled` (Boolean): Enables batch allocation of unassigned shards already existing on the disk. This allows for sending all `Unassigned`shards to the `ExistingShard` allocator. The allocator decides how to allocate the unassigned shards. Default is `False`. |
There was a problem hiding this comment.
Enables batch allocation of unassigned shards already existing on the disk as opposed to one shard at a time. This reduces the memory and transport overhead by fetching unassigned shard metadata in a batch call. Default is False.
There was a problem hiding this comment.
Updated the wordings.
Signed-off-by: Pranshu-S <pranshushukla06@gmail.com>
Naarcha-AWS
suggested changes
May 6, 2024
_install-and-configure/configuring-opensearch/cluster-settings.md
Outdated
Show resolved
Hide resolved
_install-and-configure/configuring-opensearch/cluster-settings.md
Outdated
Show resolved
Hide resolved
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Naarcha-AWS
reviewed
May 7, 2024
_install-and-configure/configuring-opensearch/cluster-settings.md
Outdated
Show resolved
Hide resolved
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Naarcha-AWS
approved these changes
May 7, 2024
Naarcha-AWS
added
Editorial review
natebower
reviewed
May 7, 2024
There was a problem hiding this comment.
@Pranshu-S @Naarcha-AWS Just a couple comments/changes. Thanks!
_install-and-configure/configuring-opensearch/cluster-settings.md
Outdated
Show resolved
Hide resolved
_install-and-configure/configuring-opensearch/cluster-settings.md
Outdated
Show resolved
Hide resolved
Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
hdhalter
added
3 - Done
and removed
Editorial review
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Adds documentation for enabling Async Batch Fetch on a cluster and setting Batch sizes to the gateway allocators as per the changes/optimisations done on Async Shard Fetch flow
Issues Resolved
Resolves #7055
Checklist
For more information on following Developer Certificate of Origin and signing off your commits, please check here.