Skip to content

[improve][client] Avoid recomputing entry-bucket hashes - #26383

Open
3pacccccc wants to merge 1 commit into
apache:masterfrom
3pacccccc:optimize-entry-bucket-batch-hash
Open

[improve][client] Avoid recomputing entry-bucket hashes#26383
3pacccccc wants to merge 1 commit into
apache:masterfrom
3pacccccc:optimize-entry-bucket-batch-hash

Conversation

@3pacccccc

Copy link
Copy Markdown
Contributor

Motivation

Entry-bucket batching already computes each message's entry-bucket hash when it selects the bucket for that message. However, while building the batch send operation, BatchMessageContainerImpl#createOpSendMsg() recomputed
the same hash for every message in the batch in order to stamp entry_hash_min and entry_hash_max.

This means the hash of each message was calculated twice during the batching path:

  1. once in EntryBucketBatchContainer#add() for bucket selection;
  2. again in BatchMessageContainerImpl#createOpSendMsg() for metadata stamping.

For producers using entry-bucket batching, this duplicated hashing work on every batch flush and made the hashing cost proportional to the batch size twice instead of once.

Modifications

  • Compute the entry-bucket hash once in EntryBucketBatchContainer#add().

  • Pass that already-computed hash to BatchMessageContainerImpl through a package-private overload:

    add(MessageImpl<?> msg, SendCallback callback, int entryBucketHash)
  • Maintain the batch-level entry-bucket hash minimum and maximum incrementally while messages are added.

  • Replace the previous hash function field with a boolean flag indicating whether entry-bucket hash stamping is enabled.

  • Stamp entry_hash_min and entry_hash_max from the maintained minimum and maximum values instead of recomputing hashes for all messages.

  • Reset the maintained minimum and maximum values when the batch container is cleared.

  • Fail fast if the two-argument add() method is used on a container that requires a precomputed entry-bucket hash.

  • Add unit tests covering:

    • incremental hash range maintenance;
    • metadata stamping;
    • state reset after the batch is discarded;
    • disabled entry-bucket hash stamping.

The additional add() overload is package-private on a package-private implementation class and is only used by EntryBucketBatchContainer; it does not change the public client API.

Verifying this change

  • Make sure that the change passes the CI checks.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

Matching PR in forked repository

PR in forked repository: 3pacccccc#39

Entry-bucket batching already computes each message's entry-bucket hash to select its bucket. Recomputing the hash for every message while building the send operation duplicated that work on every flush.

Maintain the batch's entry-hash minimum and maximum while messages are added, pass the already-computed hash into the batch container, and stamp metadata from the maintained range. Add a unit test for range maintenance and reset.

Assisted-by: Codex
@lhotari lhotari added this to the 5.0.0-M2 milestone Aug 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants