Repository navigation
Conversation
Add HoodieBroadcast and HoodieEngineContext#broadcast so engine-agnostic code can share read-only state with tasks. Spark ships the value's bytes as a broadcast deserialized once per executor; other engines hold the value in the handle.
Simple and global simple lookups run static task functions that capture only the base file, base path, key generator and a broadcast storage configuration. With the metadata table, the latest base files of all requested partitions come from one batched metadata lookup on the driver instead of one task per partition. The simple bucket index tags through a broadcast loader built from the writer's timeline, so tasks no longer reload the timeline. Global index partition updates look up one file group's merged slice, and the consistent bucket row writer builds at most one view per task.
This was referenced Sep 27, 2026
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #20095 +/- ##
============================================
- Coverage 80.40% 78.58% -1.82%
+ Complexity 34863 34118 -745
============================================
Files 2546 2549 +3
Lines 142888 143012 +124
Branches 17373 17410 +37
============================================
- Hits 114891 112390 -2501
- Misses 20079 22595 +2516
- Partials 7918 8027 +109
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
…rver syncs to the write's timeline With the metadata table, the base files for index lookups came from a new view over a fresh metadata reader, so tagging sent no request to the embedded timeline server. The write handles then made the first requests, with the timeline the table carried at tagging time. When a table service moved the timeline on in between (a compaction going inflight), the server synced to the newer timeline and rejected the handles with a 400. Tagging and the handles could also see different file groups. The lookup now loads the partitions into the table's view in batches and reads the base files from it, as the handles do.
…partitions and bound each load request by path length
Collaborator
3 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe the issue this Pull Request addresses
closes #20091
part of #20064
Stacked on #20069 (
HoodieEngineContext#broadcast), review that first.Index lookups capture the whole
HoodieTable(and the index with its write config) in their task functions. The simple and global simple indexes ship it in one task per base file to read record keys;HoodieIndexUtils#getLatestBaseFilesForAllPartitionsruns one task per partition, each building a view; the simple bucket index reloads the timeline in tasks for every partition; global index partition updates fetch every merged slice of a partition per file group; the consistent bucket row writer builds two views per task.Summary and Changelog
HoodieKeyLocationFetchHandle#locations/#globalLocations).getLatestBaseFilesForAllPartitions: with the metadata table, no tasks: the driver reads the table's view with one refresh-checked read, then loads the remaining partitions in batches bounded by path length, so the embedded timeline server stays synced to the write's timeline; without it, one task per partition as before.HoodieSimpleBucketIndex#tagLocationuses a broadcastBucketLocationLoaderbuilt from the writer's latest completed commit and pending instants; each worker builds one view and caches each partition's bucket mapping. No timeline reload in tasks. Subclasses keep the per-partition path and their hooks.getExistingRecordslooks up only the requested file group's merged slice and parses the internal schema once.Impact
Fewer bytes deserialized per index task and no per-task view, metadata table reader or timeline listing for these lookups; results are unchanged. Bucket index tagging follows the timeline snapshot the write started with instead of a per-task reload. For tables with an HFile base file format, the key-read file opens no longer go through
HoodieWrapperFileSystem, so they lose thehoodie.filesystem.operation.retry.*retries.Risk Level
medium. Touches tagging for the simple, global simple, bloom and simple bucket indexes. Covered by the new tests and the existing simple, global simple, bloom, global bloom, bucket, consistent bucket and global index partition update suites.
Documentation Update
none
Contributor's checklist