[HUDI-4699] RFC for auto record key generation #10365
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nsivabalan
changed the title
codope
approved these changes
Dec 19, 2023
| ## Abstract | ||
|
|
||
| One of the prerequisites to create an Apache Hudi table is to configure record keys(a.k.a primary keys). Since Hudi’s | ||
| origin at Uber revolved around supporting mutable workloads at large scale, these were deemed mandatory. As we started |
There was a problem hiding this comment.
Instead of mentioning Uber, maybe we can talk about while primary keys bring a lot of benefits, it also means that user need to model the data and thus some amount cognitive overhead. We can also point to Oracle where row id gets generated automatically irrespective of whether we define primary key rght upfront.
codope
reviewed
Dec 19, 2023
Comment on lines
+149
to
+156
| - What impact (if any) will there be on existing users? | ||
| - If we are changing behavior how will we phase out the older behavior? | ||
| - If we need special migration tools, describe them here. | ||
| - When will we remove the existing behavior | ||
|
|
||
| ## Test Plan | ||
|
|
||
| Describe in few sentences how the RFC will be tested. How will we know that the implementation works as expected? How will we know nothing broke?. |
There was a problem hiding this comment.
Let's briefly mention about these. If users upgrade/downgrade will it all be handled OOB?
| Combining them in a single string key as below | ||
| "${commit_timestamp}_${batch_row_id}" | ||
|
|
||
| For row-id generation we plan to use a combination of “spark partition id” and a row Id (sequential Id generation) to generate unique identity value for every row w/in batch (this particular component is available in Spark out-of-the-box, but could be easily implemented for any parallel execution framework like Flink, etc) |
There was a problem hiding this comment.
Do we need to mention the equivalent in Flink or other engines?
yihua
pushed a commit
that referenced
this pull request
Feb 27, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Change Logs
Adding RFC for auto record key generation.
Impact
Will unblock more use-cases where users do not need to set any record keys.
Risk level (write none, low medium or high below)
low.
Documentation Update
Describe any necessary documentation update if there is any new feature, config, or user-facing change
ticket number here and follow the instruction to make
changes to the website.
Contributor's checklist