[core] Data files with delete records should not be upgraded directly to max level #2962
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
JingsongLi
reviewed
Mar 7, 2024
| // Why don't we keep addRowCount and deleteRowCount? | ||
| // Because in previous versions of DataFileMeta, we only keep rowCount. | ||
| // We have to keep the compatibility. | ||
| private final long totalRowCount; |
There was a problem hiding this comment.
Can we still use rowCount naming?
Because in schema, it is still _ROW_COUNT...
JingsongLi
reviewed
Mar 11, 2024
| // Because in previous versions of DataFileMeta, we only keep rowCount. | ||
| // We have to keep the compatibility. | ||
| private final long totalRowCount; | ||
| private final @Nullable Long addRowCount; |
There was a problem hiding this comment.
It is better to introduce deleteRowCount?
tsreaper
force-pushed
the
upgrade
branch
3 times, most recently
from
06ccb22 to
53f7067
Compare
JingsongLi
reviewed
Mar 15, 2024
| private final long rowCount; | ||
| private final @Nullable Long deleteRowCount; |
There was a problem hiding this comment.
Put this field to last class member.
I've tested by hand, that the old reader code can read this newly written table. |
zhu3pang
pushed a commit
to zhu3pang/incubator-paimon
that referenced
this pull request
Mar 29, 2024
When i use 0.5 code to read a table written by 0.8-SNAPSHOT, i got this exception. Is that as expected? @tsreaper |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose
Data files on max level should not have delete records. However when upgrading files to max level we currently does not perform this check.
Tests
TableWriteTest#testUpgradeToMaxLevel.API and Format
Yes.
DataFileMetais changed, and thusManifestEntryis changed. However as long as we can passPrimaryKeyFileStoreTableTest#testStreamingChangelogCompatibility02there should not be any issue.Documentation
No.