[fix](hudi-mtmv) Fix the issue that Hudi materialized views are not involved in transparent rewriting#49513
Merged
morningman merged 1 commit intoapache:branch-hudi-mtmvfrom Mar 27, 2025
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
morningman
pushed a commit
that referenced
this pull request
Apr 2, 2025
…nvolved in transparent rewriting (#49513) It's because when Hudi performs transparent rewriting, the timestamp of the base table obtained by `loadSnapshot` is inconsistent with the timestamp stored after partition refresh, which causes the comparison of `tableSnapshot` to fail, resulting in the materialized view not being hit. Currently, the logic of `loadSnapshot` to obtain the timestamp of the base table is a bit strange and doesn't quite meet expectations. Further in - depth research is needed on how to modify it. For now, in the `getTableSnapshot` function, simply return `0L` constantly, indicating that the `tableSnapshot` is always synchronized, to bypass this problem. This modification is consistent with the original expectation of manual refresh. We'll deal with this issue later.
BePPPower
added a commit
to BePPPower/doris
that referenced
this pull request
Apr 10, 2025
…nvolved in transparent rewriting (apache#49513) It's because when Hudi performs transparent rewriting, the timestamp of the base table obtained by `loadSnapshot` is inconsistent with the timestamp stored after partition refresh, which causes the comparison of `tableSnapshot` to fail, resulting in the materialized view not being hit. Currently, the logic of `loadSnapshot` to obtain the timestamp of the base table is a bit strange and doesn't quite meet expectations. Further in - depth research is needed on how to modify it. For now, in the `getTableSnapshot` function, simply return `0L` constantly, indicating that the `tableSnapshot` is always synchronized, to bypass this problem. This modification is consistent with the original expectation of manual refresh. We'll deal with this issue later.
BePPPower
added a commit
to BePPPower/doris
that referenced
this pull request
Apr 21, 2025
…nvolved in transparent rewriting (apache#49513) It's because when Hudi performs transparent rewriting, the timestamp of the base table obtained by `loadSnapshot` is inconsistent with the timestamp stored after partition refresh, which causes the comparison of `tableSnapshot` to fail, resulting in the materialized view not being hit. Currently, the logic of `loadSnapshot` to obtain the timestamp of the base table is a bit strange and doesn't quite meet expectations. Further in - depth research is needed on how to modify it. For now, in the `getTableSnapshot` function, simply return `0L` constantly, indicating that the `tableSnapshot` is always synchronized, to bypass this problem. This modification is consistent with the original expectation of manual refresh. We'll deal with this issue later.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
It's because when Hudi performs transparent rewriting, the timestamp of the base table obtained by
loadSnapshotis inconsistent with the timestamp stored after partition refresh, which causes the comparison oftableSnapshotto fail, resulting in the materialized view not being hit.Currently, the logic of
loadSnapshotto obtain the timestamp of the base table is a bit strange and doesn't quite meet expectations. Further in - depth research is needed on how to modify it.For now, in the
getTableSnapshotfunction, simply return0Lconstantly, indicating that thetableSnapshotis always synchronized, to bypass this problem. This modification is consistent with the original expectation of manual refresh. We'll deal with this issue later.Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)