POC: Merge extension #1022
Draft
POC: Merge extension #1022
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 9 commits
June 5, 2025 17:22
JanKaul
commented
Jun 5, 2025
| // table accordingly. | ||
| pub struct MergeIntoSink { | ||
| pub input: Arc<LogicalPlan>, | ||
| pub target: DataFusionTable, |
There was a problem hiding this comment.
@jonathanc-n The target could also be an Arc<dyn TableProvoder>
There was a problem hiding this comment.
You can also pass the target as LogicalPlan
jonathanc-n
reviewed
Jun 11, 2025
| } | ||
|
|
||
| fn fmt_for_explain(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result { | ||
| write!(f, "MergeIntoSink") |
There was a problem hiding this comment.
Maybe we can specify the catalog, schema, and table here
|
|
||
| fn dyn_hash(&self, state: &mut dyn Hasher) { | ||
| "MergeIntoSink".dyn_hash(state); | ||
| self.input.dyn_hash(state); |
There was a problem hiding this comment.
Should hash target as well, if two sinks point at different tables can lead to errors
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR provides a rough structure of how the writing phase of the MERGE INTO statement looks like. There are still some parts missing. But I think the general structure with a LogicalPlan "MergeIntoSink" and a PhysicalPlan "MergeIntoSinkExec" should work.
The idea is to adopt the
merge_queryfunction to prepare a logical plan that emits a RecordBatchStream of the desired form. As a last step of themerge_queryfunction this logical plan is then passed as an input to the "MergeIntoSink" LogicalPlan. The Extension plan will then during execution convert the "MergeIntoSink" into the "MergeIntoSinkExec" which will perform the desired operations.