Add PreparedInsert flow #443
Open
+317
−9
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
theory
commented
Oct 30, 2025
Comment on lines
+1191
to
+1178
| if (chtype->GetCode() == Type::LowCardinality) { | ||
| chtype = col->As<ColumnLowCardinality>()->GetNestedType(); | ||
| } |
There was a problem hiding this comment.
I'm honestly not sure this is the right thing to do. Might one need Type::LowCardonality?
theory
commented
Oct 30, 2025
|
|
||
| void FinishInsert(); | ||
|
|
||
| void SendData(const Block& block); |
There was a problem hiding this comment.
I had to move this to public so that PreparedInsert can call it. Not in the header file, though, so shouldn't matter.
theory
commented
Oct 30, 2025
theory
commented
Oct 30, 2025
| public: | ||
| Block * GetBlock(); | ||
| void Execute(); | ||
| // XXX This shouldn't be public. |
There was a problem hiding this comment.
I couldn't figure out how to make this private. Suggestions appreciated.
There was a problem hiding this comment.
Would be nice if it worked declared public in the .cpp file, but I think I could also use an Impl class like Client does to hide such things.
theory
force-pushed
the
insert-block
branch
4 times, most recently
from
ade33f4 to
51d8216
Compare
Add a new pattern for "prepared inserts". It works like this:
* Call `PrepareInsert` with an `INSERT` query with optional columns
and ending in `VALUES`. No values should be included in the string.
* It returns a `PreparedInsert` object that has two methods:
* `Block()` returns a `Block` pre-configured with columns as
declared in the `INSERT` statement
* `Execute()` inserts data from the block then clears it.
* Call `Finish()` or just let the `PreparedInsert` object go out of
scope to send any remaining rows and to signal the server that it's
done.
This allows one to send smaller batches of blocks, thereby using less
memory, but still in a single ClickHouse `INSERT` operation.
Expected to be useful in the Postgres foreign data wrapper insert API,
where multiple rows can be inserted at once but its API handles
one-at-a-time insertion. It will also support the FDW COPY API, which
can submit huge batches of data to insert, as well.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add a new pattern for "prepared inserts". It works like this:
PrepareInsertwith anINSERTquery with optional columns and ending inVALUES. No values should be included in the string.PreparedInsertobject that has two methods:Block()returns aBlockpre-configured with columns as declared in theINSERTstatementExecute()inserts data from the block then clears it.PreparedInsertobject goes out of scope it first signals the server that it's done sending data.This allows one to send smaller batches of blocks, thereby using less memory, but still in a single ClickHouse
INSERToperation.Expected to be useful in the Postgres foreign data wrapper insert API, where multiple rows can be inserted at once but its API handles one-at-a-time insertion. It will also support the FDW COPY API, which can submit huge batches of data to insert, as well.