Skip to content

enhancement(clickhouse sink): Add ArrowStream format #24075

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 40 commits into
base: master
Choose a base branch
from
+2,668 −5
Open

enhancement(clickhouse sink): Add ArrowStream format #24075

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
8c3e47f
enhancement(clickhouse sink): add support for ArrowStream format
benjamin-awd Oct 18, 2025
5852b8b
refactor: do not allow dynamic database/table for arrow
benjamin-awd Oct 18, 2025
6583753
chore: add type assertions
benjamin-awd Oct 18, 2025
3c7b4f2
refactor: use clearer function names in request builder
benjamin-awd Oct 20, 2025
ff79ca3
chore: add snafu for request builder
benjamin-awd Oct 20, 2025
631da91
chore: move imports to top package
benjamin-awd Oct 20, 2025
1d951b4
chore: handle templates for table and database
benjamin-awd Oct 20, 2025
7ae4c2a
build(deps): move arrow under external libs
benjamin-awd Oct 20, 2025
d852d5a
build(deps): bump arrow/arrow-schema to latest
benjamin-awd Oct 20, 2025
5b8f548
chore: add support for all timeunits e.g. nanoseconds
benjamin-awd Oct 20, 2025
a1e7318
chore: add handling for LowCardinality
benjamin-awd Oct 20, 2025
8e78211
chore: add support for Clickhouse decimal
benjamin-awd Oct 20, 2025
c7f05f6
chore: add capacity and string zero copy optimization
benjamin-awd Oct 20, 2025
d7e4e33
chore: use logging macros from extern crate
benjamin-awd Oct 21, 2025
609c126
chore: add debug logs for schema
benjamin-awd Oct 21, 2025
b11d2b7
chore: use simpler message for NoSchemaProvided snafu
benjamin-awd Oct 21, 2025
8c755c9
chore: DRY target scale within decimal builders
benjamin-awd Oct 21, 2025
1babc70
docs: add note about upcasting i128 -> i256
benjamin-awd Oct 21, 2025
8003c7b
chore: move i256 import statement to top
benjamin-awd Oct 21, 2025
cbd78c3
chore: remove redundant schema log
benjamin-awd Oct 21, 2025
cfe3848
chore: only log schema a single time at startup
benjamin-awd Oct 21, 2025
939f5c2
chore: use arrow payload size instead of redundant json size
benjamin-awd Oct 21, 2025
730309a
chore: add support for uint
benjamin-awd Oct 22, 2025
f169a15
chore: add support for rfc3339 timestamp
benjamin-awd Oct 22, 2025
5148128
refactor: move arrow to clickhouse sink subfolder
benjamin-awd Oct 25, 2025
69063e5
refactor: move Arrow encoder to shared util
benjamin-awd Oct 25, 2025
060a8fe
chore: remove redundant feature gates
benjamin-awd Oct 25, 2025
3396bd7
refactor: rename schema.rs -> arrow_schema.rs
benjamin-awd Oct 25, 2025
a56da91
chore: add changelog fragment
benjamin-awd Oct 25, 2025
0734f4f
chore: add helper function to handle clickhouse type parsing
benjamin-awd Oct 25, 2025
c27b09b
refactor: remove default coercions in arrow_schema
benjamin-awd Oct 25, 2025
3ae4ff7
refactor: add constants for decimal precision values
benjamin-awd Oct 25, 2025
66ca5f5
refactor: extract type modifier unwrapping into helper
benjamin-awd Oct 25, 2025
edc4c5f
refactor: remove unnecessary vector allocation
benjamin-awd Oct 25, 2025
d8ccf91
refactor: organize type mapping with comments
benjamin-awd Oct 25, 2025
bb93d26
refactor: use constants and functional style in decimal parsing
benjamin-awd Oct 25, 2025
9f93b97
chore: update LICENSE-3rdparty
benjamin-awd Oct 25, 2025
689e33c
chore: remove duplicated tests
benjamin-awd Oct 25, 2025
57fbfe4
chore: update generated docs
benjamin-awd Oct 25, 2025
56c4b29
chore: remove redundant docstring
benjamin-awd Oct 27, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
305 changes: 305 additions & 0 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading