Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support parquet file source #2847

Merged
merged 3 commits into from
May 14, 2024
Merged

feat: support parquet file source #2847

merged 3 commits into from
May 14, 2024

Conversation

forsaken628
Copy link
Contributor

@forsaken628 forsaken628 commented May 13, 2024

Signed-off-by: coldWater <254244460@qq.com>
Copy link

codecov bot commented May 13, 2024

Codecov Report

Attention: Patch coverage is 72.97297% with 20 lines in your changes are missing coverage. Please review.

Project coverage is 65.27%. Comparing base (3830778) to head (0ba30ed).
Report is 5 commits behind head on master.

Files Patch % Lines
internal/io/file/parquet_reader.go 69.05% 8 Missing and 5 partials ⚠️
internal/io/file/file_source.go 72.00% 4 Missing and 3 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2847      +/-   ##
==========================================
- Coverage   65.28%   65.27%   -0.02%     
==========================================
  Files         353      354       +1     
  Lines       34667    34718      +51     
==========================================
+ Hits        22631    22659      +28     
- Misses       9804     9820      +16     
- Partials     2232     2239       +7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Collaborator

@ngjaying ngjaying left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the parquet mod big? Can you check how big the binary size will increase? Thanks.

internal/io/file/parquet_reader.go Outdated Show resolved Hide resolved
internal/io/file/parquet_reader.go Outdated Show resolved Hide resolved
@forsaken628
Copy link
Contributor Author

forsaken628 commented May 14, 2024

Is the parquet mod big? Can you check how big the binary size will increase? Thanks.

linux
before
24M kuiper
38M kuiperd
after
27M kuiper
41M kuiperd
There is another option, but I haven't compared it before.
https://pkg.go.dev/github.com/apache/arrow/go/v17/parquet

Signed-off-by: coldWater <254244460@qq.com>
Signed-off-by: coldWater <254244460@qq.com>
@ngjaying
Copy link
Collaborator

Is the parquet mod big? Can you check how big the binary size will increase? Thanks.

linux before 24M kuiper 38M kuiperd after 27M kuiper 41M kuiperd There is another option, but I haven't compared it before. https://pkg.go.dev/github.com/apache/arrow/go/v17/parquet

Great, thanks! If anyone complians about the size, we can make it optional by build tag later.

@ngjaying ngjaying merged commit 1c63ebb into lf-edge:master May 14, 2024
58 checks passed
ngjaying pushed a commit to ngjaying/kuiper that referenced this pull request Jul 24, 2024
Signed-off-by: coldWater <254244460@qq.com>
ngjaying pushed a commit to ngjaying/kuiper that referenced this pull request Jul 24, 2024
Signed-off-by: coldWater <254244460@qq.com>
Signed-off-by: Jiyong Huang <huangjy@emqx.io>
ngjaying pushed a commit to ngjaying/kuiper that referenced this pull request Jul 24, 2024
Signed-off-by: coldWater <254244460@qq.com>
Signed-off-by: Jiyong Huang <huangjy@emqx.io>
ngjaying pushed a commit to ngjaying/kuiper that referenced this pull request Jul 24, 2024
Signed-off-by: coldWater <254244460@qq.com>
Signed-off-by: Jiyong Huang <huangjy@emqx.io>
ngjaying added a commit that referenced this pull request Jul 24, 2024
Signed-off-by: coldWater <254244460@qq.com>
Signed-off-by: Jiyong Huang <huangjy@emqx.io>
Co-authored-by: coldWater <254244460@qq.com>
578223592 pushed a commit to 578223592/ekuiper that referenced this pull request Sep 11, 2024
Signed-off-by: coldWater <254244460@qq.com>
Signed-off-by: Jiyong Huang <huangjy@emqx.io>
Co-authored-by: coldWater <254244460@qq.com>
Signed-off-by: siwuxie <cwl990110@163.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants