feat(outputs.parquet): Introduce Parquet output #15602
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
telegraf-tiger
bot
added
feat
powersj
force-pushed
the
feat/parquet-serializer
branch
2 times, most recently
from
a2a428d
to
a0c7b38
Compare
powersj
added
the
ready for final review
DStrand1
reviewed
Jul 18, 2024
DStrand1
approved these changes
Jul 19, 2024
* extra word typo in readme * initialize timestamp field in init(), update tests * always use string arrow type for tags
powersj
force-pushed
the
feat/parquet-serializer
branch
from
62bc2e1
to
98399b6
Compare
|
Download PR build artifacts for linux_amd64.tar.gz, darwin_arm64.tar.gz, and windows_amd64.zip. 📦 Click here to get additional PR build artifactsArtifact URLs |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Introduces a new output to write metrics in Parquet format. This groups metrics by metric name and writes them to files. We need to know the schema to write beforehand, the first time we encounter a metric name we generate the schema based on the metric we found then use the buffered file writer from here on out to write files as efficiently as possible.
We must close the file correctly otherwise the file will not be a valid parquet file. As such I have avoided creating files by template name due to complications with keeping too many files open. Instead, I have a time-based rotation option that will close the existing file and create a new file.
Check out this blog post for an overview of Parquet files: https://www.influxdata.com/blog/how-good-parquet-wide-tables/
Checklist
Related issues
fixes: #14786