Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

awss3exporter: Add Protocol Buf storage format #30681

Closed
adcharre opened this issue Jan 19, 2024 · 3 comments
Closed

awss3exporter: Add Protocol Buf storage format #30681

adcharre opened this issue Jan 19, 2024 · 3 comments
Labels
enhancement New feature or request exporter/awss3

Comments

@adcharre
Copy link
Contributor

Component(s)

exporter/awss3

Is your feature request related to a problem? Please describe.

Currently the only supported format for writing trace data to an S3 bucket is JSON which is not very concise. As a result this use more space than necessary in the S3 bucket and more CPU and network resources.

Describe the solution you'd like

Add support for writing all OpenTelemetry data in ProtoBuf format.
ProtoBuf output is approximately half the size of JSON formatted data.

Describe alternatives you've considered

No response

Additional context

No response
@adcharre adcharre added enhancement New feature or request needs triage New item requiring triage labels Jan 19, 2024
@github-actions GitHub Actions
Copy link
Contributor

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@adcharre adcharre mentioned this issue Jan 19, 2024
Merged
@github-actions github-actions bot mentioned this issue Jan 23, 2024
Closed
@crobert-1
Copy link
Member

It looks like a code owner has recently filed an issue to separate encodings from this exporter (here). I think it makes sense to add support for sending data encoded in the protobuf format, but the proper implementation would be to rely on the encoding extension rather than the exporter itself implementing this logic.

@crobert-1 crobert-1 removed the needs triage New item requiring triage label Jan 23, 2024
TylerHelmuth pushed a commit that referenced this issue Jan 29, 2024
@adcharre @atoulme
awss3exporter: Add Protocol Buf storage format (#30682)
a76c319 
**Description:** Add ProtoBuf output format to the AWS S3 exporter

**Link to tracking Issue:** #30681

**Testing:** Additional unit tests have been added for the new format
type and I've also exported data to an S3 bucket in Protobuf format and
read it back to confirm that it's working end to end.

**Documentation:** Added details of the new marshaler format to the
README.md

---------

Co-authored-by: Antoine Toulme <antoine@toulme.name>
cparkins pushed a commit to AmadeusITGroup/opentelemetry-collector-contrib that referenced this issue Feb 1, 2024
@adcharre @atoulme @cparkins
awss3exporter: Add Protocol Buf storage format (open-telemetry#30682)
4c97b1a 
**Description:** Add ProtoBuf output format to the AWS S3 exporter

**Link to tracking Issue:** open-telemetry#30681

**Testing:** Additional unit tests have been added for the new format
type and I've also exported data to an S3 bucket in Protobuf format and
read it back to confirm that it's working end to end.

**Documentation:** Added details of the new marshaler format to the
README.md

---------

Co-authored-by: Antoine Toulme <antoine@toulme.name>
@atoulme
Copy link
Contributor

atoulme commented Mar 18, 2024

Please take a look at #31801, any review helps, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request exporter/awss3
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@atoulme @adcharre @crobert-1