Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

s3mapprovider for Collector: load configuration from config files in Amazon S3 #14317

Merged
merged 6 commits into from
Sep 26, 2022

Conversation

rice-junhaoyu
Copy link
Contributor

@rice-junhaoyu rice-junhaoyu commented Sep 20, 2022

Description:
An implementation of ConfigMapProvider for Amazon S3 (s3mapprovider) allows OTEL Collector the ability to load configuration for itself by fetching and reading config files stored in Amazon S3.

This PR is the same PR as #13113, but requires me to open a new one due to my deactivated email address. It was already reviewed by @mx-psi and @Aneurysm9.

Link to tracking Issue:
#12939

Testing:
First, I mocked how a S3 client works while interacting with a S3 server.
Second, I designed and passed all the unit tests for edge cases such as empty fields, invalid config files, non-exist config files and so on.
Third, I did integration tests such as:

  • Start an OTEL Collector with s3mapprovider.
  • Start an OTEL sample application to send telemetry data to check if the OTEL Collector is running well.

Documentation:
Added a README.md

Sponsor:
@Aneurysm9

.github/ALLOWLIST Outdated Show resolved Hide resolved
Copy link
Member

@mx-psi mx-psi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, I will merge once the CI passes. Thanks @rice-junhaoyu for taking the time to open this new PR and address the required changes :)

@mx-psi mx-psi merged commit 1d88657 into open-telemetry:main Sep 26, 2022
@jufemaiz
Copy link

Is this implementation operational in v0.76.0? I'm getting the following:

collector server run finished with error: unsupported scheme on URI "s3://BUCKET.s3.REGION.amazonaws.com/KEY"

@mx-psi

This comment was marked as outdated.

@jufemaiz
Copy link

Apologies >< v0.72.0.

@Aneurysm9
Copy link
Member

This is not available in the collector-contrib distribution as the collector builder does not yet have support for specifying which config providers to include. See open-telemetry/opentelemetry-collector#4759.

@jufemaiz
Copy link

@Aneurysm9 oh thank you for that clarification! It's not clear from the docs that that was the case!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants