Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add rate limiter for receivers #3509

Closed
bputt-e opened this issue Jun 28, 2021 · 0 comments
Closed

Add rate limiter for receivers #3509

bputt-e opened this issue Jun 28, 2021 · 0 comments

Comments

@bputt-e
Copy link

bputt-e commented Jun 28, 2021

Describe the solution you'd like
We'd like to add a rate limiter (records / s and/or bytes / s) that a system can send to the collector. We're standing up an ingest pipeline using the auth work that you've recently implemented. We're hoping to tie usage back to the apikey that a system is using.

Describe alternatives you've considered
We're considering offering a service upstream that takes the apikey and determines if it's exceeded the limit based on utilizing information from redis (sum(bytes of each type) or count(records of each type)). This wouldn't be ideal because we don't want to process the data multiple times and ideally bake it into an extension similar to auth so that we return proper HTTP response/headers and to prevent data from potentially being processed.

Additional context
Not sure if it makes sense to consider (at least at the contrib level) to incorporate something similar to what Github did with their rate limiting. Understand it's a different problem set, but can certainly re-use some concepts.
https://github.blog/2021-04-05-how-we-scaled-github-api-sharded-replicated-rate-limiter-redis/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants