You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the solution you'd like
We'd like to add a rate limiter (records / s and/or bytes / s) that a system can send to the collector. We're standing up an ingest pipeline using the auth work that you've recently implemented. We're hoping to tie usage back to the apikey that a system is using.
Describe alternatives you've considered
We're considering offering a service upstream that takes the apikey and determines if it's exceeded the limit based on utilizing information from redis (sum(bytes of each type) or count(records of each type)). This wouldn't be ideal because we don't want to process the data multiple times and ideally bake it into an extension similar to auth so that we return proper HTTP response/headers and to prevent data from potentially being processed.
Describe the solution you'd like
We'd like to add a rate limiter (records / s and/or bytes / s) that a system can send to the collector. We're standing up an ingest pipeline using the auth work that you've recently implemented. We're hoping to tie usage back to the apikey that a system is using.
Describe alternatives you've considered
We're considering offering a service upstream that takes the apikey and determines if it's exceeded the limit based on utilizing information from redis (sum(bytes of each type) or count(records of each type)). This wouldn't be ideal because we don't want to process the data multiple times and ideally bake it into an extension similar to auth so that we return proper HTTP response/headers and to prevent data from potentially being processed.
Additional context
Not sure if it makes sense to consider (at least at the contrib level) to incorporate something similar to what Github did with their rate limiting. Understand it's a different problem set, but can certainly re-use some concepts.
https://github.blog/2021-04-05-how-we-scaled-github-api-sharded-replicated-rate-limiter-redis/
The text was updated successfully, but these errors were encountered: