-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Proposal: Adaptive Filter Processor #32841
Comments
If accepted we will like to contribute this enhancement. |
This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping |
This issue has been closed as inactive because it has been stale for 120 days with no activity. |
The purpose and use-cases of the new component
We will like to propose a new transform for OTel for adaptively filtering the metric and log collection based on external inputs.
Motivation:
The Adaptive Filter Processor dynamically adjusts the telemetry data based on the available budget, prioritizing metrics or logs that are most relevant or indicative of critical events. It optimizes transmission by adhering to external bandwidth limits, ensuring efficient data delivery without network congestion. This ensures that telemetry data is transmitted efficiently, even in resource-constrained environments.
Example configuration for the component
The adaptive filter processor has the following parameters.
It first takes the allocated
budget
in terms of bandwidth for telemetry data processing and transmission for the cluster. A waiting period, known as thecooloffperiod
, is observed before activating the filtering mechanism, in case the surge in bandwidth usage is temporary. The key parameter is thepriorityorder
which orders the metrics and logs in terms ofpriority
. These priority are matched based on match rules on the labels present in the metrics and logs. Priority level 1 signifies the highest importance. It's important to note that if certain telemetry data falls into priority level 1, it is deemed crucial and transmitted regardless of the specified budget. Metrics and logs not covered by any priority rules in the priority order are assigned thedefaultpriority
. Furthermore, if a metric or log matches multiple priority rules, the higher priority level is chosen.As this filter can be dynamically adjusted, it's necessary to return to the default configuration once the telemetry overload is resolved, specifically when the current bandwidth usage falls below the
minthreshold
. Thecooloffperiod
is utilized again to prevent sporadic drops in bandwidth utilization. However, it may happen that the issue requires more than thecooloffperiod
which is whenbackofftimer
triggers to reduce the checks by increasing the interval either randomly and exponentially.Telemetry data types supported
metric and logs
Is this a vendor-specific component?
Code Owner(s)
No response
Sponsor (optional)
No response
Additional context
No response
The text was updated successfully, but these errors were encountered: