Circuit breaker for receiver/prometheus and exporter/prometheusremotewrite

[premendrasingh]

Component(s)

cmd/otelcontribcol, exporter/prometheusremotewrite, receiver/prometheus

Is your feature request related to a problem? Please describe.

We use receiver/prometheus to scrape prometheus metrics and exporter/prometheusremotewrite to write those metrics using otelcol to Prometheus. Sometimes the applications don't expose metrics to harvest and sometimes when prometheus server or network issues cause otel-collector queues to get filled and get OOM killed.

Describe the solution you'd like

I would like to propose a circuit breaker implemented at receiver and exporter level so that when an endpoint is unresponsive, the circuit breaker pattern can help prevent cascading failures, while the services are trying to recover. It will handle errors gracefully, reduce application downtimes and help the services to recover more efficiently. State changes of the circuit breaker can be used for error monitoring.

Describe alternatives you've considered

A retry option with back-off can work, but circuit breaker will be much more efficient.

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/prometheusremotewrite: @Aneurysm9 @rapphil
• receiver/prometheus: @Aneurysm9 @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dashpole]

I would consider using the memory limiter processor, and reducing the size of the queue in the queued retry section of the prometheus remote write exporter. You may also consider using the (disk-based) WAL in the prometheus remote write exporter to buffer to disk instead of memory.

[premendrasingh]

Thanks for the suggestions, I will try them.

[crobert-1]

@premendrasingh Have you been able to try the memory limiter processor yet? Does this fit your use case?

[github-actions]

This issue has been closed as inactive because it has been stale for 120 days with no activity.