Is There A Way To Better DeDupe Error Events To Avoid Providers (Slack in our case) From Getting Flooded During Outages #1386

Hi Team,

We have configured a Provider resource to send error events to our slack channel. On Sept 3rd there was an outage with AWS STS service in Oregon region.

Our Source watchers i.e HelmCharts & Buckets started failing reconciliation as we use S3 & ECR for them & STS service was unavailable.

Below are some error messages that we received:

AWS authentication failed: failed to create provider access token for the controller: failed to refresh cached credentials, failed to retrieve credentials, operation error STS: AssumeRoleWithWebIdentity, exceeded maximum number of attempts, 3, https response error StatusCode: 503, RequestID: 804c5f8f-e91a-4129-8962-xxxxxxxxx, api error ServiceUnavailable: Unknown

failed to get credential from 'aws': failed to get access token for artifact registry: failed to create provider access token for the controller: failed to refresh cached credentials, failed to retrieve credentials, operation error STS: AssumeRoleWithWebIdentity, exceeded maximum number of attempts, 3, https response error StatusCode: 503, RequestID: 2ceab677-380e-4f9e-8723-xxxxxxxxx api error ServiceUnavailable: Unknown

As the request ID was changing with every retry Flux was doing, error events became unique & the default 5m rate limit interval didn't apply to them & our slack channels were flooded with the error events before we finally suspended the Alert resources.

Another scenario where we get quick error events in succession is when a Kustomization watcher doesn't find the file/directory path & the tmp path makes the error message unique on every reconcile & again the rate-limit doesn't work:

kustomization path not found: stat /tmp/kustomization-373625xxxx/<watcher-name>/<path>: no such file or directory

Is there a way we can better dedupe/rate-limit Flux events so that we don't get a storm of events during outages affecting entire cluster.

If not supported as of today natively in Flux. Can this be feature be implemented.

We had explored exclusionList but we would like to keep these events, as they provide in-depth details about the error especially during the outage scenario.

Flux Version: v2.8.8