Backend not applying filters during shutdown

[emmanueljamet]

Expected Behavior

When receiving a SIGTERM, sensu backend keeps applying filters on events processed until the whole backend is properly stopped.

Current Behavior

While shutting down, sensu backend executes handlers for events that should be filtered by custom filters, but are not.

Possible Solution

Maybe the components are not stopped in the right order (?)

Steps to Reproduce (for bugs)

Sensu backend is running on a Kubernetes cluster.
Deleting the pod running sensu backend container is enough to see some events being sent to handler.

Context

We are running sensu on an EKS cluster based on spot instances.
When AWS reclaims an instance, the pod is deleted in order to be rescheduled elsewhere.
We are running about ~400 agents with an avg ~15 checks on each and each time a node is reclaimed we get a load of 10 to 30 unwanted "OK" events on our alerts dedicated slack channel.
This adds some noise on our monitoring system.

The signal received

Filters errors

Components shutdown order

Your Environment

• Sensu version used: sensu-backend v6.1.2 OSS
• Installation method: docker
• Operating System and version: kubernetes 1.17 (EKS)

[echlebek]

Thanks for the bug report. We haven't verified this behaviour but it seems fairly conclusive based on your report. I'll tag it as a bug and we'll prioritize it accordingly.

Are you expecting that your javascript event filters should fail like they are in your screenshot? It seems like you have some undefined symbols.

[emmanueljamet]

Hi @echlebek.
The 'fatigue_check' and 'period_check' are in fact the names of the filters not applied.
This is not expected to fail :)

[emmanueljamet]

Hello, do we have an update on this issue please ?

[ddes]

Hi there,
Do we have any news on this ?

[echlebek]

This is being worked on and will land in 6.5.2 or 6.6.0.

[echlebek]

The sensu-go filter/mutator/handler pipeline was substantially reworked in 6.5.0, so while I'm still looking into this issue, the problem may already be resolved. Please update to 6.5.1, as it might fix your problem. Thanks!