Scaler not being properly closed

[gjacquet]

Report

We found some instances of scalers not being closed after being refreshed leading to possible resources leak.
Some scalers, like Kafka also have background goroutines (in this case to refresh topics metadata) that are never stopped.

Expected Behavior

Any scaler should be closed when it is removed from the scaler cache or no longer in use.

Actual Behavior

Scalers are not always closed properly.

Steps to Reproduce the Problem

1. Create a scaled object with multiple scalers.
2. Change the scaled object multiple time to trigger a change in the generation number
3. There is then a chance previous scalers from the cache were not properly closed

Logs from KEDA operator

None

KEDA Version

2.14.0

Kubernetes Version

1.29

Platform

Amazon Web Services

Scaler Details

Kafka

Anything else?

I believe there are multiple issues causing this behaviour.

If scaler is outdated (different generation) it should be refreshed will would call Close(): https://github.com/kedacore/keda/blob/v2.14.0/pkg/scaling/scale_handler.go#L307 / https://github.com/kedacore/keda/blob/v2.14.0/pkg/scaling/scale_handler.go#L386-L389

There is also a possible race condition when multiple goroutines enter performGetScalerCache at the same time after ClearScalerCache. The code that checks if the entry needs to be closed is not thread safe: https://github.com/kedacore/keda/blob/v2.14.0/pkg/scaling/scale_handler.go#L386-L389. Multiple goroutines can reach this point creating each one new scaler since the cache was empty when entering the function and not closing the scaler created by the other goroutines.

[zroubalik]

@gjacquet thanks a lot for reporting and provided details!