[BUG] Outdated findings if findings history max age is reached / SAP GUI performance is sub par

[michaelSchmidMaloon]

What is the bug?
I'm not sure if his is a bug or a feature.

We're trying to use SAP for our SIEM requirements. For the longest time we used the default settings of the plugin. However this created findings indices of enormous size when lots of findings are produced. Which is often the case, as we run an evolving platform that produces a lot of false-positives at times.

E.g. the latest index of one source is .opensearch-sap-<source>-findings-2024.12.20-000024 with 469650716 documents at the moment. Using the GUI of the plugin is extremely slow and with the lack of filter options we get blinded by tens of thousand false-positive findings of the same type.

To mitigate this I tried to have one findings index per day, in hope to just delete it if such a flood of false-positives crashed down on us for that day. Or at least to remove some findings more easily.

Similarly I set other max_age-settings more tightly

plugins:
  security_analytics:
  ## https://opensearch.org/docs/latest/security-analytics/settings/
    alert_history_max_age: "1d"
    alert_history_retention_period: "90d"
    finding_history_max_age: "1d"
    finding_history_retention_period: "90d"
    correlation_history_max_age: "1d"
    correlation_history_retention_period: "90d"

But it seems that SAP is looking through old source data again and creates findings from before the findings index was created.

E.g. today's index .opensearch-sap-<source>-findings-2025.01.18-<suffix> also contains findings with timestamps of 2025-01-02 but this findings is shown in the GUI table of the findings tab with today date.

As we have daily source indices, we us a wildcard for the corresponding detector. Here is an example with falco

Falco -> daily index e.g. falco-2025-01-18 -> detector for falco uses source falco-*

Maybe that's the culprit here as the detector always goes through all matching source indices?!

Maybe my understanding of how this plugin works vs how I think it should work is the problem. Or this is a bug? In any case

• is there a way to increase the performance of the SAP GUI?
• is there a way to easily remove false-positive findings? I currently do this with the slow delete_by_query.
• are there plans to add better filter options to the SAP GUI?
• if the detectors really always go through all matching source indices, is there a way to set this up accounting for daily source indices?

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Run OpenSearch with SAP configured like in the snipped above.
2. Create source with daily indices and detector that creates a lot of findings
3. Wait a few days until a few findings indices are created
4. Check timestamp of finding details document

What is the expected behavior?
I'd expect for the SAP to create findings from the time on that the detector is enabled. Or at least from the time an that a new findings index is created.

What is your host/environment?

• OS: Kubernetes with Helm chart opensearch-2.26.0
• Version v 2.17.1
• Plugins: SAP, metrics exporter

Do you have any screenshots?
If applicable, add screenshots to help explain your problem.

Do you have any additional context?