Skip to content
This repository has been archived by the owner on Aug 2, 2022. It is now read-only.

Add result index retention for anomaly detection #143

Open
ylwu-amzn opened this issue May 14, 2020 · 0 comments
Open

Add result index retention for anomaly detection #143

ylwu-amzn opened this issue May 14, 2020 · 0 comments
Labels
AnomalyDetection Item related to Anomaly Detection and AD Kibana plugin feature new feature

Comments

@ylwu-amzn
Copy link
Contributor

ylwu-amzn commented May 14, 2020

Currently AD just rollover result index if the doc count exceeds 300k or the index age exceeds 30 days. We have no easy way to let user define how long they want to keep the results. Too many AD result indices may bring down the cluster performance.

Possible solutions:

Option1

Leverage Index Management plugin. We can ask user define some retention policy when create detector and set up actions in Index Management plugin

Option2

We build our own retention job to maintain AD result indices and provide user some dynamic settings.

Option1 will avoid reinvention work but we add another dependency on Index Management plugin.
For Option2 , we can add retention logic in AD ES plugin and no more dependency included.

@ylwu-amzn ylwu-amzn added the feature new feature label May 14, 2020
@sean-zheng-amazon sean-zheng-amazon added the AnomalyDetection Item related to Anomaly Detection and AD Kibana plugin label Jun 10, 2020
@anirudha anirudha changed the title Add AD result index retention Add result index retention for anomaly detection Jul 8, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
AnomalyDetection Item related to Anomaly Detection and AD Kibana plugin feature new feature
Projects
None yet
Development

No branches or pull requests

2 participants