-
Notifications
You must be signed in to change notification settings - Fork 2.4k
Silences: Optimize incremental mutes queries via a silence version index #4723
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
siavashs
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, left a minor comment.
|
This needs a rebase. |
| newVersionIndex.add(s.version+1, e.Silence.Id) | ||
| } | ||
| } | ||
| s.mtx.Lock() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is unrelated to your change. However, I'm wondering if this lock is necessary because it appears loadSnapshot is only called during construction of new Silences type.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Indeed, it's probably unnecessary. But also harmless as at construction there's no contention.
Signed-off-by: Guido Trotter <guido@hudson-trading.com>
Signed-off-by: Guido Trotter <guido@hudson-trading.com>
…new silences that have been added since the last version we saw Signed-off-by: Guido Trotter <guido@hudson-trading.com>
Signed-off-by: Guido Trotter <guido@hudson-trading.com>
rajagopalanand
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Had one question about lock acquisition in the loadSnapshot method
Signed-off-by: Guido Trotter <guido@hudson-trading.com>
c4cecac to
00dedba
Compare
Currently Mutes is optimized for when the global silences haven't changed at all, and either our label set had zero or a few silences. When even a single silence has been added to the state though, the entire state needs to be scanned to find any match.
Since we know that existing silences cannot start matching without being re-added to the state, and we have a state version at which we are "caught up" available, we can optimize the code path when a few silences have been added to the state by:
This trades some memory, with a greatly increased performance for the situation when we have a lot of silences, and only a few have been added to the state since we last queried.
The GC is able to keep the state and indices up to date, in place, and reallocate the index to shed memory load when the set of silences has stably halved.
There is an increase in memory usage in MutesIncremental, which we could potentially mitigate in the future allowing QSince and QIDS queries in the same call, or allowing query types which return IDs instead of silence lists. The cost is still worth it as it's only in the case of massive silence loads that would make alertmanager nowadays fail, and is minimal compared to the current size of servers.
GC is not substantially slower, and only spends more memory when it needs to free up memory after a downsize of more than 50%, doing one more allocation then.