-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Mark the fingerprints of all inhibiting rules #3768
base: main
Are you sure you want to change the base?
Mark the fingerprints of all inhibiting rules #3768
Conversation
Before this change, Alertmanager would mark just the fingerprint of the first inhibition rule to inhibit an alert, rather than the fingerprints of all inhibition rules that inhibit the alert. This is different from silences which mark the IDs of all silences that silence an alert, not just the first silence. This commit changes how inhibition rules work to match the behavior of silences, such that the fingerprints of all inhibition rules that inhibit the alert are marked. This change is expected to increase CPU usage as the inhibiter has to do more work, matching the label set against all inhibition rules instead of stopping at the first match. However, this is no different from how silences mute alerts. Signed-off-by: George Robinson <[email protected]>
I just realized this code is broken when the |
I really don't remember the detailed motivations here, just that it was complicated. Maybe the problems you have encountered above have to do with the reason. |
Signed-off-by: George Robinson <[email protected]>
Signed-off-by: George Robinson <[email protected]>
9381928
to
55080f2
Compare
OK so I created a number of benchmarks to better understand how performance would be affected with this change. The four benchmarks do the following:
Here are the benchmarks from
And here are the benchmarks from this branch:
I also benchmarked
The benchmarks shows the following:
|
Before this change, Alertmanager would mark just the fingerprint of the first inhibition rule to inhibit an alert, rather than the fingerprints of all inhibition rules that inhibit the alert. This is different from silences which mark the IDs of all silences that silence an alert, not just the first silence.
This commit changes how inhibition rules work to match the behavior of silences, such that the fingerprints of all inhibition rules that inhibit the alert are marked.
This change is expected to increase CPU usage as the inhibiter has to do more work, matching the label set against all inhibition rules instead of stopping at the first match. However, this is no different from how silences mute alerts.