Threshold breached Alert needs to be generated usage above 90%

[GowthamShanmugam]

Currently, we are raising warning alert when volume or brick or host utilization crosses 75%.
let's take an example when brick utilization reaches 80% then it will raise warning alert. But tendrl won't raise an alert when utilization is increased from 80% to 90% or above. Because tendrl already raised warning alert.

When utilization reaches 90% or above tendrl need to raise an alert with critical severity.

[GowthamShanmugam]

Problem with grafana is even if we add multiple alert conditions it won't raise alert for each alert condition. Just it will update the alert with current utilization value.

if alert dashboard panel has two alert conditions for 75% and 90% but an alert will raise only once for only one alert condition.
if alert raised for 75% then it won't again raise an alert when utilization reaches 90%.

[GowthamShanmugam]

one suggestion from @cloudbehl is for each brick or volume or node create two alert panels in grafana dashbaord. So one for 75% and another one for 90%. So when utilization reaches 75% panelA will raise an alert. if utilization is increased from 75% to 90% then panelB will raise alert. if suddenly utilization reaches to 90% from 0% or above then both will raise alert so as per current implementation duplication will be avoided. so it will raise critical alert only.

@shtripat @r0h4n suggestion please

[shtripat]

@GowthamShanmugam @cloudbehl I like the idea. But one question, how frequent somebody looks at alert dashboard? Why I ask this question is because it might look more cluttered. If we maintain only for purpose of mail/snmp traps notifications I am in favor of this actually.

Comments?

[cloudbehl]

@shtripat Ideally user is not allowed to see the alert dashboard from grafana. It is just for the user to get the email notification and get the alerts on the UI notification.

[cybernth]

Sounds good to me. But we need to make sure those are deleted/updated when there are changes. Dealing with multiple copies is always going to be an issue. @GowthamShanmugam , @cloudbehl

[GowthamShanmugam]

We are facing few problem in above-proposed logic, problems are we are decided to maintain two alert panels in grafana for each resource, one is for raise warning alert and another one is for raise critical alert (because once grafana raised alert without clearing alert it won't raise alert again, but we need to raise alert like for threshold breach of 75-90% we need warning alert and 90 to 100% also we need alert for critical. but in the same panel, if alert raised for 75-90 then it won't raise alert again, it will raise only after clearing alert). So we can receive two different alerts from two panels.

But the problem which we facing here is:

1. In warning panel, we set an alert condition like 75 above to raise alert and in the critical panel we set an alert condition like 90 and above. when threshold breach 75 and above warning panel raised an alert and when breach 90% and above critical panel raised alert and then warning alert is replace by a critical alert in a central store. But when alert threshold decreased to 85 then critical panel raised OK alert and clearing alert replaced in central store. But the threshold is still 85% it needs to be in warning state in the central store. warning alert is also already in alerting state only. Because it is already raised an alert. still, it is waiting for clearing alert.

2. To overcome the above problem we decided to put a range in warning panel. The range in grafana acts like if threshold comes under the range then only it will raise an alert. if threshold goes above range or below range it will raise OK alert. Here we faced the problem like when threshold comes between 75% and 90% warning panel raise alert. and threshold goes above 90% warning panel will raise OK alert, So in UI warning alert is cleared and then Critical will raise an alert. When threshold decreases to 85% critical panel will raise OK alert and it will clear the critical alert and then 85% is within the range of 75% to 90% so warning panel raises alert and now etcd have warning alert.
   Here the big problem is alert severity changed like when threshold increases (warning to ok) and (ok to critical), When it decreases (critical to ok) and (ok to warning).

One more problem in this approach is two panels are independent so we can't say ok comes first and alert come second, it may happen like when the threshold breach 85% warning is raised, and when threshold increases to 90% there is a chance to the critical panel may raise critical alert first and warning panel may send OK alert second. So we will see an only OK alert in UI in this case even threshold is in 90% and above.

To avoid this kind of problem we decided to approaches:

1. warning panel can have range condition like 75% to 90% only it will raise an alert, for above range and below range, it raises ok alert. The critical panel can have a condition like 90% above to raise an alert. So when threshold goes 85% warning will raise an alert. and when threshold goes 90% then if any ok alert came we need to check the current value of alert, an alert is coming from warning panel but current value is still 90 and above so skip that ok alert. then the critical panel will raise alert so the warning is replaced by critical. when threshold reduced to 85% then critical raise ok alert but still current value is above 75% so skip that ok alert. so warning panel now send warning alert so critical is replaced by a warning. when an alert comes below 75% then warning dashbaord will raise ok alert and as per condition now the current value is less than 75% only now we can allow ok alert.

So in this solution we faced problem is like grafana won't give current value for clearing alert. Without current value, we can't compare like clearing alert is below 75%. So to overcome this problem we decided when clearing the alert we decided to read current value from graphite using rest-api call.
this is one solution

2. Another solution is we decided to keep warning alert and critical alert as two different alerts. two different alerts are displayed in UI for the same resource. Like when threshold goes 85% then warning alert is raised and it is displayed in UI. When threshold goes above 90% then critical is raised. and critical alert is showed in UI. So waring panel raises ok alert because current value is not in range of 75% to 90% so warning alert is cleared. so ok alert will shows for warning alert in UI. when alert decreased to 85 to 87 again then a critical alert is cleared and a warning is raised, So in UI critical alert shows as ok and a warning will show as alerting.

need suggestion @nthomas-redhat @shtripat @cloudbehl @r0h4n

[cybernth]

warning panel can have range condition like 75% to 90% only it will raise an alert, for above range and below range, it raises ok alert. The critical panel can have a condition like 90% above to raise an alert. So when threshold goes 85% warning will raise an alert. and when threshold goes 90% then if any ok alert came we need to check the current value of alert, an alert is coming from warning panel but current value is still 90 and above so skip that ok alert. then the critical panel will raise alert so the warning is replaced by critical. when threshold reduced to 85% then critical raise ok alert but still current value is above 75% so skip that ok alert. so warning panel now send warning alert so critical is replaced by a warning. when an alert comes below 75% then warning dashbaord will raise ok alert and as per condition now the current value is less than 75% only now we can allow ok alert.
So in this solution we faced problem is like grafana won't give current value for clearing alert. Without current value, we can't compare like clearing alert is below 75%. So to overcome this problem we decided when clearing the alert we decided to read current value from graphite using rest-api call.
this is one solution

@GowthamShanmugam , Are we doing the rest-api call to get graphite stats(for aggregation etc) in tendrl(any services)?

Another solution is we decided to keep warning alert and critical alert as two different alerts. two different alerts are displayed in UI for the same resource. Like when threshold goes 85% then warning alert is raised and it is displayed in UI. When threshold goes above 90% then critical is raised. and critical alert is showed in UI. So waring panel raises ok alert because current value is not in range of 75% to 90% so warning alert is cleared. so ok alert will shows for warning alert in UI. when alert decreased to 85 to 87 again then a critical alert is cleared and a warning is raised, So in UI critical alert shows as ok and a warning will show as alerting.

The problem with this approach is that we might see multiple warning/crtical and clearing alerts during the transion from normal->warning->critical and back. This might be confusing and at the end multiple unwnated alerts to send out. This will be feasible only if the clearing alert gives the metrics point at which the alert is generated. @GowthamShanmugam , @cloudbehl , can you guys check with grafana community if there ia a way to get this?
If we are not able get around the above issue(get metric datapoint in which the alert is generated), then next feasible option is get the metric data point from graphite(rest-api call) or from etcd while handling the alert set the sevrity based on that.

@r0h4n , @shtripat , Please provide your inputs

[shtripat]

I am pretty fine with getting data points from etcd or graphite as long it serves our purpose. Also in parallel we should check with grafana community on getting data points while clearing alerts. The solution we are suggesting could be a workaround till we have prpoer solution from garafan side itself. @r0h4n thoughts?

[r0h4n]

I would suggest not to update alerts that are already triggered, instead generate new alert with whatever new threshold and severity.

The reason for this is that the alerts might be serving as triggers for the Tendrl user and if we update the same old alert with new severity instead of creating a new alert, this might cause issues with the external triggers/automation around Tendrl alerts

[ltrilety]

I agree with #346 (comment). I think that we could live with two alerts for the same resource. Lets say in your example warning alert will be active if the resource is used more than 75%. In other words no clearing for the warning alert if the 90% barrier is breached. It makes sense as the warning is still valid even when the critical is raised.

BTW another point of view is for snmp traps. Probably all software working with snmp is counting with repeating alerts which makes sense as they are send via UDP. So for snmp traps we should send repeatedly that the warning threshold is reached till the problem is no more. Or the bigger threshold is reached, in such case change the trap to critical one and send it repeatedly till it ceased to be true.
There's no need to send some clearing traps for changing the trap severity from warning to critical and vice versa. Moreover it doesn't make sense to send both warning and critical at the same time. Only clearing trap should be send when the usage goes down under warning threshold.
Moreover we could use snmp inform for the clear alert as time is not an issue anymore and by using inform we have confidence the clear alert is received.

[GowthamShanmugam]

I agree with @ltrilety, @nthomas-redhat in this case we don't need range we can directly set condition above 75% in warning panel and above 95% in critical panel

[cloudbehl]

@ltrilety @r0h4n In agreement with both of you 👍

[cybernth]

I agree with #346 (comment). I think that we could live with two alerts for the same resource. Lets say in your example warning alert will be active if the resource is used more than 75%. In other words no clearing for the warning alert if the 90% barrier is breached. It makes sense as the warning is still valid even when the critical is raised.

@ltrilety , I don't agree with this. At any point we should not see two alerts for the same type i.e., for ex warning and critical alert for CPU utilization should not exist at the same time. @r0h4n ^

@GowthamShanmugam , I don't think you can leave out the ranges, without that you end up in the above situation.

Also I am worried about the number notifications(email/snmp) send out during the transition from normal->warning(and clear)->critical(and clear)->warning(and clear)->normal. Right now it 1:1 We need to see how we can suppress some of the clear notification during this transition