Is it possible to update an alert in OpsGenie when it changes the state in Icinga?

[iuriaranda]

When an alert is in a WARNING state and it's notified to OpsGenie, and then it goes from WARNING to CRITICAL in icinga, the status is not updated in OpsGenie and users are not notified of the status change. Is there a way we can achieve this?

Current behavior

1. Check enters a WARNING state in Icinga
2. Icinga notifies OpsGenie
3. OpsGenie creates a new alert
4. OpsGenie notifies the appropriate users
5. The Icinga check goes from WARNING to CRITICAL
6. Icinga notifies of the status change to OpsGenie
7. OpsGenie handles the Icinga notification as a repetition of the previous one and increases the count

Expected behavior

1. Check enters a WARNING state in Icinga
2. Icinga notifies OpsGenie
3. OpsGenie creates a new alert
4. OpsGenie notifies the appropriate users
5. The Icinga check goes from WARNING to CRITICAL
6. Icinga notifies of the status change to OpsGenie
7. OpsGenie updates the alert with the new status (CRITICAL)
8. OpsGenie notifies the appropriate users of the status change

[halitokumus]

Updating alerts is not the typical way of doing this, but you can customize your Icinga integration to change the alert alias from the current {{host_name}}_{{service_desc}} to {{host_name}}_{{service_desc}}_{{service_state}} , so that when the state changes OpsGenie will not count it as repetition but create a new alert and start the notifications.

[iuriaranda]

Yes but then when the alert turns green in Icinga, it wouldn't be closed in Opsgenie. So it isn't a practical option IMHO

[janboll]

We struggle with the same issue. Currently we stick to the solution @halitokumus provided, but it is very unpleasant cause we have to manually close every incident.
For certain monitorings, like disk, load or memory usage it is very common to go from warning to critical. Therefore i would not consider this an edge case.

[halitokumus]

Understood. I'll get this into our next sprint and we'll try to come up with a solution. We'll keep here updated about the progress.

[halitokumus]

We have an ongoing development for Incident&Service management feature at OpsGenie. When the Incident feature is released, you're going to be able to solve this problem easily. For example, Icinga is going to create an alert which can be turned to an incident; and when the state changes from WARNING to CRITICAL, the incident's priority will be increased and notifications will start. Just an example.

[peterb-vetct]

Was this ever implemented? Still have the issue exists.

[codedesperate]

We still have this issue, we use Datadog though. When a Datadog alert is send to Opsgenie that has the remaining disk space in the title, Opsgenie does not update the title.

[jonneolie]

We still have this issue, we use Datadog though. When a Datadog alert is send to Opsgenie that has the remaining disk space in the title, Opsgenie does not update the title.

I have this same issue as you, this is so frustrated.

[codedesperate]

@jonneolie
When Datadog's own On-call solution comes out of Beta, we are thinking of switching to that because of this issue.

[jonneolie]

@jonneolie When Datadog's own On-call solution comes out of Beta, we are thinking of switching to that because of this issue.

@codedesperate I created a new issue, because this one seems closed too..
#88

For us that's gonna be a hard one, because we are a managed service provider, who has more customers who all use a separate Datadog instance. So we need something to combine it all together into one alerting tool.