Describe the bug
The is_acknowledged, is_flapping, and in_downtime flags are not historical, so they are not included in the state history stream. However, in order to prevent the problems described in Icinga/icinga-notifications#431 (review), we need to include these flags in the state history stream as well, though, these don't necessarily need to be persisted to the database.
I'll try to explain the rationale behind this in more detail:
The Icinga DB (Go) Notifications plugin/pipeline is designed to consume from the state history stream, which is a stream of state changes for hosts and services. This worked well until Icinga/icinga-notifications#431 was merged into Icinga Notifications, which moved the mute state tracking from objects to incidents. Though, unlike objects, incidents are ephemeral, and have no direct relationship to each other. So a currently muted incident due to a downtime might be replaced by a new incident that has no knowledge of the previous incident and thus no knowledge of the fact that the checkable is currently in downtime. In this case, Icinga DB (Go) would need to provide these flags with each state change event to Icinga Notifications, so that the newly opened incident can be muted immediately, preventing unnecessary notifications from being sent out. However, for this to work, Icinga DB (Go) would have to either query the database or Redis with each state change event to fetch the current state of these flags, but this has a severe flaw: For one, it would of course add a lot of overhead having to perform an additional query for each state change event, but that might be acceptable since we're already doing a similar thing for other things. However, the bigger problem is that this would not be atomic at all, meaning Icinga DB (Go) cannot guarantee that the current state of these flags aligns with the state history currently being processed, as they are streamed and processed separately.
By including these flags in the state history stream, we can ensure that they are always in sync with the state changes being processed. I don't see a way around this, but if you do, please let me know and you can gladly close this issue.
Your Environment
N/A
Describe the bug
The
is_acknowledged,is_flapping, andin_downtimeflags are not historical, so they are not included in the state history stream. However, in order to prevent the problems described in Icinga/icinga-notifications#431 (review), we need to include these flags in the state history stream as well, though, these don't necessarily need to be persisted to the database.I'll try to explain the rationale behind this in more detail:
The Icinga DB (Go) Notifications plugin/pipeline is designed to consume from the state history stream, which is a stream of state changes for hosts and services. This worked well until Icinga/icinga-notifications#431 was merged into Icinga Notifications, which moved the
mutestate tracking from objects to incidents. Though, unlike objects, incidents are ephemeral, and have no direct relationship to each other. So a currently muted incident due to a downtime might be replaced by a new incident that has no knowledge of the previous incident and thus no knowledge of the fact that the checkable is currently in downtime. In this case, Icinga DB (Go) would need to provide these flags with each state change event to Icinga Notifications, so that the newly opened incident can be muted immediately, preventing unnecessary notifications from being sent out. However, for this to work, Icinga DB (Go) would have to either query the database or Redis with each state change event to fetch the current state of these flags, but this has a severe flaw: For one, it would of course add a lot of overhead having to perform an additional query for each state change event, but that might be acceptable since we're already doing a similar thing for other things. However, the bigger problem is that this would not be atomic at all, meaning Icinga DB (Go) cannot guarantee that the current state of these flags aligns with the state history currently being processed, as they are streamed and processed separately.By including these flags in the state history stream, we can ensure that they are always in sync with the state changes being processed. I don't see a way around this, but if you do, please let me know and you can gladly close this issue.
Your Environment
N/A