-
Notifications
You must be signed in to change notification settings - Fork 584
Fix no re-notification for non OK state changes with time delay #8562
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix no re-notification for non OK state changes with time delay #8562
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Write a test protocol. Test cases: #8545
Testprotocol: Before: Here After: [2020-12-14 11:42:11 +0100] information/Checkable: Checkable 'Icinga!Icinga-Service' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2020-12-14 11:42:11 +0100] information/Notification: Sending 'Problem' notification 'Icinga!Icinga-Service!Yonas-Notification' for user 'Yonas'
[2020-12-14 11:42:11 +0100] information/Notification: Completed sending 'Problem' notification 'Icinga!Icinga-Service!Yonas-Notification' for checkable 'Icinga!Icinga-Service' and user 'Yonas' using command 'fail'.
[2020-12-14 11:42:11 +0100] warning/PluginNotificationTask: Notification command for object 'Icinga!Icinga-Service' (PID: 8145, arguments: 'bash' '-c' 'exit 42') terminated with exit code 42, output:
[2020-12-14 11:42:39 +0100] information/Checkable: Checkable 'Icinga!Icinga-Service' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2020-12-14 11:42:39 +0100] information/Notification: Sending 'Problem' notification 'Icinga!Icinga-Service!Yonas-Notification' for user 'Yonas'
[2020-12-14 11:42:39 +0100] information/Notification: Completed sending 'Problem' notification 'Icinga!Icinga-Service!Yonas-Notification' for checkable 'Icinga!Icinga-Service' and user 'Yonas' using command 'fail'.
[2020-12-14 11:42:39 +0100] warning/PluginNotificationTask: Notification command for object 'Icinga!Icinga-Service' (PID: 8163, arguments: 'bash' '-c' 'exit 42') terminated with exit code 42, output:
[2020-12-14 11:43:07 +0100] information/Checkable: Checkable 'Icinga!Icinga-Service' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2020-12-14 11:43:07 +0100] information/Notification: Sending 'Problem' notification 'Icinga!Icinga-Service!Yonas-Notification' for user 'Yonas'
[2020-12-14 11:43:07 +0100] information/Notification: Completed sending 'Problem' notification 'Icinga!Icinga-Service!Yonas-Notification' for checkable 'Icinga!Icinga-Service' and user 'Yonas' using command 'fail'.
[2020-12-14 11:43:07 +0100] warning/PluginNotificationTask: Notification command for object 'Icinga!Icinga-Service' (PID: 8172, arguments: 'bash' '-c' 'exit 42') terminated with exit code 42, output: Scenario 2: [2020-12-14 11:48:35 +0100] notice/Checkable: State Change: Checkable 'Icinga!Icinga-Service' hard state change from OK to WARNING detected.
[2020-12-14 11:48:35 +0100] information/Checkable: Checkable 'Icinga!Icinga-Service' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2020-12-14 11:48:35 +0100] notice/Notification: Attempting to send notifications of type 'Problem' for notification object 'Icinga!Icinga-Service!Yonas-Notification'.
[2020-12-14 11:48:35 +0100] information/Notification: Sending 'Problem' notification 'Icinga!Icinga-Service!Yonas-Notification' for user 'Yonas'
[2020-12-14 11:48:35 +0100] notice/Process: Running command 'bash' '-c' 'exit 42': PID 8435
[2020-12-14 11:48:35 +0100] information/Notification: Completed sending 'Problem' notification 'Icinga!Icinga-Service!Yonas-Notification' for checkable 'Icinga!Icinga-Service' and user 'Yonas' using command 'fail'.
[2020-12-14 11:48:35 +0100] notice/Process: PID 8435 ('bash' '-c' 'exit 42') terminated with exit code 42
[2020-12-14 11:48:35 +0100] warning/PluginNotificationTask: Notification command for object 'Icinga!Icinga-Service' (PID: 8435, arguments: 'bash' '-c' 'exit 42') terminated with exit code 42, output:
[2020-12-14 11:48:38 +0100] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 2; Checks/s: 0.0333333
[2020-12-14 11:48:38 +0100] notice/NotificationComponent: Reminder notification 'Icinga!Icinga-Service!Yonas-Notification': Notification was sent out once and interval=0 disables reminder notifications. Scenario 3: [2020-12-14 11:50:28 +0100] information/Checkable: Checkable 'Icinga!Icinga-Service' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2020-12-14 11:50:28 +0100] notice/Notification: Attempting to send notifications of type 'Problem' for notification object 'Icinga!Icinga-Service!Yonas-Notification'.
[2020-12-14 11:50:28 +0100] notice/Notification: Not sending notifications for notification object 'Icinga!Icinga-Service!Yonas-Notification': before specified begin time (2 seconds)
[2020-12-14 11:50:31 +0100] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 2; Checks/s: 0.0333333
[2020-12-14 11:50:36 +0100] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 2; Checks/s: 0.0333333
[2020-12-14 11:50:36 +0100] notice/NotificationComponent: Attempting to send reminder notification 'Icinga!Icinga-Service!Yonas-Notification'.
[2020-12-14 11:50:36 +0100] notice/Notification: Attempting to send reminder notifications of type 'Problem' for notification object 'Icinga!Icinga-Service!Yonas-Notification'.
[2020-12-14 11:50:36 +0100] information/Notification: Sending reminder 'Problem' notification 'Icinga!Icinga-Service!Yonas-Notification' for user 'Yonas'
[2020-12-14 11:50:36 +0100] notice/Process: Running command 'bash' '-c' 'exit 42': PID 8560
[2020-12-14 11:50:36 +0100] information/Notification: Completed sending 'Problem' notification 'Icinga!Icinga-Service!Yonas-Notification' for checkable 'Icinga!Icinga-Service' and user 'Yonas' using command 'fail'.
[2020-12-14 11:50:36 +0100] notice/Process: PID 8560 ('bash' '-c' 'exit 42') terminated with exit code 42
[2020-12-14 11:50:36 +0100] warning/PluginNotificationTask: Notification command for object 'Icinga!Icinga-Service' (PID: 8560, arguments: 'bash' '-c' 'exit 42') terminated with exit code 42, output:
[2020-12-14 11:50:41 +0100] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 2; Checks/s: 0.0333333
[2020-12-14 11:50:51 +0100] notice/NotificationComponent: Reminder notification 'Icinga!Icinga-Service!Yonas-Notification': Notification was sent out once and interval=0 disables reminder notifications.
[2020-12-14 11:50:56 +0100] notice/Checkable: State Change: Checkable 'Icinga!Icinga-Service' hard state change from WARNING to CRITICAL detected.
[2020-12-14 11:50:56 +0100] information/Checkable: Checkable 'Icinga!Icinga-Service' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2020-12-14 11:50:56 +0100] notice/Notification: Attempting to send notifications of type 'Problem' for notification object 'Icinga!Icinga-Service!Yonas-Notification'.
[2020-12-14 11:50:56 +0100] notice/Notification: Not sending notifications for notification object 'Icinga!Icinga-Service!Yonas-Notification': before specified begin time (2 seconds)
[2020-12-14 11:51:01 +0100] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 2; Checks/s: 0.0333333
[2020-12-14 11:51:01 +0100] notice/NotificationComponent: Attempting to send reminder notification 'Icinga!Icinga-Service!Yonas-Notification'.
[2020-12-14 11:51:01 +0100] notice/Notification: Attempting to send reminder notifications of type 'Problem' for notification object 'Icinga!Icinga-Service!Yonas-Notification'.
[2020-12-14 11:51:01 +0100] information/Notification: Sending reminder 'Problem' notification 'Icinga!Icinga-Service!Yonas-Notification' for user 'Yonas'
[2020-12-14 11:51:01 +0100] notice/Process: Running command 'bash' '-c' 'exit 42': PID 8572
[2020-12-14 11:51:01 +0100] information/Notification: Completed sending 'Problem' notification 'Icinga!Icinga-Service!Yonas-Notification' for checkable 'Icinga!Icinga-Service' and user 'Yonas' using command 'fail'.
[2020-12-14 11:51:01 +0100] notice/Process: PID 8572 ('bash' '-c' 'exit 42') terminated with exit code 42
[2020-12-14 11:51:01 +0100] warning/PluginNotificationTask: Notification command for object 'Icinga!Icinga-Service' (PID: 8572, arguments: 'bash' '-c' 'exit 42') terminated with exit code 42, output:
[2020-12-14 11:51:06 +0100] notice/NotificationComponent: Reminder notification 'Icinga!Icinga-Service!Yonas-Notification': Notification was sent out once and interval=0 disables reminder notifications. |
Huh? Only one/two "exit 42" in the logs of scenario 2/3? Please re-test w/ |
Scenario 2: [2020-12-14 12:05:19 +0100] notice/Checkable: State Change: Checkable 'Icinga!Icinga-Service' hard state change from OK to WARNING detected.
[2020-12-14 12:05:19 +0100] information/Checkable: Checkable 'Icinga!Icinga-Service' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2020-12-14 12:05:19 +0100] notice/Notification: Attempting to send notifications of type 'Problem' for notification object 'Icinga!Icinga-Service!Yonas-Notification'.
[2020-12-14 12:05:19 +0100] information/Notification: Sending 'Problem' notification 'Icinga!Icinga-Service!Yonas-Notification' for user 'Yonas'
[2020-12-14 12:05:19 +0100] notice/Process: Running command 'bash' '-c' 'exit 42 # WARNING': PID 9063
[2020-12-14 12:05:19 +0100] information/Notification: Completed sending 'Problem' notification 'Icinga!Icinga-Service!Yonas-Notification' for checkable 'Icinga!Icinga-Service' and user 'Yonas' using command 'fail'.
[2020-12-14 12:05:19 +0100] notice/Process: PID 9063 ('bash' '-c' 'exit 42 # WARNING') terminated with exit code 42
[2020-12-14 12:05:19 +0100] warning/PluginNotificationTask: Notification command for object 'Icinga!Icinga-Service' (PID: 9063, arguments: 'bash' '-c' 'exit 42 # WARNING') terminated with exit code 42, output:
[2020-12-14 12:05:22 +0100] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 2; Checks/s: 0.0333333
[2020-12-14 12:05:22 +0100] notice/NotificationComponent: Reminder notification 'Icinga!Icinga-Service!Yonas-Notification': Notification was sent out once and interval=0 disables reminder notifications.
[...]
[2020-12-14 12:05:48 +0100] notice/Checkable: State Change: Checkable 'Icinga!Icinga-Service' hard state change from WARNING to CRITICAL detected.
[2020-12-14 12:05:48 +0100] information/Checkable: Checkable 'Icinga!Icinga-Service' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2020-12-14 12:05:48 +0100] notice/Notification: Attempting to send notifications of type 'Problem' for notification object 'Icinga!Icinga-Service!Yonas-Notification'.
[2020-12-14 12:05:48 +0100] information/Notification: Sending 'Problem' notification 'Icinga!Icinga-Service!Yonas-Notification' for user 'Yonas'
[2020-12-14 12:05:48 +0100] notice/Process: Running command 'bash' '-c' 'exit 42 # CRITICAL': PID 9084
[2020-12-14 12:05:48 +0100] information/Notification: Completed sending 'Problem' notification 'Icinga!Icinga-Service!Yonas-Notification' for checkable 'Icinga!Icinga-Service' and user 'Yonas' using command 'fail'.
[2020-12-14 12:05:48 +0100] notice/Process: PID 9084 ('bash' '-c' 'exit 42 # CRITICAL') terminated with exit code 42
[2020-12-14 12:05:48 +0100] warning/PluginNotificationTask: Notification command for object 'Icinga!Icinga-Service' (PID: 9084, arguments: 'bash' '-c' 'exit 42 # CRITICAL') terminated with exit code 42, output:
[2020-12-14 12:05:52 +0100] notice/NotificationComponent: Reminder notification 'Icinga!Icinga-Service!Yonas-Notification': Notification was sent out once and interval=0 disables reminder notifications.
[...]
[2020-12-14 12:25:00 +0100] notice/Checkable: State Change: Checkable 'Icinga!Icinga-Service' hard state change from CRITICAL to UNKNOWN detected.
[2020-12-14 12:25:00 +0100] information/Checkable: Checkable 'Icinga!Icinga-Service' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2020-12-14 12:25:00 +0100] notice/Notification: Attempting to send notifications of type 'Problem' for notification object 'Icinga!Icinga-Service!Yonas-Notification'.
[2020-12-14 12:25:00 +0100] information/Notification: Sending 'Problem' notification 'Icinga!Icinga-Service!Yonas-Notification' for user 'Yonas'
[2020-12-14 12:25:00 +0100] notice/Process: Running command 'bash' '-c' 'exit 42 # UNKNOWN': PID 9735
[2020-12-14 12:25:00 +0100] information/Notification: Completed sending 'Problem' notification 'Icinga!Icinga-Service!Yonas-Notification' for checkable 'Icinga!Icinga-Service' and user 'Yonas' using command 'fail'.
[2020-12-14 12:25:00 +0100] notice/Process: PID 9735 ('bash' '-c' 'exit 42 # UNKNOWN') terminated with exit code 42
[2020-12-14 12:25:00 +0100] warning/PluginNotificationTask: Notification command for object 'Icinga!Icinga-Service' (PID: 9735, arguments: 'bash' '-c' 'exit 42 # UNKNOWN') terminated with exit code 42, output:
[2020-12-14 12:25:01 +0100] notice/NotificationComponent: Reminder notification 'Icinga!Icinga-Service!Yonas-Notification': Notification was sent out once and interval=0 disables reminder notifications. Scenario 3: [2020-12-14 12:09:26 +0100] notice/Checkable: State Change: Checkable 'Icinga!Icinga-Service' hard state change from OK to WARNING detected.
[2020-12-14 12:09:26 +0100] information/Checkable: Checkable 'Icinga!Icinga-Service' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2020-12-14 12:09:26 +0100] notice/Notification: Attempting to send notifications of type 'Problem' for notification object 'Icinga!Icinga-Service!Yonas-Notification'.
[2020-12-14 12:09:26 +0100] notice/Notification: Not sending notifications for notification object 'Icinga!Icinga-Service!Yonas-Notification': before specified begin time (2 seconds)
[2020-12-14 12:09:33 +0100] notice/NotificationComponent: Attempting to send reminder notification 'Icinga!Icinga-Service!Yonas-Notification'.
[2020-12-14 12:09:33 +0100] notice/Notification: Attempting to send reminder notifications of type 'Problem' for notification object 'Icinga!Icinga-Service!Yonas-Notification'.
[2020-12-14 12:09:33 +0100] information/Notification: Sending reminder 'Problem' notification 'Icinga!Icinga-Service!Yonas-Notification' for user 'Yonas'
[2020-12-14 12:09:33 +0100] notice/Process: Running command 'bash' '-c' 'exit 42 # WARNING': PID 9274
[2020-12-14 12:09:33 +0100] information/Notification: Completed sending 'Problem' notification 'Icinga!Icinga-Service!Yonas-Notification' for checkable 'Icinga!Icinga-Service' and user 'Yonas' using command 'fail'.
[2020-12-14 12:09:33 +0100] notice/Process: PID 9274 ('bash' '-c' 'exit 42 # WARNING') terminated with exit code 42
[2020-12-14 12:09:33 +0100] warning/PluginNotificationTask: Notification command for object 'Icinga!Icinga-Service' (PID: 9274, arguments: 'bash' '-c' 'exit 42 # WARNING') terminated with exit code 42, output:
[2020-12-14 12:09:38 +0100] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 2; Checks/s: 0.0333333
[2020-12-14 12:09:38 +0100] notice/NotificationComponent: Reminder notification 'Icinga!Icinga-Service!Yonas-Notification': Notification was sent out once and interval=0 disables reminder notifications.
[...]
[2020-12-14 12:09:55 +0100] notice/Checkable: State Change: Checkable 'Icinga!Icinga-Service' hard state change from WARNING to CRITICAL detected.
[2020-12-14 12:09:55 +0100] information/Checkable: Checkable 'Icinga!Icinga-Service' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2020-12-14 12:09:55 +0100] notice/Notification: Attempting to send notifications of type 'Problem' for notification object 'Icinga!Icinga-Service!Yonas-Notification'.
[2020-12-14 12:09:55 +0100] notice/Notification: Not sending notifications for notification object 'Icinga!Icinga-Service!Yonas-Notification': before specified begin time (2 seconds)
[2020-12-14 12:09:58 +0100] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 2; Checks/s: 0.0333333
[2020-12-14 12:10:03 +0100] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 2; Checks/s: 0.0333333
[2020-12-14 12:10:03 +0100] notice/NotificationComponent: Attempting to send reminder notification 'Icinga!Icinga-Service!Yonas-Notification'.
[2020-12-14 12:10:03 +0100] notice/Notification: Attempting to send reminder notifications of type 'Problem' for notification object 'Icinga!Icinga-Service!Yonas-Notification'.
[2020-12-14 12:10:03 +0100] information/Notification: Sending reminder 'Problem' notification 'Icinga!Icinga-Service!Yonas-Notification' for user 'Yonas'
[2020-12-14 12:10:03 +0100] notice/Process: Running command 'bash' '-c' 'exit 42 # CRITICAL': PID 9284
[2020-12-14 12:10:03 +0100] information/Notification: Completed sending 'Problem' notification 'Icinga!Icinga-Service!Yonas-Notification' for checkable 'Icinga!Icinga-Service' and user 'Yonas' using command 'fail'.
[2020-12-14 12:10:03 +0100] notice/Process: PID 9284 ('bash' '-c' 'exit 42 # CRITICAL') terminated with exit code 42
[2020-12-14 12:10:03 +0100] warning/PluginNotificationTask: Notification command for object 'Icinga!Icinga-Service' (PID: 9284, arguments: 'bash' '-c' 'exit 42 # CRITICAL') terminated with exit code 42, output:
[2020-12-14 12:10:08 +0100] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 2; Checks/s: 0.0333333
[2020-12-14 12:10:08 +0100] notice/NotificationComponent: Reminder notification 'Icinga!Icinga-Service!Yonas-Notification': Notification was sent out once and interval=0 disables reminder notifications.
[...]
[2020-12-14 12:20:47 +0100] notice/Checkable: State Change: Checkable 'Icinga!Icinga-Service' hard state change from CRITICAL to UNKNOWN detected.
[2020-12-14 12:20:47 +0100] information/Checkable: Checkable 'Icinga!Icinga-Service' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2020-12-14 12:20:47 +0100] notice/Notification: Attempting to send notifications of type 'Problem' for notification object 'Icinga!Icinga-Service!Yonas-Notification'.
[2020-12-14 12:20:47 +0100] notice/Notification: Not sending notifications for notification object 'Icinga!Icinga-Service!Yonas-Notification': before specified begin time (2 seconds)
[2020-12-14 12:20:50 +0100] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 2; Checks/s: 0.0333333
[2020-12-14 12:20:50 +0100] notice/NotificationComponent: Attempting to send reminder notification 'Icinga!Icinga-Service!Yonas-Notification'.
[2020-12-14 12:20:50 +0100] notice/Notification: Attempting to send reminder notifications of type 'Problem' for notification object 'Icinga!Icinga-Service!Yonas-Notification'.
[2020-12-14 12:20:50 +0100] information/Notification: Sending reminder 'Problem' notification 'Icinga!Icinga-Service!Yonas-Notification' for user 'Yonas'
[2020-12-14 12:20:50 +0100] notice/Process: Running command 'bash' '-c' 'exit 42 # UNKNOWN': PID 9587
[2020-12-14 12:20:50 +0100] information/Notification: Completed sending 'Problem' notification 'Icinga!Icinga-Service!Yonas-Notification' for checkable 'Icinga!Icinga-Service' and user 'Yonas' using command 'fail'.
[2020-12-14 12:20:50 +0100] notice/Process: PID 9587 ('bash' '-c' 'exit 42 # UNKNOWN') terminated with exit code 42
[2020-12-14 12:20:50 +0100] warning/PluginNotificationTask: Notification command for object 'Icinga!Icinga-Service' (PID: 9587, arguments: 'bash' '-c' 'exit 42 # UNKNOWN') terminated with exit code 42, output:
[2020-12-14 12:20:55 +0100] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 2; Checks/s: 0.0333333
[2020-12-14 12:20:55 +0100] notice/NotificationComponent: Reminder notification 'Icinga!Icinga-Service!Yonas-Notification': Notification was sent out once and interval=0 disables reminder notifications. |
Warning, critical, ... where's the unknown notification? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
SetNoMoreNotifications()
theoretically affects reminder notifications. Test also the following:
- Re-enable reminder notifications by commenting out interval=0 and wait for some actual reminder notifications
- Re-disable reminder notifications (see above) and wait for the same amount of time, ensure reminder notifications won't fire
If there is no re-notification interval and no time delay configured, no reminders will be sent at all but with configured time delay the reminders will be sent exactly at the same time with and without intervals. |
Sigh. My mistake: "Defaults to 30 minutes." What happens within 10m w/ interval=3m and what with interval=0 ? |
Re-notification interval set to 0 only such messages are displayed. [2020-12-14 14:21:18 +0100] notice/NotificationComponent: Reminder notification 'Icinga!Icinga-Service!Yonas-Notification': Notification was sent out once and interval=0 disables reminder notifications. But if e.g. Interval is set to 3m, reminders are also sent when a notification problem is sent. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM apart from the small style issue.
lib/icinga/notification.cpp
Outdated
@@ -287,6 +287,12 @@ void Notification::BeginExecuteNotification(NotificationType type, const CheckRe | |||
*/ | |||
SetNextNotification(checkable->GetLastHardStateChange() + timesBegin + 1.0); | |||
|
|||
/** |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please use the same comment style as right above.
ca9dcf3
to
997ad86
Compare
resolves #8545