WORKAROUND - mail-service-notifications.sh script failes to send notification - errors out with "Requirement parameters are missing."

[ak2766]

When a apply Service... block is applied to a host with assign where..., when the service goes down the front end indicates a message has been sent but nothing gets to my Inbox. However, while still waiting for my email, another host went down and I got that email. This prompted me to believe that the issue was with the mail-service-notification script.

When I ran the command icinga2 object list --name 'mail-service-notification', I got the following output:

Object 'mail-service-notification' of type 'NotificationCommand':
  % declared in '/etc/icinga2/conf.d/commands.conf', lines 21:1-21:54
  * __name = "mail-service-notification"
  * arguments = null
  * command = [ "/etc/icinga2/scripts/mail-service-notification.sh" ]
    % = modified in '/etc/icinga2/conf.d/commands.conf', lines 22:3-22:76
  * env
    % = modified in '/etc/icinga2/conf.d/commands.conf', lines 24:3-37:3
    * HOSTADDRESS = "$address$"
    * HOSTALIAS = "$host.display_name$"
    * HOSTDISPLAYNAME = "$host.display_name$"
    * LONGDATETIME = "$icinga.long_date_time$"
    * NOTIFICATIONAUTHORNAME = "$notification.author$"
    * NOTIFICATIONCOMMENT = "$notification.comment$"
    * NOTIFICATIONTYPE = "$notification.type$"
    * SERVICEDESC = "$service.name$"
    * SERVICEDISPLAYNAME = "$service.display_name$"
    * SERVICEOUTPUT = "$service.output$"
    * SERVICESTATE = "$service.state$"
    * USEREMAIL = "$user.email$"
  * execute
    % = modified in 'methods-itl.conf', lines 48:3-48:40
    * arguments = [ "notification", "user", "cr", "itype", "author", "comment", "resolvedMacros", "useResolvedMacros" ]
    * deprecated = false
    * name = "Internal#PluginNotification"
    * side_effect_free = false
    * type = "Function"
  * name = "mail-service-notification"
  * package = "_etc"
  * source_location
    * first_column = 1
    * first_line = 21
    * last_column = 54
    * last_line = 21
    * path = "/etc/icinga2/conf.d/commands.conf"
  * templates = [ "mail-service-notification", "plugin-notification-command" ]
    % = modified in '/etc/icinga2/conf.d/commands.conf', lines 21:1-21:54
    % = modified in 'methods-itl.conf', lines 47:2-47:83
  * timeout = 60
  * type = "NotificationCommand"
  * vars = null
  * zone = ""

I noticed there was no SERVICENAME = "$service.name$" defined in the list above but yet it existed in the mail-service-notification.sh script. As such, I changed all occurences of SERVICENAME to SERVICEDESC in the mail-service-notification.sh script and I finally received the notification email!

The logs had this (replaced my servername with and my affected service name with :

[2017-08-08 07:41:48 +1000] information/Checkable: Checking for configured notifications for object '<FQDN>!<Service>'
[2017-08-08 07:41:48 +1000] information/Notification: Sending 'Problem' notification '<FQDN>!<Service>!mail-icingaadmin' for user 'icingaadmin'
[2017-08-08 07:41:48 +1000] information/Notification: Completed sending 'Problem' notification '<FQDN>!<Service>!mail-icingaadmin' for checkable 'FQDN!<Service>' and user 'icingaadmin'.
[2017-08-08 07:41:48 +1000] warning/PluginNotificationTask: Notification command for object '<FQDN>!<Service>' (PID: 8076, arguments: '/etc/icinga2/scripts/mail-service-notification.sh') terminated with exit code 1, output: Requirement parameters are missing.

Required parameters:
  -4 HOSTADDRESS ($address$)
  -d LONGDATETIME ($icinga.long_date_time$)
  -e SERVICENAME ($service.name$)
  -l HOSTNAME ($host.name$)
  -n HOSTDISPLAYNAME ($host.display_name$)
  -o SERVICEOUTPUT ($service.output$)
  -r USEREMAIL ($user.email$)
  -s SERVICESTATE ($service.state$)
  -t NOTIFICATIONTYPE ($notification.type$)
  -u SERVICEDISPLAYNAME ($service.display_name$)

Optional parameters:
  -6 HOSTADDRESS6 ($address6$)
  -b NOTIFICATIONAUTHORNAME ($notification.author$)
  -c NOTIFICATIONCOMMENT ($notification.comment$)
  -i ICINGAWEB2URL ($notification_icingaweb2url$, Default: unset)
  -f MAILFROM ($notification_mailfrom$, requires GNU mailutils (Debian/Ubuntu) or mailx (RHEL/SUSE))
  -v ($notification_sendtosyslog$, Default: false)

Expected Behavior

I expected to receive an email in my Inbox.

Current Behavior

No email was received in my Inbox relating to the service now in a CRITICAL state.

Possible Solution

Change all occurences of SERVICENAME to SERVICEDESC in the /etc/icinga2/scripts/mail-servicenotification.sh script.

Steps to Reproduce (for bugs)

1. Setup a host configuration

object Host "server.domain.com" {
  import "generic-host"
  address = "server.domain.com"
  vars.cipherise = true
  vars.os = "Linux"
  vars.wload = "5,3,3"
  vars.cload = "10,6,6"
}

2. Setup a service
apply Service "Testing" {
  import "generic-service"
  check_command = "cipherise-access-point"
  vars.service.name = "My Test Service"
  vars.apiurl = "https://logserver.domain.com:9880"
  vars.accesspoint = "Some access point"
  vars.notification["mail"] = {
    groups = [ "icingaadmins" ]
  }

  assign where match("server.domain.com", host.name)
}

3. Force service failure by blocking access to the APIURL port 9880
   I used iptables to block access to port 9880 from server.domain.com

4. Check email after Notification is purportedly sent out
   No email received in Inbox.

Context

Your Environment

• Version used (icinga2 --version):

icinga2 - The Icinga 2 network monitoring daemon (version: r2.7.0-1)

Copyright (c) 2012-2017 Icinga Development Team (https://www.icinga.com/)
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Application information:
  Installation root: /usr
  Sysconf directory: /etc
  Run directory: /run
  Local state directory: /var
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

System information:
  Platform: Ubuntu
  Platform version: 16.04.3 LTS (Xenial Xerus)
  Kernel: Linux
  Kernel version: 4.4.0-77-generic
  Architecture: x86_64

Build information:
  Compiler: GNU 5.3.1
  Build host: 6f646e1a58f5

• Operating System and version: Description (lsb_release -a):

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.3 LTS
Release:        16.04
Codename:       xenial

• Enabled features (icinga2 feature list):

Disabled features: compatlog debuglog gelf graphite influxdb livestatus opentsdb perfdata statusdata syslog
Enabled features: api checker command ido-mysql mainlog notification

• Icinga Web 2 version and modules (System - About):

Version
2.4.1
Copyright
© 2013-2017 The Icinga Project

• Config validation (icinga2 daemon -C):

information/cli: Icinga application loader (version: r2.7.0-1)
information/cli: Loading configuration file(s).
information/ConfigItem: Committing config item(s).
information/ApiListener: My API identity: <COMPANY NAME>-Monitor
warning/ApplyRule: Apply rule 'satellite-host' (in /etc/icinga2/conf.d/satellite.conf: 30:1-30:41) for type 'Dependency' does not match anywhere!
information/ConfigItem: Instantiated 1 ApiUser.
information/ConfigItem: Instantiated 1 ApiListener.
information/ConfigItem: Instantiated 3 Zones.
information/ConfigItem: Instantiated 1 FileLogger.
information/ConfigItem: Instantiated 1 Endpoint.
information/ConfigItem: Instantiated 1 NotificationComponent.
information/ConfigItem: Instantiated 2 NotificationCommands.
information/ConfigItem: Instantiated 100 Notifications.
information/ConfigItem: Instantiated 214 CheckCommands.
information/ConfigItem: Instantiated 1 Downtime.
information/ConfigItem: Instantiated 9 Hosts.
information/ConfigItem: Instantiated 1 IcingaApplication.
information/ConfigItem: Instantiated 2 HostGroups.
information/ConfigItem: Instantiated 3 Comments.
information/ConfigItem: Instantiated 1 UserGroup.
information/ConfigItem: Instantiated 3 TimePeriods.
information/ConfigItem: Instantiated 1 User.
information/ConfigItem: Instantiated 91 Services.
information/ConfigItem: Instantiated 3 ServiceGroups.
information/ConfigItem: Instantiated 1 ScheduledDowntime.
information/ConfigItem: Instantiated 1 CheckerComponent.
information/ConfigItem: Instantiated 1 ExternalCommandListener.
information/ConfigItem: Instantiated 1 IdoMysqlConnection.
information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
information/cli: Finished validating the configuration file(s).

• If you run multiple Icinga 2 instances, the zones.conf file (or icinga2 object list --type Endpoint and icinga2 object list --type Zone) from all affected nodes.
  N/A

[ak2766]

For completeness sake, adding the following lines:

for u in HOSTADDRESS LONGDATETIME HOSTNAME HOSTDISPLAYNAME SERVICEDESC SERVICEDISPLAYNAME SERVICEOUTPUT SERVICESTATE USEREMAIL NOTIFICATIONTYPE; do
  echo "${u} : ${!u}"
done

just before:

if [ ! "$HOSTADDRESS" ] || [ ! "$LONGDATETIME" ] \
|| [ ! "$HOSTNAME" ] || [ ! "$HOSTDISPLAYNAME" ] \
|| [ ! "$SERVICEDESC" ] || [ ! "$SERVICEDISPLAYNAME" ] \
|| [ ! "$SERVICEOUTPUT" ] || [ ! "$SERVICESTATE" ] \
|| [ ! "$USEREMAIL" ] || [ ! "$NOTIFICATIONTYPE" ]; then
  Error "Requirement parameters are missing."
fi

helped a lot in my troubleshooting steps for this issue. Had it not been a BASH script, it would definitely have taken a while longer to identify the root cause.

The output (found in the log file /var/log/icinga2/icinga2.log) looked like:

[2017-08-08 09:59:23 +1000] information/Notification: Completed sending 'Problem' notification '<FQDN>!<Service>!mail-icingaadmin' for checkable '<FQDN>!<Service>' and user 'icingaadmin'.
[2017-08-08 09:59:23 +1000] warning/PluginNotificationTask: Notification command for object '<FQDN>!<Service>' (PID: 20784, arguments: '/etc/icinga2/scripts/mail-service-notification.sh') terminated with exit code 1, output: HOSTADDRESS : server.domain.com
LONGDATETIME : 2017-08-08 09:59:23 +1000
HOSTNAME : icinga2
HOSTDISPLAYNAME : server.domain.com
SERVICENAME : 
SERVICEDISPLAYNAME : <Service>
SERVICEOUTPUT : !! Device <device> has been disconnected for too long !!
SERVICESTATE : CRITICAL
USEREMAIL : me@domain.com
NOTIFICATIONTYPE : PROBLEM
Requirement parameters are missing.

As can be seen above, the SERVICENAME reported a blank!

[hnnnglmbrg]

I can confirm this issue and that the workaround/possible solution ist working.

• Version used (icinga2 --version)

icinga2 - The Icinga 2 network monitoring daemon (version: r2.7.0-1)

Copyright (c) 2012-2017 Icinga Development Team (https://www.icinga.com/)
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Application information:
  Installation root: /usr
  Sysconf directory: /etc
  Run directory: /run
  Local state directory: /var
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

System information:
  Platform: Ubuntu
  Platform version: 16.04.3 LTS (Xenial Xerus)
  Kernel: Linux
  Kernel version: 4.4.0-87-generic
  Architecture: x86_64

Build information:
  Compiler: GNU 5.3.1
  Build host: 6f646e1a58f5

• Operating System and version: Description (lsb_release -a)

No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.3 LTS
Release:	16.04
Codename:	xenial

• enabled features (icinga2 feature list)

Disabled features: compatlog debuglog gelf graphite influxdb livestatus opentsdb perfdata statusdata syslog
Enabled features: api checker command ido-mysql mainlog notification

[ronator]

Possible Solution

Change all occurences of SERVICENAME to SERVICEDESC in the /etc/icinga2/scripts/mail-servicenotification.sh script.

Worked for me, too (Ubuntu14.04), thanks a lot @ak2766

[dnsmichi]

Update your notification commands too when updating the notification scripts. These changes are required when upgrading to v2.7.0.

https://github.com/Icinga/icinga2/blob/master/etc/icinga2/conf.d/commands.conf

Quote from https://www.icinga.com/2017/08/02/icinga-2-v2-7-0-released/

Make sure to checkout the documentation for the updated example configuration.

https://www.icinga.com/docs/icinga2/latest/doc/03-monitoring-basics/#notification-commands

[ronator]

Sorry for any possible rough tone, but this does not work for me, either. As I said, I use zones.d/ not conf.d/ so I do not see how that applies to users with cluster/satellite setup. No mails are send again after making the suggested changes.

My /etc/icinga2/conf.d/commands.conf is already exactly like https://github.com/Icinga/icinga2/blob/master/etc/icinga2/conf.d/commands.conf. I was trying to tell you that but we got off the wrong foot.

[fdellwing]

The problem for me was, that the HOSTADDRESS was missing, even after updating my commands.conf.

I'm using the deprecated way of satellite hosts and they simply have no address, the never had. But with this update the need to...

I simply added HOSTADDRESS="127.0.0.1" before

## Check required parameters (TODO: better error message)
## Keep formatting in sync with mail-host-notification.sh
if [ ! "$HOSTADDRESS" ] || [ ! "$LONGDATETIME" ] \
|| [ ! "$HOSTNAME" ] || [ ! "$HOSTDISPLAYNAME" ] \
|| [ ! "$SERVICENAME" ] || [ ! "$SERVICEDISPLAYNAME" ] \
|| [ ! "$SERVICEOUTPUT" ] || [ ! "$SERVICESTATE" ] \
|| [ ! "$USEREMAIL" ] || [ ! "$NOTIFICATIONTYPE" ]; then
  Error "Requirement parameters are missing."
fi

and everything works as before.