Foreman properties are not added to Icinga

[nodermatt]

I am working with a Foreman-Icinga2 setup distributed on 2 machines: 1 running Foreman (server1116) and the monitoring smart proxy, the other (server1100) running icinga2 with icingaweb2 and icingadirector.

Problem

If I simply change the hostgroup value of a host, the change is performed successfully (after I enable the change in Icinga director)

I, [2018-07-25T13:13:24.636192 11ee5e9b]  INFO -- : 10.35.147.116 - - [25/Jul/2018:13:13:24 +0200] GET /monitoring/host/server1117.cs.technik.fhnw.ch HTTP/1.1 200 202 0.1171

I, [2018-07-25T13:13:24.814211 11ee5e9b]  INFO -- : 10.35.147.116 - - [25/Jul/2018:13:13:24 +0200] GET /monitoring/host/server1117.cs.technik.fhnw.ch HTTP/1.1 200 202 0.1116

I, [2018-07-25T13:13:25.134359 11ee5e9b]  INFO -- : 10.35.147.116 - - [25/Jul/2018:13:13:25 +0200] POST /monitoring/host/server1117.cs.technik.fhnw.ch HTTP/1.1 200 432 0.2434

But if I tried to add a property in Foreman but upon submit I received the following error message on-screen:

In the proxy log I can see the below messages:
I, [2018-07-25T12:53:46.925010 11ee5e9b] INFO -- : 10.35.147.116 - - [25/Jul/2018:12:53:46 +0200] GET /monitoring/host/server1117.cs.technik.fhnw.ch HTTP/1.1 200 202 0.1127 I, [2018-07-25T12:53:47.107288 11ee5e9b] INFO -- : 10.35.147.116 - - [25/Jul/2018:12:53:47 +0200] GET /monitoring/host/server1117.cs.technik.fhnw.ch HTTP/1.1 200 202 0.1129 E, [2018-07-25T12:53:47.411002 11ee5e9b] ERROR -- : Error connecting to Icinga Director at https://server1100.cs.technik.fhnw.ch/icingaweb2/director/host?name=server1117.cs.technik.fhnw.ch: 304 Not Modified
I, [2018-07-25T12:53:47.412078 11ee5e9b] INFO -- : 10.35.147.116 - - [25/Jul/2018:12:53:47 +0200] POST /monitoring/host/server1117.cs.technik.fhnw.ch HTTP/1.1 400 154 0.2303 `

creation of Foreman host properties to icinga supported?

Background

Background: We want to add an expire date to hosts which we want to monitor and send notifications to the owner. This is part of our planned automated host creation process. New servers are deployed with Salt, then they register in Foreman which creates hosts in icinga for monitoring.

system info

foreman-monitoring: 1.0.1-1
Icinga2: r2.9.0-1
Icingaweb2: 2.6.0
icinga director: master

Cheers,
Nic

[dgoetz]

If I am correct, it sends the same data to the director again resulting in 304 which the Smart Proxy handles as error for being not a 200.

@timogoebel: Should we define 304 as okay? Should we forward 304 instead of 400 so it can be handled differently in Foreman Monitoring? Or would it be better to ensure Foreman Monitoring is not send an update if nothing changed?

[nodermatt]

Hi @dgoetz thanks for your quick follow-up on this.

FYI: For the moment I bypassed the problem by using a custom grain in my salt-minion that is copied to Foreman facts and added to the Icinga host upon creation of the machine. It is then visible in the custom variables section which seems a proper solution, too.

However, I noticed that if the grain value is changed (which is a rare ocassion as grains are mainly static by nature) the new value is not replicated to the icinga machine. I discovered this because I used a date time format that is not supported by the monitoring plugin (countdown_to) and after I changed it there was a discrepancy between the facts in Foreman and Icing, see below:

We don't expect to change the expire date on the servers often and if we have to, updating it separately in icinga is not a big deal. Nevertheless, I wanted to share this with you as it could be a feature idea for the future. I know that this is two separate concerns (my initial report and this grain thing) so if you like, I will create a second issue.

Cheers,
Nic

[nodermatt]

@dgoetz @timogoebel

It depends on whether or not it should be supported to sync the properties from Foreman to Icinga. Certainly, it would be nice to have icinga hosts in sync with the properties displayed in Foreman, like the host group f.eg. Especially because rule application is often based on such properties which would be updated once the Icinga server is restarted. If I understand this issue correctly, that's what needs to be done in order to reevaluate the rules, right?

However, as long as the sync is not supported (which is totally fine for the moment) the API should not return a status code that causes the Foreman Proxy to break the save function. Right know I cant save any changes (groups, salt states, etc.) in Foreman hosts because the Icinga host cant be updated.

My question: Would you guys agree to change the status code from 304 to 200? I myself am not fluent in PHP/Ruby but I'd try to fork and pull request a patch.

[timogoebel]

My question: Would you guys agree to change the status code from 304 to 200? I myself am not fluent in PHP/Ruby but I'd try to fork and pull request a patch.

Yeah, that would be very much appreciated. rest-client throws an exception when a not modified event happens, you can catch it like this.

[nodermatt]

Hi @timogoebel Sorry for the radio silence, I couldn't get into this.

If I understood you correctly, then I'd have to change the return type in the handle_http_exception for the director_client here

smart_proxy_monitoring/lib/smart_proxy_monitoring_icingadirector/director_client.rb

Line 86 in a0995f0

raise

to the following

      when RestClient::NotModified
        #raise
        # ignore NotModified
        true

I tried to add a test method that verifies the current behaviour (throwing the NotModified) but the method failed as a 500 code was returned.

Can you spot where I going wrong? Any help would be much appreciated.

https://github.com/nodermatt/smart_proxy_monitoring/commit/f3d5cfbfe8ecd0141efc2f873235719784ba5bd5