icinga and cluster idomysqlconnection statistics zero when clustered

[jwilliams-shadowsoft]

Expected Behavior

When passive check load testing with a single master, the "idomysqlconnection*" performance data from the icinga and cluster check plugins gave reasonable, non-zero statistics.

Current Behavior

When passive check load testing with clustered masters, the "idomysqlconnection*" performance data from the icinga and cluster check plugins gave all-zero statistics.

Possible Solution

Steps to Reproduce (for bugs)

1. Run a passive check load test against a single master with services defined to show the performance data from the icinga and cluster check plugins.
2. Observe that the idomysqlconnection statistics are non-zero.
3. Run the identical passive check load test against a clustered pair of masters with same services defined.
4. Observe that the idomysqlconnection statistics are zero.

Context

Your Environment

• Version used (icinga2 --version): 2.10.2
• Operating System and version: Centos 7
• Enabled features (icinga2 feature list): api checker command ido-mysql mainlog notification
• Icinga Web 2 version and modules (System - About):
• Config validation (icinga2 daemon -C): Valid
• If you run multiple Icinga 2 instances, the zones.conf file (or icinga2 object list --type Endpoint and icinga2 object list --type Zone) from all affected nodes.
  Valid zones file -- masters clustered correctly.

[dnsmichi]

Run a passive check load test

What's that? Can you share your script?

[dnsmichi]

That depends on where the IDO feature is active as well as where the corresponding check command is executed in HA enabled zones. If IDO is running on A, but ido-mysql executed on B, this returns zero stats (this is intentionally not synced between instances).

[dnsmichi]

Two options:

• Bind the ido-mysql Service forcefully with object authority updates on where the IdoMysqlConnection object is paused=false. Not sure whether this really works.
• Sync the metrics in a 15 seconds interval to both nodes. Not sure whether the traffic between instances suffers from that.

Will evaluate this together with our trainee in the coming months.

[htriem]

@Al2Klimov

[Al2Klimov]

Easy way to reproduce:

[root@tmp60ocJ9I9_default_1596805177280_50026 ~]# cat /etc/icinga2/zones.d/master/hosts.conf
for (i in range(10)) {
	object Host i {
		check_command = "icinga"
		check_interval = 1s
	}
}
[root@tmp60ocJ9I9_default_1596805177280_50026 ~]#

One of them will be on the wrong side anyway:

[root@tmp60ocJ9I9_default_1596805177280_50026 ~]# curl -sSku root:80eb24234d981225 -H 'Accept: application/json' https://localhost:5665/v1/objects/hosts\?pretty=1 |grep -A 5 idomysqlconnection_ido-mysql_queries_1min
                            "label": "idomysqlconnection_ido-mysql_queries_1min",
                            "max": null,
                            "min": null,
                            "type": "PerfdataValue",
                            "unit": "",
                            "value": 0.0,
--
                            "label": "idomysqlconnection_ido-mysql_queries_1min",
                            "max": null,
                            "min": null,
                            "type": "PerfdataValue",
                            "unit": "",
                            "value": 768.0,
--
                            "label": "idomysqlconnection_ido-mysql_queries_1min",
                            "max": null,
                            "min": null,
                            "type": "PerfdataValue",
                            "unit": "",
                            "value": 0.0,
--
                            "label": "idomysqlconnection_ido-mysql_queries_1min",
                            "max": null,
                            "min": null,
                            "type": "PerfdataValue",
                            "unit": "",
                            "value": 756.0,
--
                            "label": "idomysqlconnection_ido-mysql_queries_1min",
                            "max": null,
                            "min": null,
                            "type": "PerfdataValue",
                            "unit": "",
                            "value": 768.0,
--
                            "label": "idomysqlconnection_ido-mysql_queries_1min",
                            "max": null,
                            "min": null,
                            "type": "PerfdataValue",
                            "unit": "",
                            "value": 0.0,
--
                            "label": "idomysqlconnection_ido-mysql_queries_1min",
                            "max": null,
                            "min": null,
                            "type": "PerfdataValue",
                            "unit": "",
                            "value": 0.0,
--
                            "label": "idomysqlconnection_ido-mysql_queries_1min",
                            "max": null,
                            "min": null,
                            "type": "PerfdataValue",
                            "unit": "",
                            "value": 768.0,
--
                            "label": "idomysqlconnection_ido-mysql_queries_1min",
                            "max": null,
                            "min": null,
                            "type": "PerfdataValue",
                            "unit": "",
                            "value": 0.0,
--
                            "label": "idomysqlconnection_ido-mysql_queries_1min",
                            "max": null,
                            "min": null,
                            "type": "PerfdataValue",
                            "unit": "",
                            "value": 756.0,
[root@tmp60ocJ9I9_default_1596805177280_50026 ~]#