Mysql input: Slave_IO_Running returns unexpected values in newer version (1.20+)

[ekk1]

Relevant telegraf.conf

[[inputs.mysql]]
  servers = ["xxxxx"]
  metric_version = 2
  gather_global_variables = true
  gather_slave_status = true

[[outputs.prometheus_client]]
  listen = "xxxxx"
  metric_version = 2
  basic_username = "xxx"
  basic_password = "xxx"
  string_as_label = false

Logs from Telegraf

telegraf --test output

Current Version (tested on 1.21.4 ):

mysql,host=xxxx,slave_slave_io_running="Yes",slave_slave_sql_running="Yes" 1645512305000000000

Previous Version ( tested on 1.18.2 ):

mysql,host=xxxx,slave_slave_io_running=1i,slave_slave_sql_running=1i 1645512448000000000

System info

Telegraf 1.21.4, Debian Buster

Docker

No response

Steps to reproduce

1. Use telegraf 1.21.4 with pasted config to scrape a running mysql instance

Expected behavior

In previous versions, the value of slave_slave_io_running and slave_slave_sql_running is Int, which is easy for prometheus to process.

Actual behavior

In telegraf output, the value of slave_slave_io_running and slave_slave_sql_running is now String, which is hard for prometheus to process.
Also it may break previously defined prometheus rules, like:
mysql_slave_slave_sql_running == 0
Which is also hard to discover.. since the metrics would simply disappear, not triggering any alert.

Additional info

I was wondering if it's possible to add a option to make metrics like slave_slave_io_running to be parsed as int, like previous versions.

[arindamchoudhury]

I hope this will not be reverted back to integer.

[scottnemes]

I agree with @ekk1; this breaks a basic use case with Prometheus/VictoriaMetrics by having the replication thread states returned as strings instead of integers now.

I would like to see either a parameter added to revert back to integers for these fields, or have it return two fields; one with the string value, and another with the integer value like before. Either way this will help support both use cases, and not sacrifice one for the other.

[JesseRhoads]

A million +1s to @scottnemes here -- This applies to any of the recent changes for metrics to send as strings from #9966
I very much hope this will have a config parameter that allows those of us who can't support strings in metrics to be able to use these metrics the way we had been using them before it was changed. This was a pretty serious breaking change for anyone with prometheus-based metrics backends.

[ekk1]

Using enum processor resolved this issue. Here's some sample config

[[processors.enum]]
  [[processors.enum.mapping]]
    field = "slave_slave_io_running"
    dest = "slave_slave_io_running_int"
    default = 0
    [processors.enum.mapping.value_mappings]
      Yes = 1

The resulting metrics will be:
slave_slave_io_running="Yes",slave_slave_io_running_int=1i

There might be ways to override the original field, but this is enough for us now.

[powersj]

Hi,

Is there a simple way to set up a cluster to reproduce this? Does anyone have a docker-compose or something similar?

[hungnt1]

Slave_IO_Running only has Yes, No and Connecting state. I think we can parse and mapping this value. I can add to task and do this.