Provide an error metric on each scrape

[Stift]

Is your feature request related to a problem? Please describe.
When scraping /metric all working queries are reported as metrics. Sometimes it happens that the software behind the database is changing the database schema and a query fails. Therefore only working metrics will be returned on /metric.

Describe the solution you'd like
I would like to propose a sql_query_error_count metric to be issued as well with the other metrics where errors or non-working queries could be counted. By that we could create an alarm when this is != 0.

If possible you could also think of a more sophisticated way and give a more detailed metric sql_query_error like

sql_query_error{collector=NAME, file=FILENAME} 1

Describe alternatives you've considered
The /sql_exporter_metrics endpoint is not helpful as this returns no information about errors and is called independently to the call of /metrics

Additional context
N/A

[burningalchemist]

Hi @Stift, thank you for your suggestion! The idea sounds good to me. 🙂👍

I'm not sure about mixing in internal metrics to the /metrics endpoint (I'd like to keep the domains separate). I do understand your concerns though. Let me think about it. We can always figure out how to control the additional metrics. 👍

[keithf4]

Making a second request for this. Current exporter we use has shown how critical a metric like this is. Allows us to know that something has gone wrong behind the scenes that we'd never know otherwise. Would be nice if it came across on the same /metrics endpoint as that's how almost every other exporter I've come across so far reports its internal metrics as well.

[burningalchemist]

Hey folks, a quick update for this one. I played a bit with it today, and prepared a prototype to expose scrape_errors per target/jobs with respective labels populated.

I need to test and merge another PR first, but this one goes shortly after (but also needs testing ofc). 🙂👍

Proofs: