Repository navigation
Incorrect json encoding for groupKey? #2172
Description
Host operating system: output of uname -a
Linux Influx-Grafana-Harbor 3.10.0-1062.4.3.el7.x86_64 #1 SMP Wed Nov 13 23:58:53 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
prometheus / blackbox_exporter / alertmanager versions:
$ rpm -q prometheus2 blackbox_exporter alertmanager
prometheus2-2.15.1-1.el7.centos.x86_64
blackbox_exporter-0.16.0-1.el7.centos.x86_64
alertmanager-0.20.0-2.el7.centos.x86_64
$ /usr/bin/blackbox_exporter --version
blackbox_exporter, version 0.16.0 (branch: HEAD, revision: 991f89846ae10db22a3933356a7d196642fcb9a9)
build user: root@64f600555645
build date: 20191111-16:27:24
go version: go1.13.4
$ prometheus --version
prometheus, version 2.15.1 (branch: HEAD, revision: 8744510c6391d3ef46d8294a7e1f46e57407ab13)
build user: root@4b1e33c71b9d
build date: 20191225-01:04:15
go version: go1.13.5
$ alertmanager --version
alertmanager, version 0.20.0 (branch: HEAD, revision: f74be0400a6243d10bb53812d6fa408ad71ff32d)
build user: root@00c3106655f8
build date: 20191211-14:13:14
go version: go1.13.5
$ yum info blackbox_exporter
prometheus 196/196
Installed Packages
Name : blackbox_exporter
Arch : x86_64
Version : 0.16.0
Release : 1.el7.centos
Size : 13 M
Repo : installed
From repo : prometheus
Summary : Blackbox prober exporter
URL : https://github.com/prometheus/blackbox_exporter
License : ASL 2.0
Description :
: The blackbox exporter allows blackbox probing of endpoints over HTTP, HTTPS, DNS, TCP and ICMP.
$ yum info prometheus2
Installed Packages
Name : prometheus2
Arch : x86_64
Version : 2.15.1
Release : 1.el7.centos
Size : 115 M
Repo : installed
From repo : prometheus
Summary : The Prometheus 2.x monitoring system and time series database.
URL : https://prometheus.io
License : ASL 2.0
Description :
: Prometheus is a systems and service monitoring system. It collects metrics from
: configured targets at given intervals, evaluates rule expressions, displays the
: results, and can trigger alerts if some condition is observed to be true.
What is the blackbox.yml module config.
$ cat /etc/prometheus/blackbox.yml
modules:
http_2xx:
prober: http
timeout: 5s
http:
valid_http_versions: ["HTTP/1.1", "HTTP/2"]
valid_status_codes: [] # Defaults to 2xx
method: GET
headers:
Accept-Language: en-US
no_follow_redirects: false
fail_if_ssl: true
fail_if_not_ssl: false
preferred_ip_protocol: "ip4" # defaults to "ip6"
ip_protocol_fallback: false # no fallback to "ip6"
What is the prometheus.yml scrape config.
# blackbox
- job_name: 'blackbox'
metrics_path: /probe
params:
module: [http_2xx] # Look for a HTTP 200 response.
static_configs:
- targets:
- http://srv1
...
What logging output did you get from adding &debug=true to the probe URL?
did not tried to add debug= to URL
What did you do that produced an error?
nothing, just site (server srv1) went down
What did you expect to see?
good (parsable) yaml received from alertmanager
What did you see instead?
received this un-parsable yaml :
{"status": "resolved", "groupLabels": {"alertname": "EndpointDown"}, "groupKey": "{}:{alertname="EndpointDown"}", "commonAnnotations": {"summary": "HTTP endpoint http://srv1 is down"}, "alerts": [{"status": "resolved", "labels": {"instance": "http://srv1", "job": "blackbox", "severity": "critical", "alertname": "EndpointDown"}, "endsAt": "2020-01-20T14:43:29.7054982Z", "generatorURL": "http://Influx-Grafana-Harbor:9090/graph?g0.expr=probe_success+%3D%3D+0&g0.tab=1", "fingerprint": "9280dcdd495fb85c", "startsAt": "2020-01-20T09:45:59.7054982Z", "annotations": {"summary": "HTTP endpoint http://srv1 is down"}}], "version": "4", "receiver": "telegram-bot", "externalURL": "http://Influx-Grafana-Harbor:9093", "commonLabels": {"instance": "http://srv1", "job": "blackbox", "severity": "critical", "alertname": "EndpointDown"}}
Further details:
used http://json.parser.online.fr/ to to parse received yml from alertmanager and it really shows that this yaml has error in it around second EndpointDown use, so if you change double-quotes to single-quotes with second EndpointDown use, it makes yaml to be parsed correctly.
Can you please suggest, who is wrong here, blackbox or alertmanager?!
Thanks
Activity
brian-brazil commentedon Jan 22, 2020
That'd be a bug in alertmanager or its json library.
[-]JSON: blackbox or alertmanager issue?[/-][+]Incorrect json encoding for groupKey?[/+]simonpasquier commentedon Jan 22, 2020
Can you provide more details? Not sure what you mean by "received this un-parsable yaml". Do you mean the payload sent by the webhook receiver?
mator commentedon Jan 23, 2020
@simonpasquier you're right, we're using telegram webhook for alerts. It is actually nopp/alertmanager-webhook-telegram and python/flask code to receive/parse alerts.
My configuration for the alertmanager:
python code to receive alerts from URL (noop/) above:
simonpasquier commentedon Jan 23, 2020
I suppose that the "erroneous" JSON that you've posted is taken from the webhook-telegram's output file. In that case, it's an issue with this application. AFAICT Alertmanager is properly escaping quotes in JSON payload (I've just verified it again and it would have broken many setups already if it was the case).
mator commentedon Jan 23, 2020
Thanks @simonpasquier
going to look further to python code what could be fixed.
Thanks again.