Description
openedon Jul 4, 2024
Hi! I've deployed the operator alongside a blackbox-exporter on an eks cluster, and i'm scraping blackbox targets thru Probe CRD. That works fine.
What's wrong?
Now i'm trying to add custom labels to each probe via the metricRelabelings config block on probe's CRD but it seems that the agent is ignoring it.
Software version
grafana/agent-operator:v0.39.0
Configuration
Example:
apiVersion: monitoring.grafana.com/v1alpha1
kind: GrafanaAgent
metadata:
name: blackbox
spec:
disableReporting: true
disableSupportBundle: false
enableConfigReadAPI: false
metrics:
externalLabels:
...
instanceSelector:
matchLabels:
blackbox: ""
scrapeInterval: 60s
---
apiVersion: monitoring.grafana.com/v1alpha1
kind: MetricsInstance
metadata:
labels:
blackbox: ""
name: blackbox
spec:
probeSelector:
matchLabels:
blackbox: ""
remoteWrite:
- headers:
X-Scope-OrgID: ...
url: ...
---
apiVersion: monitoring.coreos.com/v1
kind: Probe
metadata:
labels:
blackbox: ""
name: myprobe
spec:
jobName: someprobe
metricRelabelings:
- action: replace
replacement: hello
targetLabel: mynewlabel
module: probemodule
prober:
path: /probe
scheme: http
url: blackbox-exporter:9115
targets:
staticConfig:
static:
- <target>What should happen?
Maybe i'm mistaken, but it should create the proper scape config with the relabelConfigs? I checked out the configuration created by the operator and looks like this:
metrics:
configs:
- name: <namespace>/blackbox
remote_write:
- headers:
X-Scope-OrgID: ...
url: ...
scrape_configs:
- honor_timestamps: true
job_name: probe/<namespace>/someprobe
metrics_path: /probe
params:
module:
- probemodule
relabel_configs:
- source_labels:
- job
target_label: __tmp_prometheus_job_name
- replacement: someprobe
target_label: job
- source_labels:
- __address__
target_label: __param_target
- source_labels:
- __param_target
target_label: instance
- replacement: blackbox-exporter:9115
target_label: __address__
scheme: http
static_configs:
- labels:
namespace: <namespace>
targets:
- <target>
global:
external_labels:
__replica__: replica-$(STATEFULSET_ORDINAL_NUMBER)
...
scrape_interval: 60s
wal_directory: /var/lib/grafana-agent/data
server: {}I was hoping to see a metric_relabeling_configs block... but it is not showing up. Maybe i'm missing something? Are there any way to achieve what i'm looking for (to add custom labels to metrics via each probe CRD).
Logs
I've checked out operator logs and metrics instance logs and no relevan information there. Just usual reconciles and config reloadings.
Thank's in advance