Failed to scrape Prometheus endpoint: "error": "expected equal, got \":\" (\"INVALID\") while parsing: \"{\\\"status\\\":\""}

[balioune]

Hi all,
I'm using splunk-otel-collector to send Openshift metrics to our Splunk Enterprise.
In Openshift, scraped metrics are returned as JSON and it seems that the otel prometheus receiver does not support JSON format (see logs bellow)

# Splunk collector Pod error:
$ oc logs -f splunk-otel-collector-585b56bf77-tv6r2
2024/06/11 11:21:09 settings.go:480: Set config to [/tmp/otel-collector-config.yml]
2024/06/11 11:21:09 settings.go:541: Set memory limit to 460 MiB
2024/06/11 11:21:09 settings.go:526: Set soft memory limit set to 460 MiB
2024/06/11 11:21:09 settings.go:416: set "SPLUNK_LISTEN_INTERFACE" to "0.0.0.0"
2024-06-11T11:21:09.464Z        info    service@v0.100.0/service.go:102 Setting up own telemetry...
2024-06-11T11:21:09.464Z        info    service@v0.100.0/telemetry.go:103       Serving metrics {"address": ":8888", "level": "Normal"}
2024-06-11T11:21:09.464Z        debug   exporter@v0.100.0/exporter.go:273       Beta component. May change in the future.    {"kind": "exporter", "data_type": "metrics", "name": "splunk_hec"}
2024-06-11T11:21:09.464Z        debug   processor@v0.100.0/processor.go:301     Beta component. May change in the future.    {"kind": "processor", "name": "batch", "pipeline": "metrics"}
2024-06-11T11:21:09.464Z        debug   receiver@v0.100.0/receiver.go:308       Beta component. May change in the future.    {"kind": "receiver", "name": "prometheus", "data_type": "metrics"}
2024-06-11T11:21:09.478Z        info    service@v0.100.0/service.go:169 Starting otelcol...     {"Version": "v0.100.0", "NumCPU": 64}
2024-06-11T11:21:09.478Z        info    extensions/extensions.go:34     Starting extensions...
2024-06-11T11:21:09.479Z        info    prometheusreceiver@v0.100.0/metrics_receiver.go:275     Starting discovery manager   {"kind": "receiver", "name": "prometheus", "data_type": "metrics"}
2024-06-11T11:21:09.479Z        info    prometheusreceiver@v0.100.0/metrics_receiver.go:253     Scrape job added        {"kind": "receiver", "name": "prometheus", "data_type": "metrics", "jobName": "thanos-query"}
2024-06-11T11:21:09.479Z        debug   discovery/manager.go:286        Starting provider       {"kind": "receiver", "name": "prometheus", "data_type": "metrics", "provider": "static/0", "subs": "map[thanos-query:{}]"}
2024-06-11T11:21:09.479Z        info    service@v0.100.0/service.go:195 Everything is ready. Begin running and processing data.
2024-06-11T11:21:09.479Z        warn    localhostgate/featuregate.go:63 The default endpoints for all servers in components will change to use localhost instead of 0.0.0.0 in a future version. Use the feature gate to preview the new default.     {"feature gate ID": "component.UseLocalHostAsDefaultHost"}
2024-06-11T11:21:09.479Z        debug   discovery/manager.go:320        Discoverer channel closed       {"kind": "receiver", "name": "prometheus", "data_type": "metrics", "provider": "static/0"}
2024-06-11T11:21:09.479Z        info    prometheusreceiver@v0.100.0/metrics_receiver.go:340     Starting scrape manager {"kind": "receiver", "name": "prometheus", "data_type": "metrics"}
2024-06-11T11:21:15.987Z        debug   scrape/scrape.go:1346   Append failed   {"kind": "receiver", "name": "prometheus", "data_type": "metrics", "scrape_pool": "thanos-query", "target": "https://prometheus-k8s-openshift-monitoring.apps.myinternaldomain.fr:443/api/v1/query?query=node_ipvs_connections_total", "error": "expected equal, got \":\" (\"INVALID\") while parsing: \"{\\\"status\\\":\""}
2024-06-11T11:21:15.987Z        warn    internal/transaction.go:129     Failed to scrape Prometheus endpoint    {"kind": "receiver", "name": "prometheus", "data_type": "metrics", "scrape_timestamp": 1718104875959, "target_labels": "{__name__=\"up\", instance=\"prometheus-k8s-openshift-monitoring.apps.myinternaldomain.fr:443\", job=\"thanos-query\"}"}
2024-06-11T11:21:25.964Z        debug   scrape/scrape.go:1346   Append failed   {"kind": "receiver", "name": "prometheus", "data_type": "metrics", "scrape_pool": "thanos-query", "target": "https://prometheus-k8s-openshift-monitoring.apps.myinternaldomain.fr:443/api/v1/query?query=node_ipvs_connections_total", "error": "expected equal, got \":\" (\"INVALID\") while parsing: \"{\\\"status\\\":\""}
2024-06-11T11:21:25.965Z        warn    internal/transaction.go:129     Failed to scrape Prometheus endpoint    {"kind": "receiver", "name": "prometheus", "data_type": "metrics", "scrape_timestamp": 1718104885959, "target_labels": "{__name__=\"up\", instance=\"prometheus-k8s-openshift-monitoring.apps.myinternaldomain.fr:443\", job=\"thanos-query\"}"}
2024-06-11T11:21:35.963Z        debug   scrape/scrape.go:1346   Append failed   {"kind": "receiver", "name": "prometheus", "data_type": "metrics", "scrape_pool": "thanos-query", "target": "https://prometheus-k8s-openshift-monitoring.apps.myinternaldomain.fr:443/api/v1/query?query=node_ipvs_connections_total", "error": "expected equal, got \":\" (\"INVALID\") while parsing: \"{\\\"status\\\":\""}
2024-06-11T11:21:35.963Z        warn    internal/transaction.go:129     Failed to scrape Prometheus endpoint    {"kind": "receiver", "name": "prometheus", "data_type": "metrics", "scrape_timestamp": 1718104895959, "target_labels": "{__name__=\"up\", instance=\"prometheus-k8s-openshift-monitoring.apps.myinternaldomain.fr:443\", job=\"thanos-query\"}"}

$ cat otel-collector-config.yml : my config

receivers:
  otlp:
    protocols:
      grpc: {}
      http: {}
  prometheus:
    config:
      scrape_configs:
      - job_name: federate
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        honor_labels: true
        metrics_path: '/api/v1/query'
        params:
          'query':
            - 'node_ipvs_connections_total'
        static_configs:
          - targets: ['prometheus-k8s-federate-openshift-monitoring.apps.myinternaldomain.fr']
        scheme: https
        scrape_interval: 10s
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true
processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 75
    spike_limit_percentage: 15
  batch:
    send_batch_size: 10000
    timeout: 10s

exporters:
  splunk_hec:
    token: "XXXXXXXX-YYYY-XXXX-ZZZZ-XXXXXXXXXXXX"
    endpoint: "https://myinternaldomain.fr:8088/services/collector/event"
    source: "prometheus"
    sourcetype: "app_bit:prometheus"
    index: "app_bit"
    disable_compression: false
    timeout: 10s
    tls:
      insecure_skip_verify: true

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      exporters: [splunk_hec]

I test with curl the request that the prometheus receiver runs and here is the format of response.

# curl request: curl -H "Authorization: Bearer $TOKEN" -k "https://$HOST/api/v1/query?query=node_ipvs_connections_total" | python -m json.tool

response
{
    "status": "success",
    "data": {
        "resultType": "vector",
        "result": [
            {
                "metric": {
                    "__name__": "node_ipvs_connections_total",
                    "container": "kube-rbac-proxy",
                    "endpoint": "https",
                    "instance": "worker01",
                    "job": "node-exporter",
                    "namespace": "openshift-monitoring",
                    "pod": "node-exporter-qq5pn",
                    "service": "node-exporter"
                },
                "value": [
                    1718099280.005,
                    "0"
                ]
            }
        ]
    }
}

Any suggestion to solve this error ?

Is it possible to tell send directly the JSON to splunk instead of parsing the response ?

BR,