Grafana Operator Integration not applying metrics_relabel_config

[wltbenade]

Hi all,

I've just implemented my first Integration CR, but have noticed that my metrics_relabel_metrics rule isn't being applied.

By default, the mssql exporter brings back the up metric, but in order to align it with the rest of our scraping configurations, I'd like to rename it to mssql_up.

Integration definition:

spec:
  config:
    autoscrape:
      enable: true
      metrics_instance: monitoring-system/metrics
      metrics_relabel_configs:
      - regex: up
        replacement: mssql_up
        source_labels:
        - __name__
        target_label: __name__
    connection_string: [REDACTED]
    instance: [REDACTED]
  name: mssql
  type:
    allNodes: false
    unique: true

Any idea what I'm doing wrong?

[github-actions]

This issue has not had any activity in the past 30 days, so the needs-attention label has been added to it.
If the opened issue is a bug, check to see if a newer release fixed your issue. If it is no longer relevant, please feel free to close this issue.
The needs-attention label signals to maintainers that something has fallen through the cracks. No action is needed by you; your issue will be kept open and you do not have to respond to this comment. The label will be removed the next time this job runs if there is new activity.
Thank you for your contributions!

[wltbenade]

Is anyone looking at this? I've just come across the same problem with the cloudwatch integration, which leads me to believe it might be affecting all integrations.

[wltbenade]

Is anyone looking at this? I've just come across the same problem with the cloudwatch integration, which leads me to believe it might be affecting all integrations.

CloudWatch Integration's definition:

spec:
  name: cloudwatch
  type:
    allNodes: false
    unique: true
  config:
    autoscrape:
      enable: true
      metrics_instance: monitoring-system/absaaccess-metrics
      metrics_relabel_configs:
        - source_labels: [tag_AppId,dimension_DBInstanceIdentifier]
          action: replace
          regex: (.*);.*-(.*)$
          target_label: "clusterid"
          replacement: $1-$2
        - action: labeldrop
          regex: tag_AppId|dimension_DBInstanceIdentifier|region|account
    sts_region: af-south-1
    fips_disabled: true
    discovery:
      exported_tags:
         AWS/RDS:
           - AppId
      jobs:
        - type: AWS/RDS
          roles:
            - role_arn: "[REDACTED]"
          regions:
            - [REDACTED]
          dimension_name_requirements:
            - "DBInstanceIdentifier"
          period: 60
          length: 120
          metrics:
            - name: CPUUtilization
              statistics: [Average,Sum,Minimum,Maximum]
            - name: DatabaseConnections
              statistics: [Average,Sum,Minimum,Maximum]
            - name: FreeableMemory
              statistics: [Average,Sum,Minimum,Maximum]
            - name: FreeStorageSpace
              statistics: [Average,Sum,Minimum,Maximum]
            - name: ReadThroughput
              statistics: [Average,Sum,Minimum,Maximum]
            - name: WriteThroughput
              statistics: [Average,Sum,Minimum,Maximum]
            - name: ReadLatency
              statistics: [Average,Sum,Minimum,Maximum]
            - name: WriteLatency
              statistics: [Average,Sum,Minimum,Maximum]
            - name: ReadIOPS
              statistics: [Average,Sum,Minimum,Maximum]
            - name: WriteIOPS
              statistics: [Average,Sum,Minimum,Maximum]
            - name: CheckpointLag
              statistics: [Average,Sum,Minimum,Maximum]
            - name: ConnectionAttempts
              statistics: [Average,Sum,Minimum,Maximum]
            - name: DiskQueueDepth
              statistics: [Average,Sum,Minimum,Maximum]
            - name: ReplicaLag
              statistics: [Average,Sum,Minimum,Maximum]
            - name: SwapUsage
              statistics: [Average,Sum,Minimum,Maximum]

[tpaschalis]

cc @captncraig in case he's seen something around this.

[captncraig]

@wltbenade can you share the agent config that gets generated when that integration is used?

[wltbenade]

Hey @captncraig,

Sure :) It'll just be HEAVILY redacted.

Thanks for checking!

root@grafana-agent-integrations-deploy-5b76b79c5f-4v88d:/# cat /var/lib/grafana-agent/config/agent.yml
integrations:
    cloudwatch_configs:
        - autoscrape:
            enable: true
            metrics_instance: monitoring-system/absaaccess-metrics
            write_relabel_configs:
                - action: replace
                  regex: (.*);.*-(.*)$
                  replacement: $1-$2
                  source_labels:
                    - tag_AppId
                    - dimension_DBInstanceIdentifier
                  target_label: clusterid
                - action: labeldrop
                  regex: tag_AppId|dimension_DBInstanceIdentifier|region|account
          discovery:
            exported_tags:
                AWS/RDS:
                    - dbInstanceId
                    - AppId
            jobs:
                - dimension_name_requirements:
                    - DBInstanceIdentifier
                  length: 120
                  metrics:
                    - name: CPUUtilization
                      statistics:
                        - Average
                        - Sum
                        - Minimum
                        - Maximum
                    - name: DatabaseConnections
                      statistics:
                        - Average
                        - Sum
                        - Minimum
                        - Maximum
                    - name: FreeableMemory
                      statistics:
                        - Average
                        - Sum
                        - Minimum
                        - Maximum
                    - name: FreeStorageSpace
                      statistics:
                        - Average
                        - Sum
                        - Minimum
                        - Maximum
                    - name: ReadThroughput
                      statistics:
                        - Average
                        - Sum
                        - Minimum
                        - Maximum
                    - name: WriteThroughput
                      statistics:
                        - Average
                        - Sum
                        - Minimum
                        - Maximum
                    - name: ReadLatency
                      statistics:
                        - Average
                        - Sum
                        - Minimum
                        - Maximum
                    - name: WriteLatency
                      statistics:
                        - Average
                        - Sum
                        - Minimum
                        - Maximum
                    - name: ReadIOPS
                      statistics:
                        - Average
                        - Sum
                        - Minimum
                        - Maximum
                    - name: WriteIOPS
                      statistics:
                        - Average
                        - Sum
                        - Minimum
                        - Maximum
                    - name: CheckpointLag
                      statistics:
                        - Average
                        - Sum
                        - Minimum
                        - Maximum
                    - name: ConnectionAttempts
                      statistics:
                        - Average
                        - Sum
                        - Minimum
                        - Maximum
                    - name: DiskQueueDepth
                      statistics:
                        - Average
                        - Sum
                        - Minimum
                        - Maximum
                    - name: ReplicaLag
                      statistics:
                        - Average
                        - Sum
                        - Minimum
                        - Maximum
                    - name: SwapUsage
                      statistics:
                        - Average
                        - Sum
                        - Minimum
                        - Maximum
                  period: 60
                  regions:
                    - af-south-1
                  roles:
                    - [REDACTED]
                    - [REDACTED]
                  type: AWS/RDS
          fips_disabled: true
          sts_region: af-south-1
    metrics:
        autoscrape:
            enable: false
logs:
    positions_directory: /var/lib/grafana-agent/data
metrics:
    configs:
        - name: monitoring-system/absaaccess-metrics
          remote_write:
            - headers:
                X-Scope-OrgID: [REDACTED]
              url: [REDACTED]
          scrape_configs:
            - honor_labels: true
              job_name: clustercontrol
              metric_relabel_configs:
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - instance
                  target_label: clusterid
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - instance
                  target_label: clusterid
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - instance
                  target_label: clusterid
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - instance
                  target_label: clusterid
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - instance
                  target_label: clusterid
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - instance
                  target_label: clusterid
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - instance
                  target_label: clusterid
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - instance
                  target_label: clusterid
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - instance
                  target_label: clusterid
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - instance
                  target_label: clusterid
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - instance
                  target_label: clusterid
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - instance
                  target_label: clusterid
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - instance
                  target_label: clusterid
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - instance
                  target_label: clusterid
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - clusterid
                  target_label: vip
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - clusterid
                  target_label: vip
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - clusterid
                  target_label: vip
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - clusterid
                  target_label: vip
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - clusterid
                  target_label: vip
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - clusterid
                  target_label: vip
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - clusterid
                  target_label: vip
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - clusterid
                  target_label: vip
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - clusterid
                  target_label: vip
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - clusterid
                  target_label: vip
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - clusterid
                  target_label: vip
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - clusterid
                  target_label: vip
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - clusterid
                  target_label: vip
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - clusterid
                  target_label: vip
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - clusterid
                  target_label: vip
              metrics_path: /federate
              params:
                match[]:
                    - '{job=~"(node|haproxy|postgres|mongo)",instance=~"[REDACTED]"}'
              scrape_interval: 1m
              static_configs:
                - labels:
                    clustercontrol_source: p01
                  targets:
                    - [REDACTED]
                - labels:
                    clustercontrol_source: p02
                  targets:
                    - [REDACTED]
                - labels:
                    clustercontrol_source: p03
                  targets:
                    - [REDACTED]
                - labels:
                    clustercontrol_source: p04
                  targets:
                    - [REDACTED]
                - labels:
                    clustercontrol_source: p05
                  targets:
                    - [REDACTED]
                - labels:
                    clustercontrol_source: p06
                  targets:
                    - [REDACTED]
                - labels:
                    clustercontrol_source: p07
                  targets:
                    - [REDACTED]
                - labels:
                    clustercontrol_source: p08
                  targets:
                    - [REDACTED]
                - labels:
                    clustercontrol_source: p09
                  targets:
                    - [REDACTED]
                - labels:
                    clustercontrol_source: p10
                  targets:
                    - [REDACTED]
                - labels:
                    clustercontrol_source: p11
                  targets:
                    - [REDACTED]
                - labels:
                    clustercontrol_source: p12
                  targets:
                    - [REDACTED]
                - labels:
                    clustercontrol_source: p13
                  targets:
                    - [REDACTED]
                - labels:
                    clustercontrol_source: p14
                  targets:
                    - [REDACTED]
                - labels:
                    clustercontrol_source: p15
                  targets:
                    - [REDACTED]
                - labels:
                    clustercontrol_source: p16
                  targets:
                    - [REDACTED]
                - labels:
                    clustercontrol_source: p17
                  targets:
                    - [REDACTED]
                - labels:
                    clustercontrol_source: p18
                  targets:
                    - [REDACTED]
                - labels:
                    clustercontrol_source: p19
                  targets:
                    - [REDACTED]
            - honor_labels: true
              job_name: redisenterprise
              metric_relabel_configs:
                - action: replace
                  regex: 3;[REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - node
                    - cluster
                  target_label: nodename
                - action: replace
                  regex: 2;[REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - node
                    - cluster
                  target_label: nodename
                - action: replace
                  regex: 1;[REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - node
                    - cluster
                  target_label: nodename
                - action: replace
                  regex: 1;[REDACTED]
                  replacement: zapalnrapp1038
                  source_labels:
                    - node
                    - cluster
                  target_label: nodename
                - action: replace
                  regex: 2;[REDACTED]
                  replacement: zapsdcrapp1259
                  source_labels:
                    - node
                    - cluster
                  target_label: nodename
                - action: replace
                  regex: 3;[REDACTED]
                  replacement: zaprnbrapp1160
                  source_labels:
                    - node
                    - cluster
                  target_label: nodename
                - regex: node
                  replacement: ""
                  source_labels:
                    - __name__
                  target_label: __name__
                - action: replace
                  regex: 1;[REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - bdb
                    - cluster
                  target_label: bdb
                - action: replace
                  regex: 9;[REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - bdb
                    - cluster
                  target_label: bdb
                - action: replace
                  regex: 2;[REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - bdb
                    - cluster
                  target_label: bdb
                - action: replace
                  regex: 5;[REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - bdb
                    - cluster
                  target_label: bdb
              metrics_path: /
              scheme: https
              scrape_interval: 30s
              static_configs:
                - labels:
                    clusterid: [REDACTED]
                  targets:
                    - [REDACTED]
              tls_config:
                insecure_skip_verify: true
    global:
        external_labels:
            cluster: monitoring-system/grafana-agent
    wal_directory: /var/lib/grafana-agent/data
server:
    log_level: info