How to set alertmanager in prometheus-community/prometheus helm chart?

[s-cho-25]

I'm using this chart:

https://artifacthub.io/packages/helm/prometheus-community/prometheus

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install my-prometheus prometheus-community/prometheus --values values.yaml --version 27.11.0

In the values.yaml, I set these serverFiles:

serverFiles:
  prometheus.yml:
    scrape_configs:
      - job_name: 'prometheus'
        static_configs:
          - targets:
              - localhost:9090
      - job_name: 'opencost'
        static_configs:
          - targets: ['opencost.opencost-exporter:9003']
  alerting_rules.yml:
    groups:
      - name: opencost-alerts
        rules:
          - alert: HighCost
            expr: node_total_hourly_cost > 1
            for: 1m
            labels:
              severity: critical
            annotations:
              description: "The cost is higher than expected for the last 1 minutes"
              summary: "Cost is over $1 for the last 1 minutes"
  alertmanager.yml:
    global:
      resolve_timeout: 5m
    route:
      receiver: 'slack-notifications'
      group_by: ['alertname']
      group_wait: 10s
      group_interval: 5m
      repeat_interval: 1h
    receivers:
      - name: 'slack-notifications'
        slack_configs:
          - api_url: 'https://hooks.slack.com/services/XXXXXXXX/YYYYYYYYY/ZZZZZZZZZZZZZZZZZZ'
            channel: '#alert-channel'
            send_resolved: true
            username: 'Prometheus'
            text: "{{ .CommonLabels.alertname }}: {{ .CommonAnnotations.description }}"

But when alert firing in Prometheus UI, there isn't any send log in alertmanager pod. And sure can't receive alert from Slack.

I'm sure the slack webhook is right. Where is wrong?

[adilburaksen]

The issue is that you're placing the Alertmanager configuration under serverFiles, which mounts files into the Prometheus server pod — not the Alertmanager pod. The Alertmanager reads its config from a separate ConfigMap populated by alertmanagerFiles.

Here's the corrected structure:

# Alertmanager config goes under alertmanagerFiles, not serverFiles
alertmanagerFiles:
  alertmanager.yml:
    global:
      resolve_timeout: 5m
    route:
      receiver: 'slack-notifications'
      group_by: ['alertname']
      group_wait: 10s
      group_interval: 5m
      repeat_interval: 1h
    receivers:
      - name: 'slack-notifications'
        slack_configs:
          - api_url: 'https://hooks.slack.com/services/XXXXXXXX/YYYYYYYYY/ZZZZZZ'
            channel: '#alert-channel'
            send_resolved: true
            username: 'Prometheus'
            text: "{{ .CommonLabels.alertname }}: {{ .CommonAnnotations.description }}"

serverFiles:
  prometheus.yml:
    scrape_configs:
      - job_name: 'prometheus'
        static_configs:
          - targets:
              - localhost:9090
      - job_name: 'opencost'
        static_configs:
          - targets: ['opencost.opencost-exporter:9003']
    # Note: when alertmanager.enabled=true (the default), the chart
    # automatically injects the alertmanagers block into prometheus.yml

  alerting_rules.yml:
    groups:
      - name: opencost-alerts
        rules:
          - alert: HighCost
            expr: node_total_hourly_cost > 1
            for: 1m
            labels:
              severity: critical
            annotations:
              description: "The cost is higher than expected for the last 1 minutes"
              summary: "Cost is over $1 for the last 1 minutes"

A few things to verify:

1. alertmanager.enabled: true — this is the default, but confirm it isn't set to false in your values. When enabled, the chart automatically configures alerting.alertmanagers in prometheus.yml pointing to the chart-deployed Alertmanager service.

2. Alertmanager pod is running — check with kubectl get pods | grep alertmanager. If the alertmanagerFiles config is malformed, the pod may be crash-looping.

3. Check Prometheus targets page — visit <prometheus-url>/targets and look for an alertmanager job to confirm Prometheus can reach it.

4. Alertmanager logs — kubectl logs <alertmanager-pod> will show if the Slack webhook call is failing (e.g., SSL errors, invalid URL, etc.).

The serverFiles.alertmanager.yml key you had would just create an unused file in the Prometheus server's /etc/config/ directory — Alertmanager never sees it.