Helm chart fails to create the sidecar in the ruler pod in microservices (distributed) mode

[mwtzzz]

Describe the bug
In SimpleScaleable mode, the helm chart correctly deploys a sidecar inside the backend statefulset when the following parameters are set.

loki:
  rulerConfig:
    enable_alertmanager_v2: true
    alertmanager_url: http://alertmanager.services.stellar-ops.com:9093
    storage:
      type: local
      local:
        directory: /rules

ruler:
  replicas: 0
  maxUnavailable: 0
  extraArgs:
    - '-config.expand-env=true'
  extraEnvFrom:
    - secretRef:
        name: loki-dev-secrets

sidecar:
  rules:
    enabled: true
    folder: /rules/1
    logLevel: debug

But in Distributed mode, it fails to create the sidecar, which results in a ruler pod that does not work properly, with the following errors:

level=error ts=2025-02-21T21:11:16.53272825Z caller=log.go:216 msg="error running loki" err="mkdir /rules: read-only file system

I can fix the ruler by inserting a sidecar myself. But it would be nice if this were handled automatically by helm.

To Reproduce
Steps to reproduce the behavior:

1. loki 3.4.2 with helm chart loki-6.27.0
2. cat loki-dev-values-distributed | helm template loki grafana/loki --values=- --namespace ops > loki-01-distributed.yaml
3. look inside loki-01-distributed.yaml and you will see there is no sidecar container in the loki-ruler statefulset

Expected behavior
I expect a sidecar container inside the loki-ruler StatefulSet, in the same way as taken care of automatically by helm in deploymentMode: SimpleScaleable

Environment:

• Infrastructure: kubernetes cluster deployed with kubeadm
• Deployment tool: helm

Screenshots, Promtail config, or terminal output
N/A above description gives reproducible steps

[Jayclifford345]

Hi @mwtzzz, I spoke with the team again on this one. Their advice in microservice mode is to use object storage for rules. They understand your use case but believe rules that need to be uploaded in this way can also be added to object storage through automation.

We can leave this ticket open and if more support comes for this issue we can bump the sidecar issue up the list to revisit. If a community member also works on the issue and adds a PR I can look at fast-tracking as well. Apologies it's not the answer you are looking for at this time. :(

[mwtzzz]

@Jayclifford345 Thanks for the update, no worries I appreciate you looked into it, yes please leave the ticket open, it would be useful, not only to ourselves, but to others too.

[madaraszg-tulip]

I am also interested in migrating our loki-sc-rules sidecar to microservices mode, as we need to deploy rules from kubernetes automatically, and as far as I can tell there's no automation to deploy them to the object storage.

[mwtzzz]

I am also interested in migrating our loki-sc-rules sidecar to microservices mode, as we need to deploy rules from kubernetes automatically, and as far as I can tell there's no automation to deploy them to the object storage.

No, there isn't. You'd have to create your own custom tooling to POST api requests, which is pretty kludgey. Instead of doing that, I opted to not deploy the ruler from helm, instead I'm using a manifest like this:

# Source: loki/templates/ruler/statefulset-ruler.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: loki-ruler-custom
  namespace: ops-services
  labels:
    helm.sh/chart: loki-6.27.0
    app.kubernetes.io/name: loki
    app.kubernetes.io/instance: loki
    app.kubernetes.io/version: "3.4.2"
    app.kubernetes.io/component: ruler
    app.kubernetes.io/part-of: memberlist
  annotations:
    reloader.stakater.com/auto: "true"
    ignore-check.kube-linter.io/non-existent-service-account: "Argo creates the service account with helm"
spec:
  replicas: 1
  serviceName: loki-ruler
  selector:
    matchLabels:
      app.kubernetes.io/name: loki
      app.kubernetes.io/instance: loki
      app.kubernetes.io/component: ruler
  template:
    metadata:
      labels:
        helm.sh/chart: loki-6.27.0
        app.kubernetes.io/name: loki
        app.kubernetes.io/instance: loki
        app.kubernetes.io/version: "3.4.2"
        app.kubernetes.io/component: ruler
        app.kubernetes.io/part-of: memberlist
    spec:
      serviceAccountName: loki
      terminationGracePeriodSeconds: 300
      containers:
        - name: loki-sc-rules
          image: "kiwigrid/k8s-sidecar:1.30.0"
          imagePullPolicy: IfNotPresent
          env:
            - name: METHOD
              value: WATCH
            - name: LABEL
              value: "loki_rule"
            - name: FOLDER
              value: "/rules/1"
            - name: RESOURCE
              value: "both"
            - name: WATCH_SERVER_TIMEOUT
              value: "60"
            - name: WATCH_CLIENT_TIMEOUT
              value: "60"
            - name: LOG_LEVEL
              value: "debug"
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            readOnlyRootFilesystem: true
          volumeMounts:
            - name: sc-rules-volume
              mountPath: "/rules/1"
        - name: ruler
          image: docker.io/grafana/loki:3.4.2
          imagePullPolicy: IfNotPresent
          args:
            - -config.file=/etc/loki/config/config.yaml
            - -target=ruler
            - -config.expand-env=true
          ports:
            - name: http-metrics
              containerPort: 3100
              protocol: TCP
            - name: grpc
              containerPort: 9095
              protocol: TCP
            - name: http-memberlist
              containerPort: 7946
              protocol: TCP
          envFrom:
            - secretRef:
                name: loki-prd-secrets
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            readOnlyRootFilesystem: true
          readinessProbe:
            httpGet:
              path: /ready
              port: http-metrics
            initialDelaySeconds: 30
            timeoutSeconds: 1
          volumeMounts:
            - name: config
              mountPath: /etc/loki/config
            - name: runtime-config
              mountPath: /etc/loki/runtime-config
            - name: data
              mountPath: /var/loki
            - name: tmp
              mountPath: /tmp/loki
            - name: sc-rules-volume
              mountPath: "/rules/1"
          resources:
            {}
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app.kubernetes.io/component: ruler
            topologyKey: kubernetes.io/hostname
      volumes:
        - name: sc-rules-volume
          emptyDir: {}
        - name: config
          configMap:
            name: loki
            items:
              - key: "config.yaml"
                path: "config.yaml"
        - name: runtime-config
          configMap:
            name: loki-runtime
        - name: tmp
          emptyDir: {}
        - name: data
          emptyDir: {}