[Helm - Loki - memcachedExporter.extraArgs] memcached.tls.enable and memcached.tls.insecure-skip-verify is not working as documented

[sasi1212]

Ref: https://github.com/grafana/loki/blob/main/production/helm/loki/values.yaml
Example content from values.yaml

  # -- Extra args to add to the exporter container.
  # Example:
  # extraArgs:
  #   memcached.tls.enable: true
  #   memcached.tls.cert-file: /certs/cert.crt
  #   memcached.tls.key-file: /certs/cert.key
  #   memcached.tls.ca-file: /certs/ca.crt
  #   memcached.tls.insecure-skip-verify: false
  #   memcached.tls.server-name: memcached

Especially memcached.tls.enable and memcached.tls.insecure-skip-verify is not accepting values. Meaning, if we add true or false to those directives, the memcached container is failing to start.

  extraArgs:
    log.level: info
    memcached.tls.ca-file: /loki-cache-certs/ca.crt
    memcached.tls.cert-file: /loki-cache-certs/tls.crt
    memcached.tls.enable: true
    memcached.tls.insecure-skip-verify: false
    memcached.tls.key-file: /loki-cache-certs/tls.key
    memcached.tls.server-name: loki-chunks-cache.ns-logging.svc.cluster.local
    web.config.file: /web-config/memcached-exporter-tls.conf

With the above settings added to Loki values.yaml, the memcahced pod

$ oc get pod logging-loki-chunks-cache-1
NAME                                READY   STATUS             RESTARTS       AGE
logging-loki-chunks-cache-1   1/2     CrashLoopBackOff   13 (77s ago)   42m

$ oc logs logging-loki-chunks-cache-1 -c exporter
memcached_exporter: error: unexpected true, try --help

Further, I have checked the container args and memcached_exporter utility help menu,

  - args:
    - --memcached.address=localhost:11211
    - --web.listen-address=0.0.0.0:9150
    - --log.level=info
    - --memcached.tls.ca-file=/loki-cache-certs/ca.crt
    - --memcached.tls.cert-file=/loki-cache-certs/tls.crt
    - --memcached.tls.enable=true
    - --memcached.tls.insecure-skip-verify
    - --memcached.tls.key-file=/loki-cache-certs/tls.key
    - --memcached.tls.server-name=loki-chunks-cache.ns-logging.svc.cluster.local
    - --web.config.file=/web-config/memcached-exporter-tls.conf

$ oc rsh -c exporter logging-loki-results-cache-0 bash -c "memcached_exporter --help 2>&1 | grep -A1 -E 'memcached.tls.enable|memcached.tls.insecure-skip-verify'"
      --[no-]memcached.tls.enable
                                 Enable TLS connections to memcached
--
      --[no-]memcached.tls.insecure-skip-verify
                                 Skip server certificate verification

Problems:

1. --memcached.tls.enable=true is not valid
2. memcached.tls.insecure-skip-verify: false we passed is translated as --memcached.tls.insecure-skip-verify, which is exactly opposite of the requested configuration.

As a workaround,

1. I removed "true" from memcached.tls.enable
2. I used no-memcached.tls.insecure-skip-verify: instead of memcached.tls.insecure-skip-verify: false

  extraArgs:
    log.level: info
    memcached.tls.ca-file: /loki-cache-certs/ca.crt
    memcached.tls.cert-file: /loki-cache-certs/tls.crt
    memcached.tls.enable: 
    no-memcached.tls.insecure-skip-verify: 
    memcached.tls.key-file: /loki-cache-certs/tls.key
    memcached.tls.server-name: loki-chunks-cache.ns-logging.svc.cluster.local
    web.config.file: /web-config/memcached-exporter-tls.conf

With the above extraArgs, the container args are formed as below,

  - args:
    - --memcached.address=localhost:11211
    - --web.listen-address=0.0.0.0:9150
    - --log.level=info
    - --memcached.tls.ca-file=/loki-cache-certs/ca.crt
    - --memcached.tls.cert-file=/loki-cache-certs/tls.crt
    - --memcached.tls.enable
    - --no-memcached.tls.insecure-skip-verify
    - --memcached.tls.key-file=/loki-cache-certs/tls.key
    - --memcached.tls.server-name=loki-chunks-cache.ns-logging.svc.cluster.local
    - --web.config.file=/web-config/memcached-exporter-tls.conf

Container process:

$ oc rsh -c exporter logging-loki-results-cache-0 bash -c "ps auxf | cat"
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
11211         50  0.0  0.0   4188  2688 pts/0    Ss+  08:12   0:00 bash -c ps auxf | cat
11211         56  0.0  0.0   7160  2688 pts/0    R+   08:12   0:00  \_ ps auxf
11211         57  0.0  0.0   5068  1792 pts/0    S+   08:12   0:00  \_ /usr/bin/coreutils --coreutils-prog-shebang=cat /usr/bin/cat
11211          1  0.0  0.0 1235556 12056 ?       Ssl  06:24   0:00 /usr/local/bin/memcached_exporter --memcached.address=localhost:11211 --web.listen-address=0.0.0.0:9150 --log.level=info --memcached.tls.ca-file=/loki-cache-certs/ca.crt --memcached.tls.cert-file=/loki-cache-certs/tls.crt --memcached.tls.enable --no-memcached.tls.insecure-skip-verify --memcached.tls.key-file=/loki-cache-certs/tls.key --memcached.tls.server-name=loki-chunks-cache.ns-logging.svc.cluster.local --web.config.file=/web-config/memcached-exporter-tls.conf

To Reproduce
Steps to reproduce the behavior:

1. Add extraArgs with memcached.tls.enable: true and memcached.tls.insecure-skip-verify: false
2. Deploy Loki helm chart

Expected behavior
extraArgs should work as documented. Either update the documentation or form the correct commandline arguments to memcached_exporter command.

Environment:

• Infrastructure: K8s, OCP
• Deployment tool: Helm