Ruler use a lot memory during quit

[canghai118]

During ruler stop，it consume a lot memory, and probably killed by oom. In this case the information about itself is not unregister from the ring

below is the log captured by kubectl logs -f

level=info ts=2020-01-06T10:20:03.315150389Z caller=cortex.go:230 msg=initialising module=distributor
level=info ts=2020-01-06T10:20:03.315258575Z caller=cortex.go:230 msg=initialising module=ruler
level=info ts=2020-01-06T10:20:03.318200027Z caller=ruler.go:188 msg="ruler up and running"
level=info ts=2020-01-06T10:20:03.318295245Z caller=main.go:76 msg="Starting Cortex" version="(version=, branch=, revision=)"
level=info ts=2020-01-06T10:20:03.325922888Z caller=scheduler.go:246 msg="scheduler: updating rules for user" user_id=cidata num_groups=7 is_deleted=false
level=info ts=2020-01-06T10:20:03.326266046Z caller=lifecycler.go:460 msg="entry not found in ring, adding with no tokens"
level=info ts=2020-01-06T10:20:03.330674539Z caller=lifecycler.go:352 msg="auto-joining cluster after timeout"
level=error ts=2020-01-06T10:22:00.885438513Z caller=etcd.go:144 msg="error decoding key" key=ruler/ring err="snappy: corrupt input"
level=info ts=2020-01-06T10:22:03.333375976Z caller=lifecycler.go:599 msg="found empty ring, inserting tokens"
level=info ts=2020-01-06T10:22:25.497223122Z caller=signals.go:54 msg="=== received SIGINT/SIGTERM ===\n*** exiting"
level=info ts=2020-01-06T10:22:25.499116956Z caller=cortex.go:256 msg=stopping module=ruler
level=info ts=2020-01-06T10:22:25.503165591Z caller=notifier.go:602 msg="Stopping notification manager..."
level=info ts=2020-01-06T10:22:25.504910468Z caller=ruler.go:202 msg="shutting down rules scheduler"
level=info ts=2020-01-06T10:22:25.505102585Z caller=ruler.go:205 msg="waiting for workers to finish"
rpc error: code = Unknown desc = Error: No such container: 8c98cca62de0a0c28ba696daa5360f2f90c649ce9de178db5abcd04553156c8a%

Has anyone encountered this problem?

[jtlisi]

@canghai118 Would you mind posting your config? Specifically how many rulers you are running and what backend stores you are using?

[canghai118]

There are about 53 recording rules total, and only 2 alert rules, our backend is cassandra for both index and chunk store. The memory usage only increase during stop process， During normal running the memory is about 100MB per ruler instance

I run ruler as k8s deployment. here is the pod spec

apiVersion: v1
kind: Pod
metadata:
  annotations:
    prometheus.io/path: /metrics
    prometheus.io/port: "9009"
    prometheus.io/scrape: "true"
  creationTimestamp: "2020-01-07T03:01:41Z"
  generateName: cortex-ruler-77d4c6549b-
  labels:
    app: cortex
    app.kubernetes.io/component: ruler
    app.kubernetes.io/instance: cortex
    app.kubernetes.io/name: cortex
    pod-template-hash: 77d4c6549b
  name: cortex-ruler-77d4c6549b-bhxcg
  namespace: cidata
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: cortex-ruler-77d4c6549b
    uid: 06329dff-30fa-11ea-8759-00163e0e50bd
  resourceVersion: "87963368"
  selfLink: /api/v1/namespaces/cidata/pods/cortex-ruler-77d4c6549b-bhxcg
  uid: 077daede-30fa-11ea-8759-00163e0e50bd
spec:
  containers:
  - args:
    - -config.file=/etc/cortex.yaml
    - -target=ruler
    - -ruler.alertmanager-url=http://cortex-alertmanager:9009/api/prom/alertmanager/
    - -ruler.configs.url=http://cortex-configs:9009
    - -ruler.num-workers=16
    - -ruler.enable-sharding=true
    - -ruler.prefix=ruler/
    - -ruler.store=etcd
    - -ruler.etcd.endpoints=http://192.168.52.9:2379
    - -ruler.etcd.endpoints=http://192.168.52.4:2379
    - -ruler.etcd.endpoints=http://192.168.52.8:2379
    - -ruler.final-sleep=10s
    env:
    - name: GODEBUG
      value: madvdontneed=1
    image: quay.azk8s.cn/cortexproject/cortex:v0.4.0
    imagePullPolicy: IfNotPresent
    name: ruler
    ports:
    - containerPort: 9009
      name: http
      protocol: TCP
    resources:
      limits:
        cpu: "1"
        memory: 4Gi
      requests:
        cpu: 101m
        memory: 256Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /etc/cortex.yaml
      name: config
      subPath: cortex.yaml
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-khmhf
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  imagePullSecrets:
  - name: sa-default-aliyun-hangzhou
  - name: sa-default-aliyun-shanghai
  nodeName: prod-w-32-65
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 300
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - configMap:
      defaultMode: 420
      items:
      - key: cortex.yaml
        path: cortex.yaml
      name: cortex
    name: config
  - name: default-token-khmhf
    secret:
      defaultMode: 420
      secretName: default-token-khmhf
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2020-01-07T03:01:41Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2020-01-07T03:01:43Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2020-01-07T03:01:43Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2020-01-07T03:01:41Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: docker://d7b19ce1651daaa7014c00df00d9feada442ea3005d0f89e34ca1694b4fd082b
    image: quay.azk8s.cn/cortexproject/cortex:v0.4.0
    imageID: docker-pullable://quay.azk8s.cn/cortexproject/cortex@sha256:98b875dc9069ada20593e08c5720750755a78606ec6a97bee3e8caabcf2c5360
    lastState: {}
    name: ruler
    ready: true
    restartCount: 0
    state:
      running:
        startedAt: "2020-01-07T03:01:42Z"
  hostIP: 192.168.32.65
  phase: Running
  podIP: 10.62.42.224
  qosClass: Burstable
  startTime: "2020-01-07T03:01:41Z"

The content of cortex config file is:

auth_enabled: true

server:
  http_listen_port: 9009 
  grpc_listen_port: 9095
  grpc_server_max_recv_msg_size: 104857600
  grpc_server_max_send_msg_size: 104857600
  grpc_server_max_concurrent_streams: 1000

query_range:
  split_queries_by_interval: 24h
  align_queries_with_step: true
  cache_results: true
  results_cache:
    max_freshness: 5m
    cache:
      enable_fifocache: true
      fifocache:
        size: 4096
        validity: 1h
      redis:
        endpoint: cortex-redis:6379
        expiration: 168h

frontend:
  max_outstanding_per_tenant: 100
  log_queries_longer_than: 5s
  

chunk_store:
  chunk_cache_config:
    enable_fifocache: true
    fifocache:
      size: 4096
      validity: 1h
    redis:
      endpoint: cortex-redis:6379
      expiration: 24h

distributor:
  shard_by_all_labels: true
  pool:
    health_check_ingesters: true

ingester_client:
  grpc_client_config:
    max_recv_msg_size: 104857600
    max_send_msg_size: 104857600
    use_gzip_compression: true

ingester:
  #chunk_idle_period: 15m 
  lifecycler: 
    # We want to start immediately and flush on shutdown.
    join_after: 0
    claim_on_rollout: false
    final_sleep: 0s
    num_tokens: 512

    ring:
      kvstore:
        store: etcd
        etcd:
          endpoints:
            - http://192.168.52.9:2379
            - http://192.168.52.4:2379
            - http://192.168.52.8:2379
            

      replication_factor: 2

schema:
  configs:
  - from: 2019-11-01
    store: cassandra
    object_store: cassandra
    schema: v10
    index:
      prefix: index_
      period: 168h
    chunks:
      prefix: chunk_
      period: 24h

storage:
  cassandra:
    addresses: 192.168.52.6,192.168.52.7,192.168.52.5 
    keyspace: cortex
    timeout: 60s

limits:
  ingestion_rate: 100000
  ingestion_burst_size: 200000

table_manager:
  retention_deletes_enabled: true
  retention_period: 840h

@jtlisi

[jtlisi]

I haven't been able to replicate this behaviour. @canghai118 if you could run pprof on a ruler when it is exiting and share the trace, I would be really interested in seeing it.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.