Skip to content

Queries will return wrong results during Ingester rolling when replication-factor=2 and unregister-on-shutdown=true #3844

Open

Description

Describe the bug

Queries will return wrong results during Ingester rolling.

To Reproduce

Steps to reproduce the behavior:

  1. Start Mimir 2.14 in K8s.
  2. Perform Operations(Read/Write/Others)
    Firstly, restart a Ingester pod.
    During the pod rolling (from the status 'Terminating' to 'Running 0/1'), some queries will return wrong results.
    After the pod status becomes 'Running 0/1', the right results come back.

Right result:
01
Wrong result during Ingester rolling:
02

Expected behavior

The result should keep the same during the Ingester rolling.

Environment

  • Infrastructure: k8s
  • Deployment tool: helm
  • some key config:
  • ingester.ring.replication-factor = 2
  • ingester.ring.unregister-on-shutdown = true

Others

I tried to change the config replication-factor and unregister-on-shutdown and test. Only 2 and true will reproduce this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions