"too many failed ingesters" using memberlist

[github-vincent-miszczak]

Describe the bug
Cluster is down while it should not.

To Reproduce
Using Loki 2.1.0
The initial setup is 2 monolithic Loki 2.1.0 running with replication_factor: 2.
I add 2 nodes to the cluster, they all show ACTIVE looking at /ring.
I remove the first 2 nodes. They first show as LEAVING then they go Unhealthy.
They never leave this state (could not find a relevant config option).
At this point the cluster is down. Read or writes fail with something like:
level=warn ts=2021-02-19T14:42:46.766880514Z caller=logging.go:71 traceID=44198a5667db211f msg="POST /loki/api/v1/push (500) 147.959µs Response: \"at least 3 live replicas required, could only find 2\\n\"

Forgetting a single Unhealthy node using /ring buttons is enough to recover.

Expected behavior
2 ACTIVE nodes is sufficient for the cluster to be healthy, so the cluster should not be down when this condition is met.
Unhealthy nodes should leave the ring at some configurable point.

Environment:

• Infrastructure: ECS
• Deployment tool: Terraform

Screenshots, Promtail config, or terminal output

[github-vincent-miszczak]

Looks related to #2131

[github-vincent-miszczak]

Looking at the code, I found this on the path:
https://github.com/grafana/loki/blob/v2.1.0/pkg/distributor/distributor.go#L266
https://github.com/cortexproject/cortex/blob/v1.4.0/pkg/ring/replication_strategy.go#L29-L36

Both are computing a minimum success number to reach, while I would expect the value to just be replication_factor one as described in the doc: The number of ingesters to write to and read from.

I guess I'm missing something.

[github-vincent-miszczak]

I've been able to dig into the issue.
In fact it even happens with replication_factor: 2 and a single ingester down out of 3.

Promtail tells msg="error sending batch, will retry" status=500 error="server returned HTTP status 500 Internal Server Error (500): at least 2 live replicas required, could only find 1 while I expect the write to success.

My investigation points to https://github.com/grafana/loki/blob/master/vendor/github.com/cortexproject/cortex/pkg/ring/ring.go#L307 function that looks bogus.

Suppose you have 3 healthy/ACTIVE nodes.
When a Write occurs, nodes to write will be picked up to ReplicationFactor and then passed to the Filter function (that looks also bogus https://github.com/grafana/loki/blob/master/vendor/github.com/cortexproject/cortex/pkg/ring/replication_strategy.go#L30-L35)
When all the nodes are healthy, that's fine and the write succeed.

When a node is down/unhealthy, it's still marked ACTIVE in the ring and the state never changes. Unhealthy in the UI is the hearbeat timeout alone. But this timeout is not used when picking nodes in ring.Get() before filtering. So it picks 2 ACTIVE servers, 1 that is healthy and the other that is not(may be random depending on the tokens) and filtering removes the unhealthy one. Only one node remains and the issue occurs. 💥

The intended design as documented is Dynamo-style. My understanding is that we don't really care where the data goes, as long as we can write somewhere (the read path needs appropriate consideration to handle this) so the system is available. This is 4.6 sloppy quorum in the Dynamo paper. Writes should be performed on the first N healthy nodes from the preference list, which may not always be the first N nodes encountered while walking the consistent hashing ring.
The health check that is at the moment done in the filter should be done in the first phase, much like GetReplicationSetForOperation.

Note that it also happens when picking 5 nodes, RF=3, killing 2 nodes, same with 7 nodes, RF=4, killing 3 nodes.
I made a simple code to illustrate the issue: https://gist.github.com/vmiszczak-teads/29f61e803765087d7ff4b303583b51ca

Some other discussions around this:
cortexproject/cortex#1290
cortexproject/cortex#1521

Also it looks there is already support to get rid of dead nodes, let me know if you think it could be useful to plug it to Loki. https://github.com/grafana/loki/blob/v2.1.0/vendor/github.com/cortexproject/cortex/pkg/ring/basic_lifecycler_delegates.go#L137
(and sorry for the length of this post, I tried to give as much as details I could)

[the-maldridge]

I think this would be a fantastic addition. Its not super practical to constantly need to check /ring and manually dump the failed nodes.

[osipovdaniil]

Hi there! I have same problem grafana/helm-charts#213

Need helps

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[the-maldridge]

This ticket is referenced by in-progress PRs. Not stale.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.