Random MOVED errors.

[Hamper]

I use MaxRedirects: 1 in cluster client and redis 4.0.2, and sometimes see MOVED errors, but cluster nodes output not changes, cluster not resharded.

[vmihailenco]

Are you able to isolate the problem and write a program that reproduces the problem?

[Hamper]

I will try to isolate, but it's very difficult, with total load of 300000 rps I see about 3-10 errors per second.

[Hamper]

I think that I get this error because with MaxRedirects: 1 if first request is RetryableError than client select random node and that node returns MOVED error to first failed node. In this case MOVED error message looks unclear. You can close issue if you think that this behavior is right.

[sqaxomonophonen]

We're seeing this too. Here's a simple program to reproduce it (it simply GETs random keys concurrently):

package main

import "github.com/go-redis/redis"
import "math/rand"
import "fmt"

func main() {
        client := redis.NewClusterClient(&redis.ClusterOptions{
                Addrs:    []string{":7000"},
                PoolSize: 16,
        })

        const concurrency = 10

        for i := 0; i < concurrency; i++ {
                go func() {
                        for {
                                rnd := ""
                                for i := 0; i < 10; i++ {
                                        rnd += string(rune(int(rune('a')) + rand.Intn(int(rune('z') - rune('a')))))
                                }
                                fmt.Printf("GET %s\n", rnd)
                                err := client.Get(rnd).Err()
                                if err != nil && err != redis.Nil {
                                        panic(err)
                                }
                        }
                }()
        }

        select{} // sleep
}

If I set const concurrency = 1 the error doesn't happen, so it seems to be a bug with multiple goroutines using the same ClusterClient.

If it's of any relevance: I'm using a Docker Redis Cluster (docker run -p 7000:7000 -p 7001:7001 -p 7002:7002 -p 7003:7003 -p 7004:7004 -p 7005:7005 -p 7006:7006 -p 7007:7007 -i -t -d grokzen/redis-cluster:3.2.11) running on macOS or Linux.

It doesn't happen when running the program on Linux against an Amazon Web Services Elasticache Redis Cluster (engine compatibility 3.2.10), although I think this is because there are no shards (so it never gets a "MOVED" response)

Sorry about the noise if this is a problem with the grokzen redis cluster (however I'm unable to reproduce it with xargs/redis-cli)

Update: the above program also fails when using the create-cluster utility from a redis-3.2.11 tarball

More update: I coerced grokzen redis cluster into running with only one slot, so that redirects don't happen, and that also causes the problem to disappear.

[sqaxomonophonen]

My test program still fails after commit 0929713

[vmihailenco]

I did not try to run your program, because there is nothing special in it and I can't believe that it is broken because of concurrency (who does not use Redis concurrently?). Also it is obvious that you have a different problem so it is better to open new issue.

Are you able to reproduce the problem using test suite? E.g. there are concurrent tests in https://github.com/go-redis/redis/blob/master/race_test.go#L288-L343 and lots of cluster tests https://github.com/go-redis/redis/blob/master/cluster_test.go