Noticing increase in latency when upgrading to v9.x.x

[AlexMihov96]

Description

I noticed an increase in latency for my service. I was using v8.8.0 go-redis lib version with elasticache engine v6.0.5 and I was getting ~2-3ms (p90).

Now I updated go redis to v9.3.0 and the elasticache engine to v7.0.7 with the same code and getting ~25ms (p90).

In both ways we're serving 100k rps.

Both redis clusters are exactly the same instance type (cache.r6g.xlarge) and 6 shards/24 nodes.
I am using my .Get() inside a Pipelined(), I saw that Pipelines are not thread-safe anymore but I'm not using it concurrently.

Expected Behavior

Using the newest go redis library with v7.0.7 elasticache engine to have the same latency as when I was using v8.8.0 library with the v6.0.5 elasticache.

Steps to Reproduce

1. Have a code like this

func Query() {
  predefinedKeys := []string{"bar", "foo", "test", "x", "y"}

  tStart := time.Now()

  ctx := context.Background()
  _, err := redisClient.Pipelined(ctx, func(pipe redis.Pipeliner) error {
     for _, key := range predefinedKeys {
        pipe.Get(ctx, key)
     }
     return nil
  })
  if err != nil {
     fmt.Println("Error querying data from Redis", err)
     return
  }

  tEnd := time.Now()
  fmt.Println("time elapsed:", tEnd.Sub(tStart))
}

2. Use v6.x.x redis cluster with v8.x.x go redis lib
3. Fetch the elapsed time from the code
4. Now redirect the traffic towards v7.x.x cluster with newest go redis lib
5. (I used go 1.20) doing the testing and all clusters have cluster mode ON
6. Compare the query results

Notes

I did try using library v9.3.0 with the v6 elasticache, resulting in the same ~25ms (p90) latency as when using v9.3.0 library with v7 elasticache.

[rahulkushwaha12]

@AlexMihov96 I am also facing the same, what did you do to solve this? Did you revert to prev version?

[gugu]

We don't use elasticache and have increased latency as well

[AlexMihov96]

@rahulkushwaha12 @gugu

Initially i reverted to the old go-redis version but later on i updated it back to latest + using elasticache v7.x.x
And the issue was still there but I ended up reworking the code, introducing caching and somehow managed to achieve the desired p90. But the original issue still persist.

[ndyakov]

@AlexMihov96 would you like to work on investigating what is causing the increase in latency? Maybe using a vendored version of the code and adding some traces inside the library?