PUT and GET calls to redis is taking more time than Expected

[prateeeka-nadimpalli]

Iam trying to put and get data into Redis from my go application. My application is typically running as a pod and redis as another pod for which I am trying to connect and publish data. I have 200 go routines running simultaneously for each poll with frequency 5 min and nearly these routines will do 7000 Redis PUT and GET calls. Its taking more time for processing than expected.

Expected Behavior
For each poll i need to process all these data and publish in atmost 1 min.

Current Behavior
I have enabled logs and and observed that for most of the calls its taking more than 2 minutes. This results in late processing of data. And even if i set any timeout the call is being automatically discarded.

Possible Solution
Please suggest if there need to be any configuration changes from the redis we are using. If its issue from the go redis library itself, check if its possible to increase the asynchronous processing of the internal redis client PUT GET function. It would be better if you can provide any go code snippet on how to connect and do asynchronous calls to redis in an efficient way to avoid these kind of issues.

Steps to Reproduce
Write a standalone where 200 go routines are running
For each go routine post 30 random GET or PUT calls
before put and get calls add loggers with start time, end time and time taken so that you might observe the issue

Context (Environment)
We have a frequency of 5 minutes for posting data. But due to the time taking in this calls, each poll is taking more than 5 minutes, due to which, the previous and current poll is getting collided and its being infinite process. Because of this all the polls data is getting discarded and couldn't publish to cloud. Its effecting our monitoring widely.

Can someone share any possible solutions to avoid this.

[srikar-jilugu]

what is the connection pool configuration that is being used for this client? spawning 200 go-routines might need larger idle pool