fix consistent high timeout err rate caused by prepareStatement #892
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
A more detailed description of the problem can be found in this post. |
Zariel
reviewed
Apr 18, 2017
conn.go
Outdated
| @@ -713,6 +713,9 @@ func (c *Conn) prepareStatement(ctx context.Context, stmt string, tracer Tracer) | |||
| if err != nil { | |||
| flight.err = err | |||
| flight.wg.Done() | |||
| if err.Error() == "context deadline exceeded" { | |||
There was a problem hiding this comment.
instead of doing this we should compare the value of context.ErrTimeout, but I think in this case we can just always remove the value on errors
|
@Zariel Fixed |
|
Great thanks! |
|
lol, this has been plaguing us for months and I just spent the last day digging into this same bug. I was literally writing up an issue report to describe my findings and see what people thought a good fix would be... and I went to grab some links from the latest commit to the code which causes the problem, and noticed that it just got fixed. Thanks! :D |
zhixinwen
added a commit
to zhixinwen/gocql
that referenced
this pull request
Apr 19, 2017
…he#892) * fix timeout bug * remove cache no matter which err returns
mincai
pushed a commit
to uber/peloton
that referenced
this pull request
Jan 7, 2019
Summary: Upgrading to a version which includes the fix for apache/cassandra-gocql-driver#892 which we have been hitting quite often in production. Resolves T1359093 Reviewers: min, #peloton Reviewed By: min, #peloton Subscribers: jenkins Maniphest Tasks: T1359093 Differential Revision: https://code.uberinternal.com/D1337813
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently gocql will cache the timeout err caused by network instability when prepare a statement.
As a result, if gocql fails to prepare a statement due to network issue at the first time on a connection, the rest of the call to that statement on that connection would return "context deadline exceeded”.
It can result in high and consistent err rate and can only fixed by restart a host.
We have been suffering from this in production for a while.