Skip to content

Conversation

@jentfoo
Copy link
Member

@jentfoo jentfoo commented Jun 17, 2019

This adds unit tests which originally failed due to the client remaining in-progress or in queue.
The solution is to add a failure listener which will attempt to remove from queue or in-progress if the client is still active or queued.

@lwahlmeier Do you see any better way to do this, or was there any other mechanism to fix this? I considered trying to make sure we either regularly process over the queue / in-progress looking for failed requests to remove. That may avoid the overhead of the failure listener per-request, but I was unsure on the best way to ensure that it was being regularly inspected.

…d or in-progress

This adds unit tests which originally failed due to the client remaining in-progress or in queue.
The solution is to add a failure listener which will attempt to remove from queue or in-progress if the client is still active or queued.
@jentfoo jentfoo requested a review from lwahlmeier June 17, 2019 18:17
@lwahlmeier
Copy link
Member

@jentfoo I Dont really get the condition here. I mean obviously the test completely stops any queue processing so it will break things. But what I dont get is how you would end up in that situation in the first place. I totally get that the processQueue could be super busy and something would end up in the queue for a while, while other things process, but it should be removed once we get to it in the queue. If the queue/stuck/blocked, or for some reason not processing. at all I dont see how adding a side channel for removing things from it would help at all since nothing will ever process (all calls will time out) if the thread is stuck.

Do you have a stack trace from this happening? My guess would be this is on a SingleThread processor or maybe the NoThreadProcessor and something is keeping it from processing?

jentfoo added 6 commits June 17, 2019 22:03
This closes the connection due to it being in a potentially unknown / unhealthy condition.
In doing this we can make the `finished` code go back to closer to what it used to be.  However the `hasError` now gets the additional cleanup logic so we can defer to that on any failures coming from the timed out future
@jentfoo
Copy link
Member Author

jentfoo commented Jun 18, 2019

Iterated on this together in DM's with @lwahlmeier

We believe that there is a better solution here, however this is probably the simplest solution, and what we want to move forward with for now

@jentfoo jentfoo merged commit 80e4d8a into master Jun 18, 2019
@jentfoo jentfoo deleted the HttpClient-Timeout_deadlock_fix branch June 18, 2019 18:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants