test: Fix random test failures in test/plugin/test_out_forward.rb #1881
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
fujimotos
force-pushed
the
sf/fix-random-failure-in-out-forward
branch
from
7f55d8d
to
8fdbe31
Compare
I notice that the following test cases (which are supposed to test the handling of broken ack responses) are randomly failing on Travis CI: 1. 'a destination node not supporting responses by disconnection' 2. 'a destination node not supporting responses by just ignoring' The root problem is that, since the destination node gets labelled as "unavailable" after an ack failure, there really shouldn't be any flush attempt after the testing steps (or it will raise a NoNodesAvailable exception). In most environments, this possibility does not pose much problems because the test execution is sufficiently fast (so there is very little space for an additional flush attempt to occur). But since the CI server is known to become very slow from time to time, we need to take special care for it.
fujimotos
force-pushed
the
sf/fix-random-failure-in-out-forward
branch
from
8fdbe31
to
29d2f7e
Compare
|
Thanks! Just merged. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a fix for the test failures occurring on Travis CI every now and then.
For actual cases of this failure, please see job#4143.2, job#4066.5 and job#4167.2.
How the failure happens
The failing test cases are supposed to test the handling of "broken ack responses";
To cut it short, these test cases are designed to perform the following steps:
in_forwardserver (which sends back a broken ack response).out_forwardserver.The problem is that there might be another flush attempt occurring between step 3 and 4.
If one occurs, it will cause an unexpected
NoNodeAvailableexception (since the destinationnode is already labelled as "unavailable" due to the broken ack response in step 3).
In most environments, this possibility does not pose any real problem because the test execution
is modestly fast (so there is very little space for an additional flush attempt to occur). But since
the CI server is known to become very slow from time to time, we need to take special care for it.
Solution
This patch fixes this issue by taking additional care to prevent unintended buffer flushes:
flush_buffer_at_cleanupof the testing driver for the test cases.Also, this should resolve the annoying "unexpected error while after_shutdown" warnings
produced while the test execution.