-
Notifications
You must be signed in to change notification settings - Fork 8.9k
test: fix brittle RPC status test #7819
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: 2.x
Are you sure you want to change the base?
Conversation
|
|
||
| @Test | ||
| public void endCount() { | ||
| // ensure a correct state before testing |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How about placing it BeforeEach?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @slievrly thank you for the reply:
the only problematic function is the endCount and not the other one, and asserts to 0 only works if there's a beginCount and a beginCount that doesn't run before removed service, so endCount evaluates to +1-1==0, if I use beforeEach (for example adding removeStatus on beforeEach), I can do that but I have to change assertions for the endCount to be -1, instead of 0, do you prefer that to be used instead of the current one?
Thank you for your guidance.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## 2.x #7819 +/- ##
============================================
- Coverage 71.10% 71.03% -0.08%
Complexity 797 797
============================================
Files 1294 1294
Lines 49528 49528
Branches 5871 5871
============================================
- Hits 35216 35181 -35
- Misses 11403 11431 +28
- Partials 2909 2916 +7 🚀 New features to boost your workflow:
|
|
Please apply |
Okay, spotless:apply applied, thanks |
xingfudeshi
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Ⅰ. Describe what this PR did
This pull request addresses several flaky test, in the
apache/incubator-seataproject. The test was failingintermittently due to assumptions about the ordering of the test. I've noticed that when running the test in specific ordering will produce a test failure (maven surefire plugin unfortunately does not guarantee test ordering), such as:
mvn test -Dtest=org.apache.seata.common.rpc.RpcStatusTest#CbeginCount,org.apache.seata.common.rpc.RpcStatusTest#AendCount,org.apache.seata.common.rpc.RpcStatusTest#BremoveStatusThe tests will fail with output:
This is due to the necessity of the count being set by beginCount first before the eventual endCount runs, but the test fails to explicitly mention ordering (such as using import org.junit.jupiter.api.Order; etc), making this a flaky and brittle test depending on the version of surefire plugin used / if there's a method rename.
Ⅱ. Does this pull request fix one issue?
It does not fixes a particular github issue.
Ⅲ. Why don't you add test cases (unit test/integration test)?
I've only changed test and not code under test
Ⅳ. Describe how to verify it
We can permute the run order, after fixing the test, any run order by surefire will pass the test
mvn test -Dsurefire.runOrder=alphabetical -Dtest=org.apache.seata.common.rpc.RpcStatusTest#beginCount,org.apache.seata.common.rpc.RpcStatusTest#removeStatus,org.apache.seata.common.rpc.RpcStatusTest#endCountⅤ. Special notes for reviews
Let me know if further improvements or clarifications are needed!