Skip to content

Conversation

@cperkkk
Copy link
Contributor

@cperkkk cperkkk commented Nov 29, 2025

Ⅰ. Describe what this PR did

This pull request addresses several flaky test, in the apache/incubator-seata project. The test was failing
intermittently due to assumptions about the ordering of the test. I've noticed that when running the test in specific ordering will produce a test failure (maven surefire plugin unfortunately does not guarantee test ordering), such as:

mvn test -Dtest=org.apache.seata.common.rpc.RpcStatusTest#CbeginCount,org.apache.seata.common.rpc.RpcStatusTest#AendCount,org.apache.seata.common.rpc.RpcStatusTest#BremoveStatus

The tests will fail with output:

org.opentest4j.AssertionFailedError: expected: <-1> but was: <0>
	at org.apache.seata.common.rpc.RpcStatusTest.AendCount(RpcStatusTest.java:54)

This is due to the necessity of the count being set by beginCount first before the eventual endCount runs, but the test fails to explicitly mention ordering (such as using import org.junit.jupiter.api.Order; etc), making this a flaky and brittle test depending on the version of surefire plugin used / if there's a method rename.

Ⅱ. Does this pull request fix one issue?

It does not fixes a particular github issue.

Ⅲ. Why don't you add test cases (unit test/integration test)?

I've only changed test and not code under test

Ⅳ. Describe how to verify it

We can permute the run order, after fixing the test, any run order by surefire will pass the test

mvn test -Dsurefire.runOrder=alphabetical -Dtest=org.apache.seata.common.rpc.RpcStatusTest#beginCount,org.apache.seata.common.rpc.RpcStatusTest#removeStatus,org.apache.seata.common.rpc.RpcStatusTest#endCount
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------

Ⅴ. Special notes for reviews

  • The fix ensures the test is robust to non-deterministic behaviors of ordering, making it compatible with future updates to it or related dependencies.

Let me know if further improvements or clarifications are needed!

@cperkkk cperkkk marked this pull request as ready for review November 29, 2025 23:21

@Test
public void endCount() {
// ensure a correct state before testing
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about placing it BeforeEach?

Copy link
Contributor Author

@cperkkk cperkkk Nov 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @slievrly thank you for the reply:
the only problematic function is the endCount and not the other one, and asserts to 0 only works if there's a beginCount and a beginCount that doesn't run before removed service, so endCount evaluates to +1-1==0, if I use beforeEach (for example adding removeStatus on beforeEach), I can do that but I have to change assertions for the endCount to be -1, instead of 0, do you prefer that to be used instead of the current one?

Thank you for your guidance.

@codecov
Copy link

codecov bot commented Dec 9, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 71.03%. Comparing base (8629d18) to head (4eba1d9).

Additional details and impacted files
@@             Coverage Diff              @@
##                2.x    #7819      +/-   ##
============================================
- Coverage     71.10%   71.03%   -0.08%     
  Complexity      797      797              
============================================
  Files          1294     1294              
  Lines         49528    49528              
  Branches       5871     5871              
============================================
- Hits          35216    35181      -35     
- Misses        11403    11431      +28     
- Partials       2909     2916       +7     

see 11 files with indirect coverage changes

Impacted file tree graph

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@lokidundun
Copy link
Contributor

Please apply mvn spotless:apply

@cperkkk
Copy link
Contributor Author

cperkkk commented Dec 12, 2025

Please apply mvn spotless:apply

Okay, spotless:apply applied, thanks

Copy link
Member

@xingfudeshi xingfudeshi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xingfudeshi xingfudeshi changed the title test: bugfix brittle RPC status test test: fix brittle RPC status test Dec 18, 2025
@WangzJi WangzJi added the type: test test case label Dec 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type: test test case

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants