Implement Parallel Method Execution in JUnit-Vintage engine

[YongGoose]

Resolves: #4238

Overview

Implements parallel execution of test methods in the junit-vintage-engine, inspired by the parallelization support available in JUnit 4 through ParallelComputer.

• This feature allows for enhanced test execution performance by leveraging modern multi-threaded environments.

Changes

1. Implemented Custom RunnerScheduler

• Created a RunnerScheduler using ExecutorService.
• Ensures proper scheduling and execution of test methods in parallel.

2. Updated Existing Tests

• Adjusted tests to verify the parallel execution of methods.
• Added new test cases to ensure backward compatibility and correctness.

3. Documentation Updates

• Added documentation on how to enable and configure parallel method execution.
• Explained performance implications and scenarios where parallelization is beneficial.

---

I hereby agree to the terms of the JUnit Contributor License Agreement.

---

Definition of Done

• There are no TODOs left in the code
• Method preconditions are checked and documented in the method's Javadoc
• Coding conventions (e.g. for logging) have been followed
• Change is covered by automated tests including corner cases, errors, and exception handling
• Public API has Javadoc and @API annotations
• Change is documented in the User Guide and Release Notes

[YongGoose]

@marcphilipp

I created this Draft PR to ask a few questions.
First, I’ve added two independent configuration parameters as you suggested.

However, I have a question regarding their behavior:

• What happens if the user sets the junit.vintage.execution.parallel.enabled parameter to true, but both junit.vintage.execution.parallel.classes and junit.vintage.execution.parallel.methods are set to false?

• Additionally, what occurs if junit.vintage.execution.parallel.enabled is set to false, while junit.vintage.execution.parallel.classes or junit.vintage.execution.parallel.methods set to true?

[marcphilipp]

What happens if the user sets the junit.vintage.execution.parallel.enabled parameter to true, but both junit.vintage.execution.parallel.classes and junit.vintage.execution.parallel.methods are set to false?

I think we should check for that, log a warning (like we do for invalid configuration parameters in Jupiter), and continue as if junit.vintage.execution.parallel.enabled was false.

Additionally, what occurs if junit.vintage.execution.parallel.enabled is set to false, while junit.vintage.execution.parallel.classes or junit.vintage.execution.parallel.methods set to true?

I think turning off parallel execution (using the ...enabled=false parameter) while keeping the rest of the parallel configuration in place is a valid use case. Therefore, I think we should ignore the ...classes and ...methods parameters in this case without logging a warning.

[YongGoose]

What happens if the user sets the junit.vintage.execution.parallel.enabled parameter to true, but both junit.vintage.execution.parallel.classes and junit.vintage.execution.parallel.methods are set to false?

I think we should check for that, log a warning (like we do for invalid configuration parameters in Jupiter), and continue as if junit.vintage.execution.parallel.enabled was false.

Additionally, what occurs if junit.vintage.execution.parallel.enabled is set to false, while junit.vintage.execution.parallel.classes or junit.vintage.execution.parallel.methods set to true?

I think turning off parallel execution (using the ...enabled=false parameter) while keeping the rest of the parallel configuration in place is a valid use case. Therefore, I think we should ignore the ...classes and ...methods parameters in this case without logging a warning.

Done in be5234f !
If you have a better approach, please let me know in the comments 🙂

[YongGoose]

@marcphilipp

I implemented this to get feedback on parallelization at the class and method levels. 0b70615

The code is intended to facilitate understanding, so it hasn't been fully polished yet❕

There are three scenarios when running parallel tests:

1. Running methods in parallel.
2. Running classes in parallel.
3. Running both methods and classes in parallel.

Currently facing challenges with designing for these scenarios.

I want to design it similarly to JUnit 4's ParallelComputer, but I need help..🥹

• https://github.com/junit-team/junit4/blob/df00d5eced3a7737b88de0f6f9e3673f0cf88f88/src/main/java/org/junit/experimental/ParallelComputer.java#L54-L66

[YongGoose]

@marcphilipp

While writing test code, I discovered an issue caused by using the same ExecutorService.

• https://github.com/junit-team/junit5/actions/runs/12841358007/job/35811074310?pr=4242

I concluded that the test did not complete due to a deadlock.

Assumption: Lets assume the thread pool size of the ExecutorService is 2.

Scenario:

1. Two tests, A and B, are initiated in parallel from an outer loop.
2. Test A internally attempts to execute two subtasks (A1, A2).
3. Similarly, Test B internally attempts to execute two subtasks (B1, B2).
4. A starts A1, and B starts B1. At this point, all threads in the pool are occupied.
5. A waits for A2 to complete, and B waits for B2 to complete.
6. However, A2 and B2 cannot start because there are no available threads in the pool.
7. As a result, both A and B enter a deadlock, waiting indefinitely.

To prevent this issue, we could either set a timeout on allOf.get() (as a way to resolve the deadlock even if the test doesn't execute successfully) or separate the ExecutorService instances.

WDYT?

---

I’m also curious whether this issue will be included in version 5.12M1 or 5.13.

[YongGoose]

While writing test code, I discovered an issue caused by using the same ExecutorService.

• https://github.com/junit-team/junit5/actions/runs/12841358007/job/35811074310?pr=4242

I concluded that the test did not complete due to a deadlock.

Assumption: Lets assume the thread pool size of the ExecutorService is 2.

Scenario:

1. Two tests, A and B, are initiated in parallel from an outer loop.
2. Test A internally attempts to execute two subtasks (A1, A2).
3. Similarly, Test B internally attempts to execute two subtasks (B1, B2).
4. A starts A1, and B starts B1. At this point, all threads in the pool are occupied.
5. A waits for A2 to complete, and B waits for B2 to complete.
6. However, A2 and B2 cannot start because there are no available threads in the pool.
7. As a result, both A and B enter a deadlock, waiting indefinitely.

To prevent this issue, we could either set a timeout on allOf.get() (as a way to resolve the deadlock even if the test doesn't execute successfully) or separate the ExecutorService instances.

WDYT?

I’m also curious whether this issue will be included in version 5.12M1 or 5.13.

@marcphilipp

I am eager to resolve this issue😀
If you have time, may I kindly ask for your response?

[marcphilipp]

@YongGoose Sorry for the slow response! I was traveling for the last few days.

That's a good catch! For Jupiter, we use a ForkJoinPool to avoid this problem. Are you familiar with that?

[YongGoose]

@YongGoose Sorry for the slow response! I was traveling for the last few days.

Apologies if I came across as rude.
I didn’t know you were traveling. 😅
Hope you had a great trip!

That's a good catch! For Jupiter, we use a ForkJoinPool to avoid this problem. Are you familiar with that?

I didn’t know that!
I’ll look into the part where ForkJoinPool is used in Jupiter.

[marcphilipp]

Apologies if I came across as rude.
I didn’t know you were traveling. 😅

No worries!

I’ll look into the part where ForkJoinPool is used in Jupiter.

Jupiter uses ForkJoinPoolHierarchicalTestExecutorService from junit-platform-engine.

[YongGoose]

Jupiter uses ForkJoinPoolHierarchicalTestExecutorService from junit-platform-engine.

I’m running into some challenges trying to use ForkJoinPoolHierarchicalTestExecutorService in Vintage.

Before moving forward, I’d like to get your feedback on the best approach here!

1. Update VintageTestEngine to use ForkJoinPoolHierarchicalTestExecutorService.
2. Create a custom ForkJoinPool and use that as the thread pool instead.

Which direction do you think makes more sense?

[marcphilipp]

I think option 1 won't work because Vintage can't implement HierarchicalTestEngine so I'd go with option 2.

[YongGoose]

I think option 1 won't work because Vintage can't implement HierarchicalTestEngine so I'd go with option 2.

Thank you!

I’ll proceed as you suggested.

As for another PR I’m working on, I accidentally included commits from the main branch while resolving git conflicts. So, I’m planning to create a new PR 😅

[YongGoose]

It's quite a complex mechanism (implementation-wise). Moreover, it requires declaring exclusive resources "statically" (as part of the TestDescriptor tree that is returned from TestEngine.discover). The last bit is problematic for the Vintage engine, as the TestDescriptor tree depends on Description objects provided by the runner specified using @RunWith. Since we can't extend that API we'd have to find some other means to determine which class/method a Descriptor represents. We do this to determine the TestSource for each TestDescriptor but it's not as reliable as we'd need it to implement resource-based locking. Therefore, I think we should refrain from implementing that for the Vintage engine, at least at this time.

Thank you for the detailed explanation.

I wasn't very familiar with the internal structure of the Vintage engine, so it was difficult to make a judgment, but your explanation helped me understand it better.

As you advised, I will not implement this feature in the Vintage engine for now.
If I determine that implementation is necessary in the future, I will address it through a separate issue.

[YongGoose]

@marcphilipp

I have completed the documentation and changed the PR status from Draft to Ready for Review.
If there's anything I haven't done yet or any additional work needed, please feel free to let me know.

PS. I don't think I could have completed this work without your help.
I sincerely appreciate it.

[YongGoose]

@marcphilipp @sbrannen

Both of you, thank you! 🙂
Thanks to you, the document editing process went smoothly.

[marcphilipp]

@YongGoose Once again, thanks for all your work! 👍