Intellij in test summary not printed with `--threads 2`

[lslonina]

👓 What did you see?

After upgrading to cucumber-jvm version 7.24.0 (or any version above 7.23.0), I started seeing "Test framework quit unexpectedly" in the test summary instead of the actual test results. This issue did not occur in version 7.23.0, where everything worked as expected.

Intellij version: 2025.1 with cucumber plugin version: - 251.23774.318: https://plugins.jetbrains.com/plugin/7212-cucumber-for-java/versions/stable/711023

Additional context:
When running tests through Gradle, both versions (7.23.0 and 7.24.0) work correctly.
Junit dependencies in versions 5.13.4, 1.13.4

✅ What did you expect to see?

Correct status of feature file execution in intellij

📦 Which tool/library version are you using?

cucumber-jvm 7.24.0
junit 5.13.4, 1.13.4

🔬 How could we reproduce it?

No response

📚 Any additional context?

No response

[mpkorstanje]

A few questions:

1. Can you reproduce this issue by running the scenario from the https://github.com/cucumber/cucumber-java-skeleton?
2. Is there any other output besides "Test framework quit unexpectedly"?
3. Are you using the JUnit BOM? See: https://docs.junit.org/current/user-guide/#running-tests-build-gradle-bom

edit:

4. Bonus question: Are you using the Ccucumber BOM? https://github.com/cucumber/cucumber-jvm/tree/main/cucumber-bom

[lslonina]

1. The cucumber-java-skeleton project works fine on its own. However, when I copy the sample into my project, I encounter the same issue when running from IntelliJ.
2. Unfortunately, there is no additional output. I checked the IntelliJ logs, but found nothing related to running Cucumber tests.
3. I also tried using BOMs (both JUnit and Cucumber), but that made no difference. Up to version 7.23 everything works fine, and IntelliJ correctly shows test progress and status.

When comparing the dependencies between cucumber-java-skeleton and my project, everything looks the same. I also attempted to debug the test execution, but wasn’t able to attach to the code where the failure occurs.

[mpkorstanje]

Without an error message or reproducer it would be really hard to find the issue.

Can you strip down your project until the problem goes alway? Then add the part that made it crash back in and then strip everything else.

[mpkorstanje]

Another thing you could do is bisect this range of commits. If you can point out the offending commit that would also help.

v7.23.0...v7.24.0

[lslonina]

It looks like the issue is caused by Use a message based TeamCityPlugin #3007. Reverting cucumber-core/src/test/java/io/cucumber/core/backend/StubHookDefinition.java confirmes to address the problem (confirmed across versions up to 2.17.1).

After some debugging, I also found a way to reproduce the break in cucumber-java-skeleton: setting --threads to a value greater than 1 makes the runtime multithreaded, which in turn causes the Runtime class to select the serial event bus.
With --threads 2 (we're using much higher value here, I wasn't considering that this can have impact) - no output is printed to System.out that as I understand is then processed by Intellij

[mpkorstanje]

Oh. Yeah in hindsight that makes complete sense.

The TeamCityPlugin implements EventListener which was introduced before Cucumber supported parallelism. Because there was no parallel execution the existing plugins all assumed events would be published in order. To make these plugins work when parallelism is enabled, events from io.cucumber.plugin are held back and published after the test run in a canonical order.

We also have a new protocol in io.cucumber.messages with with the assumption of parallel execution build in. The TeamCityPlugin is halfway a migration from one to another. Because it is still an EventListener it doesn't get the messages from io.cucumber.messages.

[mpkorstanje]

Fixed in 7.27.2. Cheers!