Skip to content

Acceptance tests do not stop Besu subprocesses causing resource exhaustion #7596

Open
@fab-10

Description

Description

When running ATs that spawn Besu subprocesses, these subprocesses are not correctly stopped and the linger on, eventually causing resource exhaustion on the system before the tests end.

See this video for an example of what is happening: https://github.com/user-attachments/assets/19ed4ac9-50d8-4fd0-960b-3d510c3cbbb2

you can see that the number of Java Besu processes spawn is always increasing as the tests go, while they should be closed as soon as a test is done.

This is at least present on Linux and Windows, but could depends also on system configuration, because the current way to stop the Besu subprocess is to kill (TERM) the PID related to the start script, but this does not cause the kill (TERM) of the children of that script, including the Java Besu process.
A solution for this problem was implemented in this PR #6820, but it was reverted since it made the ATs take longer on CI, this probably because without the fix, there was already a Besu up and running and able to response to the requests, while with the PR we need to wait for the Besu process to stop and restart, and that adds a delay.

Ideas on how to remove the delay?

  1. Force kill the subprocesses, without waiting for the Besu stop, should not be a problem because these Besus are single use
  2. Refactors ATs to avoid having to stop and start Besu every time, this requires grouping tests that can run with the same configuration, if 2 tests require configurations that are incompatible they cannot be grouped
  3. other ideas?

Acceptance Criteria

Running ATs does not consume all the system resources, because subprocesses are closed when not needed

Steps to Reproduce (Bug)

  1. ./gradlew --max-workers=1 acceptanceTestNoPrivacy

Versions (Add all that apply)

  • Software version: 24.9
  • Java version: Java 17
  • OS Name & Version: Ubuntu 22.04.4 LTS

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingdev experienceThe build system, things that enable easier development etc.techdebtmaintenance, cleanup, refactoring, documentationtesting

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions