Docker containers used in tests are not removed

[daniel-westerfeld]

Describe the bug
Docker containers used to run tests in are not cleaned up. To be concrete, in the BUILD file of my test folder I have:

python_tests(
    environment="test_docker_environment",
)

and in the parent folder the environment is defined

docker_environment(
    name="local_test_environment",
    image="image_sha",
)

and in my pants.toml I have added

[environments-preview.names]
test_docker_environment = "//parent/folder:local_test_environment"
local = "//:local-environment"

This all works fine and using pants test :: the test runs successfully in its own docker container.
However, the container remains after the tests and is not removed and remains active.

Pants version
2.16.0

OS
Are you encountering the bug on MacOS, Linux, or both?
Linux

[stuhood]

Containers are cached for reuse within a single pantsd instance, but they should be cleaned up when pantsd restarts cleanly. If pantsd restarts uncleanly (due to kill -9 for example) then they may be orphaned.

Try making a noop edit to pants.toml, and following .pants.d/pants.log: after pantsd has restarted, you should no longer see any running docker containers.

[daniel-westerfeld]

Unfortunately, this did not work for me @stuhood. Here is a little more context. Those are the Logs from the terminal after running pants test:

The test is run in a Docker container with some random name, in this case it was goofy_einstein. This container remains running after the successfull test and changes in the pants.toml have no effect on it.

Also note, that it is not reused. Running the test again, a new container is created and the old one remains active. This leads to a bunch of running containers, one for each time the test is run:

[ahyatt-continua]

+1, I'm also seeing this - containers are not re-used, and they really should be, otherwise you wind up with too many containers.

[francesco-esp]

2.29.0.dev1 which includes #22237 partially solved this issue 👍

Partially because the containers are now reused, but they are never stopped or deleted. I stopped pantsd by sending SIGTERM and SIGINT, but the running containers aren't stopped or deleted. Restarting pants(d) afterwards recreated the containers and leaves the old ones still running.