Integration test containers not being removed after test failure.

[neuronull]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Steps to reproduce:

• first run of make test-integration-aws fails the tests scrapes_metrics_v*
• observe test containers are still running
• fix the cause of the scrapes_metrics_v* tests failing (fix(aws_ecs_metrics source): integration test failures due to 'null' type or missing fields #13016)
• run make test-integration-aws again, observe now an ES timeout failure , presumably because of the stale containers.
• manually remove the stale integration containers, ES timeout failure goes away.

The focus of this issue is why the integration containers were not removed upon the test failure.

Configuration

No response

Version

0.22.0

Debug Output

No response

Example Data

$ docker container ls
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

// run the test

$ make test-integration-aws
RUST_VERSION=1.58.1 docker-compose -f scripts/integration/docker-compose.aws.yml build
mock-ec2-metadata uses an image, skipping
mock-localstack uses an image, skipping
mock-watchlogs uses an image, skipping
mock-ecs uses an image, skipping
Building runner
Sending build context to Docker daemon  18.43MB
Step 1/4 : ARG RUST_VERSION
Step 2/4 : FROM docker.io/rust:${RUST_VERSION}-slim-bullseye
 ---> c3d2ecb28032
Step 3/4 : RUN apt-get update && apt-get install -y --no-install-recommends     build-essential     cmake     curl     g++     libclang1-9     libsasl2-dev     libssl-dev     llvm-9     pkg-config     zlib1g-dev   && rm -rf /var/lib/apt/lists/*
 ---> Using cache
 ---> 3abf89d672ea
Step 4/4 : RUN curl -LsSf https://get.nexte.st/0.9/linux | tar zxf - -C /usr/local/bin
 ---> Using cache
 ---> c8be481d0988
Successfully built c8be481d0988
Successfully tagged integration_runner:latest
RUST_VERSION=1.58.1 docker-compose -f scripts/integration/docker-compose.aws.yml run --rm runner
Creating integration_mock-localstack_1   ... done
Creating integration_mock-ecs_1          ... done
Creating integration_mock-watchlogs_1    ... done
Creating integration_mock-ec2-metadata_1 ... done
Creating integration_runner_run          ... done
    Finished test [unoptimized + debuginfo] target(s) in 0.40s
    Starting 65 tests across 1 binaries (264 skipped)

...

--- TRY 4 STDERR:        vector sources::aws_ecs_metrics::integration_tests::scrapes_metrics_v4 ---
thread 'sources::aws_ecs_metrics::integration_tests::scrapes_metrics_v4' panicked at 'Failed to assert compliance, errors:
  - Missing emitted event `EventsReceived`
  - Missing emitted event `EventsSent`
  - Missing metric `component_received_events_total`
  - Missing metric `component_received_event_bytes_total`
  - Missing metric `component_sent_events_total`
  - Missing metric `component_sent_event_bytes_total`
', src/test_util/components.rs:252:18
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

error: test run failed
ERROR: 100
make: *** [Makefile:361: test-integration-aws] Error 100

// containers are still present

 $ docker container ls
CONTAINER ID   IMAGE                                                COMMAND                  CREATED              STATUS              PORTS                     NAMES
acb9706e6a59   timberiodev/mock-ec2-metadata:latest                 "./mock-ec2-metadata"    About a minute ago   Up About a minute                             integration_mock-ec2-metadata_1
e6a5f1ff8038   localstack/localstack-full:0.11.6                    "docker-entrypoint.sh"   About a minute ago   Up About a minute   4566-4597/tcp, 8080/tcp   integration_mock-localstack_1
bc0aefc5eaf1   luciofranco/mockwatchlogs:latest                     "/mockwatchlogs/mock…"   About a minute ago   Up About a minute                             integration_mock-watchlogs_1
55c6e5973eaa   amazon/amazon-ecs-local-container-endpoints:latest   "/local-container-en…"   About a minute ago   Up About a minute   80/tcp                    integration_mock-ecs_1

Additional Context

$ uname -a
Linux ip-172-31-59-47 5.15.0-1008-aws #10-Ubuntu SMP Wed May 18 17:28:39 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

References

No response