Container logs are not uploaded when dependencies fail to start

[software-opal]

When we are using this plugin to run our integration tests, we are not seeing the artefacts uploaded on failure, nor when we specify upload-container-logs: always. I think this is related to this issue #247. This issue appears on v3.7.0 and v3.9.0.

Running the latest commit(a281a19) using upload-container-logs: always yields interesting results(I've attached the pipeline.yml and docker-compose.yml file I used to gather these results):

• When all dependent services start up(I.E. the docker-compose up --scale=<target>=0 <target> command succeeds) the logs will be uploaded.
  • Logs are uploaded for a service that remains running throughout the test
    (successful-service-test in the reproduction files stays running)
  • Logs are also uploaded for a service that crashes during the test as long as it crashes after the 'start dependent services' section finishes.
    (delayed-failure-service-test in the reproduction files demonstrates this by exiting with a status code of 1 after 5 seconds)
• When any dependent service fails to start up, or fails to become healthy during the startup, logs will NOT be uploaded.
  • This affects a service that immediately fail
    (failure-service-test in the reproduction files exits with a code of 1).
  • This also affects a service with a health check that does not succeed within the required startup time
    (unhealthy-service-test in the reproduction files has a health check that always fails to simulate this)

The success or failure of the step's commands does not appear to effect the plugin's log upload.

Expand for reproduction pipeline & docker-compose file

With this pipeline there should be 6 failed steps and 2 passing tests, but all steps should upload logs for the dependent services(Postgres, and the test-specific service).

pipeline.yml

steps:
  - label: "successful-service-test"
    plugins:
     - docker-compose#a281a190a1cf93c36a6f0dbba9e6c946d84cf1d8:"
    command:
      - sleep 10
      - exit 0
    plugins:
      - docker-compose#a281a190a1cf93c36a6f0dbba9e6c946d84cf1d8:
          upload-container-logs: always
          config: .buildkite/docker-compose.issue-327.yml
          run: successful-service-test

  - label: "failure-service-test"
    plugins:
     - docker-compose#a281a190a1cf93c36a6f0dbba9e6c946d84cf1d8:"
    command:
      - sleep 10
      - exit 0
    plugins:
      - docker-compose#a281a190a1cf93c36a6f0dbba9e6c946d84cf1d8:
          upload-container-logs: always
          config: .buildkite/docker-compose.issue-327.yml
          run: failure-service-test

  - label: "delayed-failure-service-test"
    plugins:
     - docker-compose#a281a190a1cf93c36a6f0dbba9e6c946d84cf1d8:"
    command:
      - sleep 10
      - exit 0
    plugins:
      - docker-compose#a281a190a1cf93c36a6f0dbba9e6c946d84cf1d8:
          upload-container-logs: always
          config: .buildkite/docker-compose.issue-327.yml
          run: delayed-failure-service-test

  - label: "unhealthy-service-test"
    plugins:
     - docker-compose#a281a190a1cf93c36a6f0dbba9e6c946d84cf1d8:"
    command:
      - sleep 10
      - exit 0
    plugins:
      - docker-compose#a281a190a1cf93c36a6f0dbba9e6c946d84cf1d8:
          upload-container-logs: always
          config: .buildkite/docker-compose.issue-327.yml
          run: unhealthy-service-test


  - label: "successful-service-test - Fails"
    plugins:
     - docker-compose#a281a190a1cf93c36a6f0dbba9e6c946d84cf1d8:"
    command:
      - sleep 10
      - exit 1
    plugins:
      - docker-compose#a281a190a1cf93c36a6f0dbba9e6c946d84cf1d8:
          upload-container-logs: always
          config: .buildkite/docker-compose.issue-327.yml
          run: successful-service-test

  - label: "failure-service-test - Fails"
    plugins:
     - docker-compose#a281a190a1cf93c36a6f0dbba9e6c946d84cf1d8:"
    command:
      - sleep 10
      - exit 1
    plugins:
      - docker-compose#a281a190a1cf93c36a6f0dbba9e6c946d84cf1d8:
          upload-container-logs: always
          config: .buildkite/docker-compose.issue-327.yml
          run: failure-service-test

  - label: "delayed-failure-service-test - Fails"
    plugins:
     - docker-compose#a281a190a1cf93c36a6f0dbba9e6c946d84cf1d8:"
    command:
      - sleep 10
      - exit 1
    plugins:
      - docker-compose#a281a190a1cf93c36a6f0dbba9e6c946d84cf1d8:
          upload-container-logs: always
          config: .buildkite/docker-compose.issue-327.yml
          run: delayed-failure-service-test

  - label: "unhealthy-service-test - Fails"
    plugins:
     - docker-compose#a281a190a1cf93c36a6f0dbba9e6c946d84cf1d8:"
    command:
      - sleep 10
      - exit 1
    plugins:
      - docker-compose#a281a190a1cf93c36a6f0dbba9e6c946d84cf1d8:
          upload-container-logs: always
          config: .buildkite/docker-compose.issue-327.yml
          run: unhealthy-service-test

.buildkite/docker-compose.issue-327.yml

version: '3'
services:

  postgres:
    image: postgres:12.5
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
    healthcheck:
      # This will check if Postgres is ready every 2 seconds, waiting up to 60
      #  seconds for the container to fully start.
      test: [ "CMD", "pg_isready" ]
      timeout: 5s
      interval: 2s
      start_period: 60s
      retries: 5

  successful:
    image: "buildpack-deps:stable-curl"
    command:
      [
        'bash',
        '-c',
        'echo "I should appear in logs"; sleep 999999999'
      ]
    depends_on:
      postgres:
        condition: service_healthy
    healthcheck:
      test: [ "CMD", "true" ]
      timeout: 1s
      interval: 1s
      start_period: 10s
      retries: 5

  failure:
    image: "buildpack-deps:stable-curl"
    command: [ 'bash', '-c', 'echo "I should appear in logs"; exit 1' ]
    depends_on:
      postgres:
        condition: service_healthy
    healthcheck:
      test: [ "CMD", "true" ]
      timeout: 1s
      interval: 1s
      start_period: 10s
      retries: 5

  delayed-failure:
    image: "buildpack-deps:stable-curl"
    command:
      [
        'bash',
        '-c',
        'echo "I should appear in logs"; sleep 5; exit 1'
      ]
    depends_on:
      postgres:
        condition: service_healthy
    healthcheck:
      test: [ "CMD", "true" ]
      timeout: 1s
      interval: 1s
      start_period: 10s
      retries: 5

  unhealthy:
    image: "buildpack-deps:stable-curl"
    command:
      [
        'bash',
        '-c',
        'echo "I should appear in logs"; sleep 999999999'
      ]
    depends_on:
      postgres:
        condition: service_healthy
    healthcheck:
      test: [ "CMD", "false" ]
      timeout: 1s
      interval: 1s
      start_period: 10s
      retries: 5

  successful-service-test:
    image: "buildpack-deps:stable-curl"
    depends_on:
      successful:
        condition: service_healthy

  failure-service-test:
    image: "buildpack-deps:stable-curl"
    depends_on:
      failure:
        condition: service_healthy

  delayed-failure-service-test:
    image: "buildpack-deps:stable-curl"
    depends_on:
      delayed-failure:
        condition: service_healthy

  unhealthy-service-test:
    image: "buildpack-deps:stable-curl"
    depends_on:
      unhealthy:
        condition: service_healthy