Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Build/CI] Fixing 'docker run' to re-enable AMD CI tests. #4642

Merged
merged 9 commits into from
May 7, 2024
Merged

[Build/CI] Fixing 'docker run' to re-enable AMD CI tests. #4642

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
85aa7cc
Update run-amd-test.sh
Alexei-V-Ivanov-AMD May 7, 2024
2369747
Update run-amd-test.sh
Alexei-V-Ivanov-AMD May 7, 2024
391c684
Update run-amd-test.sh
Alexei-V-Ivanov-AMD May 7, 2024
9817aeb
Update test-template.j2
Alexei-V-Ivanov-AMD May 7, 2024
c589b7c
Update run-amd-test.sh
Alexei-V-Ivanov-AMD May 7, 2024
e76f531
Update test-template.j2
Alexei-V-Ivanov-AMD May 7, 2024
5e46c6e
Update test-template.j2
Alexei-V-Ivanov-AMD May 7, 2024
550a626
Update test-template.j2
Alexei-V-Ivanov-AMD May 7, 2024
81a35ad
Update test-pipeline.yaml
Alexei-V-Ivanov-AMD May 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .buildkite/run-amd-test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,5 +40,5 @@ docker run \
-e HF_TOKEN \
--name ${container_name} \
${container_name} \
/bin/bash -c $(echo $1 | sed "s/^'//" | sed "s/'$//")
/bin/bash -c "${@}"

8 changes: 4 additions & 4 deletions .buildkite/test-pipeline.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ steps:
- pytest -v -s test_pynccl.py

- label: Engine Test
mirror_hardwares: [amd]
#mirror_hardwares: [amd]
command: pytest -v -s engine tokenization test_sequence.py test_config.py test_logger.py

- label: Entrypoints Test
Expand All @@ -73,13 +73,13 @@ steps:
parallelism: 4

- label: Models Test
mirror_hardwares: [amd]
#mirror_hardwares: [amd]
commands:
- bash ../.buildkite/download-images.sh
- pytest -v -s models --ignore=models/test_llava.py --ignore=models/test_mistral.py

- label: Llava Test
mirror_hardwares: [amd]
#mirror_hardwares: [amd]
commands:
- bash ../.buildkite/download-images.sh
- pytest -v -s models/test_llava.py
Expand All @@ -101,7 +101,7 @@ steps:
command: pytest -v -s worker

- label: Speculative decoding tests
mirror_hardwares: [amd]
#mirror_hardwares: [amd]
command: pytest -v -s spec_decode

- label: LoRA Test %N
Expand Down
2 changes: 1 addition & 1 deletion .buildkite/test-template.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ steps:
- label: "AMD: {{ step.label }}"
agents:
queue: amd
command: bash .buildkite/run-amd-test.sh "'cd {{ (step.working_dir or default_working_dir) | safe }} && {{ step.command or (step.commands | join(' && ')) | safe }}'"
command: bash .buildkite/run-amd-test.sh "cd {{ (step.working_dir or default_working_dir) | safe }} ; {{ step.command or (step.commands | join(" ; ")) | safe }}"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might not fail the bash command inside. If the test failed, the whole command will not exit with 1.

Copy link
Contributor Author

@Alexei-V-Ivanov-AMD Alexei-V-Ivanov-AMD May 7, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've checked (https://buildkite.com/vllm/ci/builds/6722)
I does fail on the failing test inside.

Even on the partially failing test, it still fails. See e.g. "AMD: Speculative decoding tests" or "AMD: Models Test" or "AMD: Engine Test" in the above build.

env:
DOCKER_BUILDKIT: "1"
{% endif %}
Expand Down
Loading