[Build/CI] Fixing 'docker run' to re-enable AMD CI tests.

[Alexei-V-Ivanov-AMD]

This PR achieves the following goals:

1. Corrects docker run interface to launch containers properly;

2. Trims the number of AMD tests.

[simon-mo]

This might not fail the bash command inside. If the test failed, the whole command will not exit with 1.

[Alexei-V-Ivanov-AMD]

I've checked (https://buildkite.com/vllm/ci/builds/6722)
I does fail on the failing test inside.

Even on the partially failing test, it still fails. See e.g. "AMD: Speculative decoding tests" or "AMD: Models Test" or "AMD: Engine Test" in the above build.

[comaniac]

Looks like the AMD CI is broken after this PR? I saw the same error message in many CI runs for AMD tests:

Unable to open /dev/kfd read-write: Invalid argument
--
  | Failed to get user name to check for render group membership
  | 🚨 Error: The command exited with status 1

[Alexei-V-Ivanov-AMD]

Looks like the AMD CI is broken after this PR? I saw the same error message in many CI runs for AMD tests:

No, this error is proven to be un-related to the present PR. We have definitely seen this issue before this PR.
The best current remedy is to request a "re-run" of the failed test. All CI nodes are at times affected, regardless of their HW or SW particularities. The error occurs in "bursts": if you see it coming, wait ~ 1 min and request a re-run.

[comaniac]

I see. Also I'm working on #4535 that changes AMD kernels a bit, but I keep seeing the compilation errors which I didn't see in the NVIDIA build. So I tried to find an existing success build for reference. If you have any idea about that error (https://buildkite.com/vllm/ci/builds/6803#018f550a-e708-4a0a-a48c-97a5a4d85a40/1106-1856) please let me know.

[Alexei-V-Ivanov-AMD]

I see. Also I'm working on #4535 that changes AMD kernels a bit, but I keep seeing the compilation errors which I didn't see in the NVIDIA build. So I tried to find an existing success build for reference. If you have any idea about that error (https://buildkite.com/vllm/ci/builds/6803#018f550a-e708-4a0a-a48c-97a5a4d85a40/1106-1856) please let me know.

The error you're referring to appears to be a cmake error during the container build. It is apparently persistent through multiple attempts across different AMD tests in the referred build. It is definitely not related to the PR #4642, though, as rocm containers were getting built before it.

Cmake must have complained about something at some point above the final error message. To isolate and analyze the cause of the error during this CI build you'll need to make a fresh clone of the repo and then build a standard rocm docker container:

...vllm$ docker build -t {container name} -f Dockerfile.rocm .

That how it gets built in the CI anyway (

vllm/.buildkite/run-amd-test.sh

Line 20 in 8344f77

echo "--- Building container"

)

The stdout dump will give you plenty of information about your issue.