[Build] Add initial conditional testing spec #6841

simon-mo · 2024-07-26T17:50:26Z

Very initial stab at conditional testing. Few remaining tests

Work with @khluu to get this enabled
Iron-out some bugs: if the test file itself is changed, some tests should rerun?
Getting the dependencies more accuracy and precise.

github-actions · 2024-07-26T17:50:39Z

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which consists a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of default ones by unblocking the steps in your fast-check build on Buildkite UI.

Once the PR is approved and ready to go, please make sure to run full CI as it is required to merge (or just use auto-merge).

To run full CI, you can do one of these:

Comment /ready on the PR
Add ready label to the PR
Enable auto-merge.

🚀

simon-mo · 2024-07-26T17:51:23Z

.buildkite/test-pipeline.yaml

  command: pytest -v -s test_regression.py
  working_dir: "/vllm-workspace/tests" # optional

- label: AsyncEngine Test


removed because it is covered in Async Engine, Inputs, Utils, Worker Test

simon-mo · 2024-07-26T17:51:41Z

.buildkite/test-pipeline.yaml

- label: Tracing Test
-  commands: 
-    - "pip install \
-        opentelemetry-sdk \
-        opentelemetry-api \
-        opentelemetry-exporter-otlp \
-        opentelemetry-semantic-conventions-ai"
-    - pytest -v -s tracing


covered in fastcheck that combiend metrics and tracing

simon-mo · 2024-07-26T17:52:01Z

.buildkite/test-pipeline.yaml

  commands:
    - apt-get install -y curl libsodium23
    - export VLLM_WORKER_MULTIPROC_METHOD=spawn
    - pytest -v -s tensorizer_loader

- label: Metrics Test


covered in fastcheck that combiend metrics and tracing

simon-mo · 2024-08-01T18:12:29Z

.buildkite/test-pipeline.yaml

+# Documentation
+# label(str): the name of the test. emoji allowed.
+# fast_check(bool): whether to run this on each commit by default, without the /ready tag. 
+# fast_check_only(bool): whether to skip this test on full suite.
+# command(str): the single command to run for tests. incompatible with commands.
+# commands(list): the list of commands to run for test. incompatbile with command.
+# mirror_hardwares(list): the list of hardwares to run the test on as well. currently only supports [amd]
+# gpu(str): override the GPU selection for the test. default is on L4 GPUs. currently only supports a100
+# num_gpus(int): override the number of GPUs for the test. default to 1 GPU. currently support 2,4.
+# num_nodes(int): whether to simulate multi-node setup by launch multiple containers on one host, 
+#     in this case, commands must be specified. the first command runs on first host, the second
+#     command runs on the second host.
+# working_dir(str): override the place where command execute. default to "/vllm-workspace/tests".
+# source_file_dependencies(list): the list of prefix to opt-in the test for, if empty, the test will always run.


@khluu can you review some of the docs here? I'm afraid I might have mis-interpret some fields.

khluu · 2024-08-01T18:20:36Z

.buildkite/test-pipeline.yaml

@@ -5,11 +5,27 @@
 # https://github.com/vllm-project/buildkite-ci/blob/main/scripts/test-template-aws.j2 
 # to generate the final pipeline yaml file.

+# Documentation
+# label(str): the name of the test. emoji allowed.
+# fast_check(bool): whether to run this on each commit by default, without the /ready tag. 


whether to run this on each commit on fastcheck pipeline

khluu · 2024-08-01T18:23:28Z

.buildkite/test-pipeline.yaml

+# num_nodes(int): whether to simulate multi-node setup by launch multiple containers on one host, 
+#     in this case, commands must be specified. the first command runs on first host, the second
+#     command runs on the second host.
+# working_dir(str): override the place where command execute. default to "/vllm-workspace/tests".


Specify the place where command should execute, default to /vllm-workspace/tests

khluu · 2024-08-01T18:24:02Z

.buildkite/test-pipeline.yaml

+# Documentation
+# label(str): the name of the test. emoji allowed.
+# fast_check(bool): whether to run this on each commit by default, without the /ready tag. 
+# fast_check_only(bool): whether to skip this test on full suite.


Run this test on fastcheck pipeline only

simon-mo · 2024-08-01T18:46:05Z

.buildkite/test-pipeline.yaml

  commands:
  - pytest -v -s async_engine # Async Engine
  - pytest -v -s test_inputs.py
  - pytest -v -s multimodal
  - pytest -v -s test_utils.py # Utils
  - pytest -v -s worker # Worker

- label: Metrics, Tracing Test


moved to later section

khluu · 2024-08-04T22:52:38Z

.buildkite/test-pipeline.yaml

  fast_check: true
-  fast_check_only: true
+  no_gpu: True
+  source_file_dependencies: [] # always run


if test always run, maybe this key should be here at all

khluu · 2024-08-04T23:16:04Z

.buildkite/test-pipeline.yaml

  mirror_hardwares: [amd]
  fast_check: true
+  source_file_dependencies:


Besides the source file dependencies, I think we should add the tests themselves as dependency too. Like if tests/core/ changes, this test should also run. Maybe we can rename source_file_dependencies to dependencies to be inclusive?

Signed-off-by: Alvant <alvasian@yandex.ru>

add initial spec

1a00ade

simon-mo marked this pull request as draft July 26, 2024 17:50

simon-mo commented Jul 26, 2024

View reviewed changes

rename

6dca813

simon-mo commented Aug 1, 2024

View reviewed changes

khluu reviewed Aug 1, 2024

View reviewed changes

simon-mo added 3 commits August 1, 2024 11:41

refactor yaml

3448af7

Merge branch 'main' of github.com:vllm-project/vllm into test-deps

ccb7727

fix flashinfer

6aa346d

simon-mo commented Aug 1, 2024

View reviewed changes

Merge branch 'main' into test-deps

c2b2a22

simon-mo marked this pull request as ready for review August 4, 2024 06:17

simon-mo added the ready ONLY add when PR is ready to merge/full CI is needed label Aug 4, 2024

khluu approved these changes Aug 4, 2024

View reviewed changes

khluu reviewed Aug 4, 2024

View reviewed changes

khluu added 2 commits August 4, 2024 19:35

don't specify dependencies if always run

7f0f10d

Add test itself to be dependency

d15d918

simon-mo merged commit e3c664b into vllm-project:main Aug 6, 2024
36 of 37 checks passed

sfc-gh-mkeralapura pushed a commit to sfc-gh-mkeralapura/vllm that referenced this pull request Aug 12, 2024

[Build] Add initial conditional testing spec (vllm-project#6841)

abaea82

kylesayrs pushed a commit to neuralmagic/vllm that referenced this pull request Aug 17, 2024

[Build] Add initial conditional testing spec (vllm-project#6841)

502ce5f

fialhocoelho pushed a commit to opendatahub-io/vllm that referenced this pull request Aug 22, 2024

[Build] Add initial conditional testing spec (vllm-project#6841)

163f3da

Alvant pushed a commit to compressa-ai/vllm that referenced this pull request Oct 26, 2024

[Build] Add initial conditional testing spec (vllm-project#6841)

3368e1c

Signed-off-by: Alvant <alvasian@yandex.ru>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Build] Add initial conditional testing spec #6841

[Build] Add initial conditional testing spec #6841

simon-mo commented Jul 26, 2024

github-actions bot commented Jul 26, 2024

simon-mo Jul 26, 2024

simon-mo Jul 26, 2024

simon-mo Jul 26, 2024

simon-mo Aug 1, 2024

khluu Aug 1, 2024

khluu Aug 1, 2024

khluu Aug 1, 2024

khluu Aug 1, 2024

simon-mo Aug 1, 2024

khluu Aug 4, 2024

khluu Aug 4, 2024

[Build] Add initial conditional testing spec #6841

[Build] Add initial conditional testing spec #6841

Conversation

simon-mo commented Jul 26, 2024

github-actions bot commented Jul 26, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment