[ci] Add grouped tests & mark tests to run by default for fastcheck pipeline

[khluu]

• Add grouped test for Async Engine, Inputs, Utils, Worker tests & Tensorizer, Metrics, Tracings tests
• Mark Regression, Basic Correctness, Core, Distributed (4 GPU), and Entrypoints tests to run on fastcheck by default

Template for fastcheck pipeline is merged here: https://github.com/vllm-project/buildkite-ci/blob/main/scripts/test-template-fastcheck.j2

[DarkLight1337]

Is there a reason why we have to group the tests together rather than marking the individual tests fast_check? Having duplicate definitions in test-pipeline.yaml might lead to copy-and-paste mistakes.

[khluu]

Is there a reason why we have to group the tests together rather than marking the individual tests fast_check?

It's mostly to save on launching instances & image pull overhead. Each job has to pull the Docker image which takes roughly 2-3 mins, while the test itself only runs for 3-4 mins... Grouping tests together like this can save like 10mins or more of billable time for each build

[DarkLight1337]

It's mostly to save on launching instances & image pull overhead. Each job has to pull the Docker image which takes roughly 2-3 mins, while the test itself only runs for 3-4 mins... Grouping tests together like this can save like 10mins or more of billable time for each build

If that's the case, perhaps we should get rid of the individual tests in favour of grouping them together, regardless of fast_check.

[khluu]

It's mostly to save on launching instances & image pull overhead. Each job has to pull the Docker image which takes roughly 2-3 mins, while the test itself only runs for 3-4 mins... Grouping tests together like this can save like 10mins or more of billable time for each build

If that's the case, perhaps we should get rid of the individual tests in favour of grouping them together regardless of fast_check.

Yup I was planning to do that even for full CI run too, but let's start with fastcheck first

[simon-mo]

• What is fast_check_only?
• Can we add docs build as well.

[Jeffwan]

@khluu I am new to buildkite tool. I can not find fast_check in official docs. Is there something I am missing? https://buildkite.com/docs/pipelines/command-step#command-step-attributes