[Misc]: call for help to fix tensorizer tests

[youkaichao]

Anything you want to discuss about vllm.

The tensorizer feature, introduced in #3476 , has been flaky recently.

An example failure: https://buildkite.com/vllm/fastcheck/builds/1061#0190ec55-50c1-4cf2-8380-bd7238e99cea

We call for help from the community, to fix the tests. Ideally, maybe the author of that PR, @sangstar .

We will mark it as flaky and non-blocking for tests first. If we cannot fix it in weeks, then we need to consider removing the test.

[sangstar]

Hi there @youkaichao and thanks for making me aware of this.

Are there any other test failing other test_vllm_tensorized_model_has_same_outputs's out of memory erroring? If not, why does #6810 remove tensorizer tests entirely from the CI pipeline, rather than simply skipping that test? Breaking changes to tensorizer's support can now be surreptitiously merged, unless I'm unaware of the implications of a soft fail.

Also, have there been any changes to test runners? This test wasn't complaining about OOM errors previously. Can we revert #6810 relegating tensorizer tests to a soft fail and just skip the failing test(s)? In the meantime I'll look towards fixing them.

Is there a possibility that this test is actually failing helpfully? Why is the runner unable to allocate the 1.5 GB of VRAM? An OOM occurred merely from trying to instantiate this

vllm_runner(model_ref)

which makes me scratch my head as to why this would be indicative of the test itself being flaky. Are we sure proper cleanup is being carried out between tests? Are loaded models properly garbage collected between tests? Performing functional tests with large models and a garbage collected language will naturally present OOM risks unless proper cleanup is enforced.

[youkaichao]

#6810 separates the test, but tensorizer test is still in the full ci test:

vllm/.buildkite/test-pipeline.yaml

Line 221 in 062a1d0

- label: Tensorizer Test

previously we test it twice, combined with some other tests.

[sangstar]

#6810 separates the test, but tensorizer test is still in the full ci test:

vllm/.buildkite/test-pipeline.yaml

Line 221 in 062a1d0

- label: Tensorizer Test

previously we test it twice, combined with some other tests.

I see, thank you for the clarification! I will investigate this issue to ensure we can maintain full tensorizer testing in the CI suite.

[youkaichao]

Are we sure proper cleanup is being carried out between tests? Are loaded models properly garbage collected between tests?

proper cleanup and garbage collection is indeed a possible cause. You can try to investigate, and, if this is the cause, you can at least isolate the tests.

[sangstar]

Are we sure proper cleanup is being carried out between tests? Are loaded models properly garbage collected between tests?

proper cleanup and garbage collection is indeed a possible cause. You can try to investigate, and, if this is the cause, you can at least isolate the tests.

Sounds good. Thank you for bringing this to my attention! I'll be investigating this matter.

[youkaichao]

@sangstar it seems it becomes flaky again:

https://buildkite.com/vllm/ci-aws/builds/5830#0190f9e8-82ac-4fef-a7b7-53b012f20372

can you take a further look? thanks!

[sangstar]

@sangstar it seems it becomes flaky again:

https://buildkite.com/vllm/ci-aws/builds/5830#0190f9e8-82ac-4fef-a7b7-53b012f20372

can you take a further look? thanks!

Hi, thanks for bringing this to my attention. Will take a look. Working on it here #6881

[sangstar]

@youkaichao PR for the fix ready!

[youkaichao]

Looks like it is fixed now https://github.com/vllm-project/vllm/commits/main/ .

@sangstar thanks for the quick response!