[CI Failure]: Spec Decoding - spec_decode/e2e/test_multistep_correctness.py

[DarkLight1337]

Name of failing test

test_spec_decode_e2e_greedy_correctness_tiny_model_large_bs_diff_output_len

Basic information

• Flaky test
• Can reproduce locally
• Caused by external libraries (e.g. bug in transformers)

🧪 Describe the failing test

https://buildkite.com/vllm/ci/builds/21085/steps?jid=01971f59-be20-45f4-9e11-dcd3b1e67173

📝 History of failing test

Started failing since yesterday since today's nightly caught the failure

https://buildkite.com/organizations/vllm/analytics/suites/ci-1/tests/a8e2f4a9-a7cf-81ec-ba3e-3a1e8b022a79?period=1day&tags=scm.branch%3Amain

CC List.

@mgoin @njhill

[njhill]

This should hopefully be fixed by #18978 and/or #18934.

cc @simon-mo

[DarkLight1337]

test_eagle_correctness.py has been fixed by #18978 but there test_multistep_correctness.py still fails.

[yewentao256]

Able to reproduce the issue today in main branch:

pytest tests/spec_decode/e2e/test_multistep_correctness.py -x

Traceback (most recent call last):
  File "/home/wentao/vllm-source/tests/utils.py", line 729, in wrapper
    f(*args, **kwargs)
  File "/home/wentao/vllm-source/tests/spec_decode/e2e/test_multistep_correctness.py", line 266, in test_spec_decode_e2e_greedy_correctness_tiny_model_large_bs
    run_equality_correctness_test(vllm_runner,
  File "/home/wentao/vllm-source/tests/spec_decode/e2e/conftest.py", line 236, in run_equality_correctness_test
    check_outputs_equal(outputs_0_lst=[out[0:2] for out in org_outputs],
  File "/home/wentao/vllm-source/tests/models/utils.py", line 44, in check_outputs_equal
    assert output_str_0 == output_str_1, fail_msg
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Test2:
org:    ' the largest city in the world.\nThe city is located in the heart of the city, the capital of the French province of the French Republic. ...'
sd:     ' the largest city in the world.\nThe city is located in the heart of the city, the capital of the French Empire. The city is the capital of the French Empire, ...'
==================================================================== 1 failed, 10 passed, 11 warnings in 346.06s (0:05:46) =====================================================================

[mgoin]

@DarkLight1337 Is this test important? We could skip the failing tests for now while this tracking issue is up

[DarkLight1337]

I'm not familiar with speculative decoding code so I'm not sure... cc @simon-mo

[DarkLight1337]

It seems that those in charge of this feature aren't really working towards keeping CI green...

[njhill]

Fixed by @houseroad in #19633