[Misc] [CI]: Flaky test failure in test_chatglm3_lora

[cadedaniel]

• main CI failed after [Misc] [Core] Implement RFC "Augment BaseExecutor interfaces to enable hardware-agnostic speculative decoding" #3837 was merged
• tracking in an issue in case we need to fix-forward/revert/or maybe the test is flaky @youkaichao confirmed the test is flaky

    def test_chatglm3_lora(chatglm3_lora_files):
        llm = vllm.LLM(MODEL_PATH,
                       max_model_len=1024,
                       enable_lora=True,
                       max_loras=4,
                       max_lora_rank=64,
                       trust_remote_code=True)

        expected_lora_output = [
            "SELECT count(*) FROM singer",
            "SELECT avg(age) ,  min(age) ,  max(age) FROM singer WHERE country  =  'France'",  # noqa: E501
            "SELECT name ,  country ,  age FROM singer ORDER BY age",
        ]

        output1 = do_sample(llm, chatglm3_lora_files, lora_id=1)
        for i in range(len(expected_lora_output)):
            assert output1[i] == expected_lora_output[i]
        output2 = do_sample(llm, chatglm3_lora_files, lora_id=2)
        for i in range(len(expected_lora_output)):
>           assert output2[i] == expected_lora_output[i]
E           AssertionError: assert '' == 'SELECT count(*) FROM singer'
E             
E             - SELECT count(*) FROM singer

[youkaichao]

I re-tried the test and it passed. Should be a flaky one.

[cadedaniel]

thanks for that @youkaichao . let's reopen this issue if it happens again -- we can skip the test