AIFQA-393 BLK-007: [vLLM/lm_eval] ChatGLM2/3 tokenizer — empty stop string crash in lm_eval

[jklawikowski]

Automated implementation for AIFQA-393.

Summary: BLK-007: [vLLM/lm_eval] ChatGLM2/3 tokenizer — empty stop string crash in lm_eval

Branch: dev/sys_qaplatformbot/AIFQA-393--20260330160725

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[jklawikowski]

Reviewer -- Automated Code Review

Verdict: APPROVED ✅

Summary: This PR fixes a real bug where ChatGLM2/3 tokenizers decode their EOS token to an empty string, which gets passed as a stop sequence and causes downstream errors in sampling backends like vLLM. The fix correctly filters out empty/whitespace-only stop sequences in handle_stop_sequences() and includes comprehensive test coverage. The change is minimal, well-scoped, and the docstring clearly explains the motivation.

Stats: +62 -3 in 2 file(s)

Comments

1. 🔵 [SUGGESTION] lm_eval/models/utils.py -- The filtering uses s.strip() which is truthy-based. This is fine for the current list[str] contract, but if a non-string value ever sneaks into the until list, this would raise an AttributeError. Consider whether a guard like s and s.strip() or an isinstance check would be warranted for defensive programming. This is very low priority given the type annotation already constrains to str.