[Performance]: JSONLogitsProcessor repeats the same build_regex_from_schema
again and again
#8383
Labels
performance
Performance-related issues
Proposal to improve performance
Why does the regex get recompiled on every request? the object gets instantiated for every request and the regex is rebuilt even though most of the time the schema is likely to remain the same.
vllm/vllm/model_executor/guided_decoding/outlines_logits_processors.py
Line 142 in b1f3e18
probably
@lru_cache
should do the trick here? does it work on object constructors? or would it be cleaner to add a wrapper aroundbuild_regex_from_schema
and cache that?This is otherwise similar to #8308?
The text was updated successfully, but these errors were encountered: