[Feature]: Support outlines versions > v0.1.0 #10489
Labels
Comments
torymur
pushed a commit
to dottxt-ai/outlines-core
that referenced
this issue
Dec 2, 2024
I understand that `pickleable` is not your priority right now. But the `RegexGuide` needs to be pickled for `vllm` production use, which is multiprocessing-based. This PR reintroduces this pickling capability + some tests. I understand that this introduces more effort on your side. References: dottxt-ai/outlines#1274 vllm-project/vllm#10490 vllm-project/vllm#10576 vllm-project/vllm#10489 It would also tackle the current caching issues: huggingface/text-generation-inference#2766 dottxt-ai/outlines#1283 Closes: #95
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
🚀 The feature, motivation and pitch
Outlines (used for guided decoding) has had a minor upgrade on 7th of Oct (ref below).
This has improved their internal workings and should have a good impact on the performance downstream (on vllm).
At the moment the version used by vllm is pinned but we have successfully spinned up vllm ourselves internally with just upgrading this dependency (there are no relevant breaking changes for vllm) and it works out of the box.
https://github.com/dottxt-ai/outlines/releases/tag/0.1.0
Alternatives
No response
Additional context
No response
Before submitting a new issue...
The text was updated successfully, but these errors were encountered: