-
-
Notifications
You must be signed in to change notification settings - Fork 5.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support arbitrary json_object in OpenAI and Context Free Grammar #3211
Conversation
|
||
def __init__(self, regex_string: str, tokenizer): | ||
"""Compile the FSM that drives the regex-structured generation. | ||
def adapt_tokenizer(self, tokenizer): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for this nice feature! Left some comments.
|
||
|
||
class GuidedDecodingMode(Enum): | ||
JSON = "json" | ||
REGEX = "regex" | ||
CHOICE = "choice" | ||
CFG = "cfg" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A dumb question: does CFG
mean config
? only for grammar config?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Context free grammar, good point that I should probably change it to grammar
to disambiguate
|
||
class CFGLogitsProcessor(BaseLogitsProcessor): | ||
|
||
def __init__(self, cfg: str, tokenizer): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Adding type definition for tokenizer
in this whole file is better.
tokenizer.decode = change_decoder(tokenizer.decode) | ||
setattr(tokenizer, "_outlines_adapted", True) # noqa: B010 | ||
|
||
return tokenizer |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need return this?
Excellent new feature! Thanks everyone! I will definitely test it when the next vLLM release comes out. |
Is this available in the new vLLM library? How do I use this feature with Mixtral8x7b? |
This is available in latest release. https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html |
Please keep this issue closed, but just had to say: Oh man I love vLLM much, you guys are all awesome. Thank you! |
Closes #3148
Closes #1229