[Feature] Guided Decoding

[QwertyJack]

Motivation

Currently, vLLM allows users to specify the output format via the parameter extra_body, where the under-the-hood magic comes from Outlines / LMFE.

_ = client.chat.completions.create(
    ...
    messages=[
        {'role': 'user', 'content': 'What colour it the sea?'},
    ],
    extra_body={
        'guided_choice': ['red', 'yellow', 'blue', 'green'],
    },
)

# output: blue

Apart from classification, type / Pydantic / json schema / regex / grammar are also supported.

Related resources

Refs:

• Outlines: https://github.com/outlines-dev/outlines
• LM-Format-Enforcer: https://github.com/noamgat/lm-format-enforcer
• vLLM - JSON formatting issue
• vLLM - Add guided decoding for OpenAI API server

Additional context

No response

[vody-am]

@QwertyJack It would be cool to have this built in, but if you find this helpful, here's an example of using instructor to do this with the OpenAI interface! It's not the fastest though.

from openai import OpenAI
from pydantic import BaseModel, Field
import base64
import instructor
import time
from enum import Enum
from typing import List
from glob import glob

class Property(BaseModel):
    key: str = Field(description="Must be snake case")
    value: str

class House(Enum):
    Griffindor = "gryffindor"
    Hufflepuff = "hufflepuff"
    Ravenclaw = "ravenclaw"
    Slytherin = "slytherin"
    Targaryen = "targaryen"

# This schema is what guides generation
class Character(BaseModel):
    name: str
    age: int
    house: House
    properties: List[Property]

api_client = OpenAI(api_key='YOUR_API_KEY', base_url='http://0.0.0.0:23333/v1')
client = instructor.from_openai(api_client, mode=instructor.Mode.MD_JSON)
model_name = api_client.models.list().data[0].id

def run(url):
    return client.chat.completions.create(
        model=model_name,
        response_model=Character,
        messages=[{
            'role':
            'user',
            'content': [
                {
                    'type': 'text',
                    'text': 'Who is the character in this image?',
                },
                {
                    'type': 'image_url',
                    'image_url': {
                        'url': url,
                    },
                },
            ],
        }],
        temperature=0.0,
    )

for url in [
    'https://upload.wikimedia.org/wikipedia/en/d/d7/Harry_Potter_character_poster.jpg',
    'https://upload.wikimedia.org/wikipedia/en/thumb/d/d3/Hermione_Granger_poster.jpg/220px-Hermione_Granger_poster.jpg',
    'https://upload.wikimedia.org/wikipedia/en/5/5e/Ron_Weasley_poster.jpg',
    'https://upload.wikimedia.org/wikipedia/en/1/16/Draco_Mal.JPG',
    'https://upload.wikimedia.org/wikipedia/en/0/0d/Daenerys_Targaryen_with_Dragon-Emilia_Clarke.jpg',
    ]:
    print(run(url))

# Output
# name='Harry Potter' age=17 house=<House.Griffindor: 'gryffindor'> properties=[Property(key='Wand', value='Holly and Phoenix Feather'), Property(key='Eye Color', value='Green'), Property(key='Hair Color', value='Brown')]
# name='Hermione Granger' age=16 house=<House.Griffindor: 'gryffindor'> properties=[Property(key='wand', value='10 1/2 inches, vine wood, phoenix feather'), Property(key='patronus', value='Otter'), Property(key='best friends', value='Harry Potter and Ron Weasley')]
# name='Harry Potter' age=17 house=<House.Griffindor: 'gryffindor'> properties=[Property(key='Wand', value='Holly and Phoenix Feather'), Property(key='Eye Color', value='Green'), Property(key='Hair Color', value='Red')]
# name='Harry Potter' age=11 house=<House.Griffindor: 'gryffindor'> properties=[Property(key='Wand', value='Holly and Phoenix Feather'), Property(key='Eye Color', value='Green'), Property(key='Hair Color', value='Blonde')]
# name='Daenerys Targaryen' age=30 house=<House.Targaryen: 'targaryen'> properties=[Property(key='dragons', value='three'), Property(key='titles', value='Mother of Dragons, Khaleesi of the Great Grass Sea, the Unburnt, Breaker of Chains')]

[QwertyJack]

Thanks