[Feature]: Update Outlines Integration from `FSM` to `Guide`

[simon-mo]

🚀 The feature, motivation and pitch

Recently outlines updated their interface from FSM to Guide to support "acceleration"/"fast-forward" which will output next sets of tokens if they are directly available. For JSON schema, the cases are the keys, the ", and } etc.

This is non-trivial but very useful to improve vLLM for. It should also help other framework like AICI #3714.

Alternatives

No response

Additional context

No response

[br3no]

@simon-mo do you think this is an issue that could be driven by a community contribution? I'd be interested in working on it, but would most probably need a more knowledgeable counterpart to guide me through it.

[simon-mo]

Yes. Contributions welcomed.

[br3no]

Great.

Here a first conceptual sketch:

1. introduce a new abstraction (or extend the logits processor interface) that can be implemented with the support of an outlines Guide (https://github.com/outlines-dev/outlines/blob/main/outlines/fsm/guide.py).
2. In essence there is the need to introduce a new call to the guided generation engine on each LLM generation step to check whether there are ff-tokens and if so, add them to the sequence.
3. This engine "instance" must be the same that is used in the LogitsProcessor layers in the models when there are no ff-tokens but logit biases.

Would execute_model in model_runner.py be the right place to put this call?

I'm not sure how well this sketch aligns with AICI's protocol (https://github.com/microsoft/aici/blob/main/docs/aicirt-proto.md) – it seems to be a bit more involved, but it might be close enough to count as a first step.

What do you think? Is this the right direction?

[simon-mo]

I would break this down into two steps. First for compatible we should migrate the interface without considering fast forward. Next when the spec decode framework is merged, we will supply the fast forward token because it becomes easy to add them by then.