Skip to content

Conversation

@Ubospica
Copy link
Collaborator

@Ubospica Ubospica commented Jan 21, 2025

This PR

  • Supports the parsing and mask generation algorithm for TagDispatch in GrammarMatcherBase and GrammarMatcher
  • Refactors GrammarMatcherBase and GrammarMatcher along the way
  • Designs the CSRArray and FSM data structures which are useful in later architecture
  • Supports the structural tag feature with the above backend support.

Description of the structural tag:

The structural tag handles the dispatching
of different grammars based on the tags and triggers: it initially allows any output,
until a trigger is encountered, then dispatch to the corresponding tag; when the end tag
is encountered, the grammar will allow any following output, until the next trigger is
encountered.

The tags parameter is used to specify the output pattern. It is especially useful for LLM
function calling, where the pattern is:
<function=func_name>{"arg1": ..., "arg2": ...}.
This pattern consists of three parts: a start tag (<function=func_name>), a parameter list
according to some schema ({"arg1": ..., "arg2": ...}), and an end tag (). This
pattern can be described in a StructuralTagItem with a start tag, a schema, and an end tag.
The structural tag is able to handle multiple such patterns by passing them into multiple
tags.

The triggers parameter is used to trigger the dispatching of different grammars. The trigger
should be a prefix of a provided start tag. When the trigger is encountered, the
corresponding tag should be used to constrain the following output. There can be multiple
tags matching the same trigger. Then if the trigger is encountered, the following output
should match one of the tags. For example, in function calling, the triggers can be
["<function="]. Then if "<function=" is encountered, the following output must match one
of the tags (e.g. <function=get_weather>{"city": "Beijing"}</function>).

The corrrespondence of tags and triggers is automatically determined: all tags with the
same trigger will be grouped together. User should make sure any trigger is not a prefix
of another trigger: then the corrrespondence of tags and triggers will be ambiguous.

To use this grammar in grammar-guided generation, the GrammarMatcher constructed from
structural tag will generate a mask for each token. When the trigger is not encountered,
the mask will likely be all-1 and not have to be used (fill_next_token_bitmask returns
False, meaning no token is masked). When a trigger is encountered, the mask should be
enforced (fill_next_token_bitmask will return True, meaning some token is masked) to the
output logits.

The benefit of this method is the token boundary between tags and triggers is automatically
handled. The user does not need to worry about the token boundary.

@Ubospica
Copy link
Collaborator Author

cc @tqchen @zanderjiang

@Ubospica Ubospica merged commit 69e45d2 into mlc-ai:main Jan 21, 2025
1 check passed
@Ubospica Ubospica mentioned this pull request Jan 21, 2025
Ubospica added a commit that referenced this pull request Jan 21, 2025
This PR removes the debug checking in #162.
Ubospica added a commit that referenced this pull request Feb 13, 2025
This PR removes the grammar_functor.cc mistakenly added in #162.
@CharlieFRuan CharlieFRuan mentioned this pull request Jul 15, 2025
14 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant