ConstrainCall is an experimental structured-generation engine that converts natural-language requests into typed function calls using a small language model and token-level constrained decoding.
The project explores an important question:
Can a relatively small language model be transformed from a free-form text generator into a deterministic structured-output engine by controlling the decoding process?
Instead of allowing the language model to generate arbitrary text and attempting to repair the result afterward, ConstrainCall moves part of the responsibility from the model into the decoder.
The basic architecture is:
Natural Language
│
▼
┌────────────┐
│ Tokenizer │
└─────┬──────┘
│
▼
Token IDs
│
▼
┌──────────────────┐
│ Transformer LLM │
└────────┬─────────┘
│
▼
Logits
│
▼
┌──────────────────┐
│ Constraint Layer │
└────────┬─────────┘
│
Valid token set
│
▼
Masked logits
│
▼
Token selection
│
▼
Next Token ID
│
└──────────────┐
│
▼
Repeat decoding
The central observation is simple:
The model does not need to be allowed to choose every possible token.
If the decoder knows that the next token must be part of a function name, it can eliminate every token that cannot produce a valid function name.
If the decoder knows that the next value must be numeric, it can eliminate tokens that cannot continue a valid numeric representation.
This is the fundamental mechanism behind constrained decoding.
A standard causal language model estimates:
At every generation step, it produces a probability distribution over the vocabulary.
For a vocabulary (V):
for Qwen3-0.6B.
Therefore, at every step, the model conceptually says:
token A -> probability 0.00001
token B -> probability 0.42
token C -> probability 0.003
...
token N -> probability 0.0002
The model has no inherent obligation to produce valid JSON.
For example, given:
What is the sum of 2 and 3?
an unconstrained model could produce:
The answer is 5.
or:
{"name":"fn_add_numbers","parameters":{"a":2,"b":3}}or:
Sure! Here is the function call:
{"name": ...}
For a human, all three might be understandable.
For an automated function-calling interface, only one may be acceptable.
ConstrainCall therefore separates:
Semantic generation
from:
Structural validity
The model decides what should be generated.
The constraint engine decides what is legally allowed to be generated.
The project uses uv for dependency management.
uv syncuv run python -m constraincall \
--functions_definition data/input/functions_definition.json \
--input data/input/function_calling_tests.json \
--output data/output/functions_result.jsonuv run flake8 .uv run mypy . --strictAt a high level:
┌───────────────────────────────────────────────┐
│ Input Files │
│ │
│ functions_definition.json │
│ function_calling_tests.json │
└───────────────────────┬───────────────────────┘
│
▼
┌──────────────────┐
│ Input Validation │
└────────┬─────────┘
│
┌─────────┴──────────┐
│ │
▼ ▼
Functions Prompts
│ │
└─────────┬──────────┘
│
▼
Model Instance
│
▼
Tokenizer / Vocabulary
│
▼
ConstrainedDecoder
│
▼
Structured Function Call
│
▼
Type Validation
│
▼
JSON Output
The important design principle is that the decoder does not simply ask:
"What text should the model generate?"
Instead, it asks:
"Given the model's preferences, which of the currently legal tokens should be selected?"
A useful mental model is:
Text
│
▼
Tokenizer
│
▼
Token IDs
│
▼
Embedding Lookup
│
▼
Positional Information
│
▼
Transformer Layers
│
├── Self Attention
│
├── MLP
│
├── Normalization
│
└── Residual Connections
│
▼
Hidden State
│
▼
Language Model Head
│
▼
Logits
│
▼
Constraint / Sampling
│
▼
Next Token ID
│
▼
Decode
│
▼
Text
Generation repeats this process token by token.
Consider:
What is the sum of 2 and 3?
The neural network does not directly operate on this string.
The tokenizer converts it into integer IDs.
Conceptually:
"What" -> 120
"is" -> 54
"the" -> 88
"sum" -> 230
...
The exact tokenization depends on the tokenizer.
Modern LLM tokenizers usually operate on subwords or byte-level representations rather than simply splitting on spaces.
For example:
Programming
might become:
Program
ming
while another string might remain one token.
The tokenizer therefore implements a mapping:
where:
-
$\Sigma^*$ is the set of possible input strings. -
$V$ is the model vocabulary. -
$V^*$ is a sequence of vocabulary tokens.
The result is a sequence of integers:
std::vector<int> input_ids;For Qwen3-0.6B, the configured vocabulary size is 151,936.
The important point is:
A token ID is not a word ID.
A token can represent:
a word
or:
a subword
or:
punctuation
or:
whitespace
or:
a byte-level fragment
or:
a special token
This is extremely important for constrained decoding.
Suppose:
"hello" -> 120
Then:
int token_id = 120;does not mean that 120 mathematically represents "hello".
It is simply an index into the vocabulary and model embedding table.
Conceptually:
Token ID
│
▼
120
│
▼
Embedding Matrix Row 120
The model contains an embedding matrix:
where:
-
$|V|$ = vocabulary size -
$d$ = hidden/embedding dimension
For Qwen3-0.6B, the hidden size is 1024 according to the model configuration.
Conceptually:
If:
token_id = 120
then:
is selected.
In C++-like pseudocode:
std::vector<float> embedding =
embedding_matrix[120];This is an embedding lookup, not a matrix multiplication over the entire vocabulary.
The token ID is simply used as an index.
Transformers must know that:
A B C
is different from:
C B A
Therefore, the representation contains information about token position.
Modern transformer models commonly use positional mechanisms such as rotary positional embeddings (RoPE).
Conceptually:
Token embedding
+
Position information
│
▼
Transformer representation
Without positional information, the model would have difficulty distinguishing different permutations of the same token set.
The resulting representations pass through multiple transformer blocks.
A simplified block is:
Input
│
▼
Normalization
│
▼
Self Attention
│
▼
Residual Connection
│
▼
Normalization
│
▼
MLP
│
▼
Residual Connection
│
▼
Output
Qwen3-0.6B is configured with 28 hidden layers and 16 attention heads.
Self-attention is the mechanism allowing tokens to exchange contextual information.
For hidden representation matrix (X), attention conceptually computes:
and:
The matrix:
produces compatibility scores between positions.
For:
What is the sum of 2 and 3?
the representation associated with sum can receive information from tokens such as:
2
3
and the surrounding context.
This does not mean that the model literally stores a symbolic relation:
sum -> 2
sum -> 3
Instead, those relationships emerge through learned numerical representations and attention weights.
The original Transformer architecture is described in Attention Is All You Need.
After the transformer processes the sequence, the model produces a hidden representation for the current position.
The language-model head projects that representation into vocabulary space.
Conceptually:
where:
- (h) is the current hidden state.
- (W_{out}) projects into vocabulary space.
- (z) is the logit vector.
Therefore:
For Qwen3-0.6B:
logits[0]
logits[1]
logits[2]
...
logits[151935]
Each entry corresponds to one vocabulary token.
Example:
Token Logit
"5" 12.4
"4" 8.1
"cat" -2.7
"dog" -4.2
"hello" -5.8
The logits are scores, not probabilities.
To convert logits into probabilities:
This produces:
For example:
5 -> 0.81
4 -> 0.12
cat -> 0.01
dog -> 0.003
...
The decoder can then either:
argmax
or sample from the distribution.
Greedy decoding:
means:
Choose the token with the highest score.
Sampling instead draws a token according to a probability distribution.
Common techniques include:
- temperature
- top-k
- top-p / nucleus sampling
- repetition penalties
- custom logits processors
Hugging Face, for example, exposes a LogitsProcessor abstraction specifically for modifying logits before token selection.
Suppose the model needs to output:
{
"name": "fn_add_numbers",
"parameters": {
"a": 2,
"b": 3
}
}A normal LLM is free to generate:
Sure! Here's the result:
or:
{"name": "fn_add_numbers",}or:
{"name": "fn_add_numbers", "parameters": {"a": "two", "b": 3}}The model's objective is probabilistic next-token prediction.
It is not inherently a JSON parser.
This leads to a fundamental distinction:
LLM probability
≠
formal validity
Constrained decoding bridges that gap.
Let:
be the complete model vocabulary.
At generation step
where:
-
$x$ is the prompt. -
$y_{\lt t}$ is the generated prefix.
A constrained decoder defines a set:
where:
-
$s_t$ is the current constraint state. -
$A(s_t)$ contains only tokens that are legal at this state.
Then generation becomes:
This equation captures the core idea of ConstrainCall.
The model still determines preference.
The decoder determines legality.
Suppose the model wants to generate:
"a": 2
and the current grammar state expects a number.
The complete vocabulary might contain:
"hello"
"cat"
"2"
"3"
"-"
"."
"{"
"}"
...
The constraint system might calculate:
Allowed:
2
3
4
5
...
9
-
.
and reject:
hello
cat
{
}
Therefore:
Instead of physically deleting tokens from the model's output vector, we modify their logits.
For every token
Then:
assigns zero probability to every forbidden token because:
This is the fundamental mathematical mechanism used by many constrained-generation systems.
ConstrainCall's procedural approach is essentially:
1. Run model
2. Obtain logits
3. Determine valid token IDs
4. Mask invalid logits
5. Select highest-scoring valid token
6. Append token
7. Update constraint state
8. Repeat
ConstrainCall uses a procedural, phase-based constrained decoder.
Instead of implementing a general-purpose grammar engine, the decoder knows the expected function-call structure.
Conceptually:
JSON object
│
├── "name"
│ └── constrained function-name generation
│
└── "parameters"
│
├── parameter 1
│ ├── number generation
│ └── string generation
│
├── parameter 2
│
└── ...
This is substantially simpler than a general JSON grammar engine.
Suppose the available functions are:
[
{
"name": "fn_add_numbers",
"parameters": {
"a": {"type": "number"},
"b": {"type": "number"}
}
},
{
"name": "fn_greet",
"parameters": {
"name": {"type": "string"}
}
},
{
"name": "fn_reverse_string",
"parameters": {
"s": {"type": "string"}
}
}
]Given:
What is the sum of 2 and 3?
the desired result is:
{
"name": "fn_add_numbers",
"parameters": {
"a": 2,
"b": 3
}
}The model must solve two separate problems:
Which function is appropriate?
fn_add_numbers
How must that function call be serialized?
{
"name": "...",
"parameters": {
"a": 2,
"b": 3
}
}ConstrainCall uses the model for semantic prediction and the decoder for structural control.
The decoder does not ask the model to freely generate:
{"name":"...","parameters":{...}}Instead, it constructs the output in phases.
Conceptually:
FORCE:
{"name": "
GENERATE:
fn_add_numbers
FORCE:
", "parameters": {"
FORCE:
"a":
GENERATE:
2
FORCE:
, "b":
GENERATE:
3
FORCE:
}}
This is why the technique can be thought of as:
Procedural constrained infilling.
The model fills only the parts where semantic generation is needed.
The decoder needs to know not only:
string -> token ID
but also:
token ID -> string
Why?
Because the model returns:
logits[12345]
and the decoder needs to know:
What text does token 12345 represent?
Therefore:
id_to_token[12345]might return:
"fn_"
This reverse mapping is essential for token-level constraints.
For example:
token_string = id_to_token[token_id]allows the decoder to inspect whether the token contains:
"
or:
0
1
2
...
or whether it continues a function-name prefix.
The Qwen tokenizer files are distributed with the model; the model repository includes tokenizer.json and tokenizer configuration.
Suppose:
token ID 100 = "fn_"
token ID 101 = "add"
token ID 102 = "_numbers"
token ID 103 = "hello"
The decoder has logits:
100 -> 4.5
101 -> 8.7
102 -> 2.1
103 -> 9.2
The highest raw logit is:
103
but:
103 = "hello"
If the decoder is currently constructing a function name, "hello" may be illegal.
Therefore it needs:
ID -> token string
to determine legality.
This is one of the fundamental differences between ordinary inference and custom token-level constrained inference.
Suppose the valid function names are:
fn_add_numbers
fn_greet
fn_reverse_string
fn_get_square_root
fn_substitute_string_with_regex
The decoder starts with:
partial = ""
Every function is initially compatible.
After generating:
fn_
all functions are still possible.
After:
fn_a
only:
fn_add_numbers
remains.
The constraint becomes:
For:
prefix = "fn_a"
we obtain:
C("fn_a")
=
{
"fn_add_numbers"
}
The decoder can therefore eliminate all vocabulary tokens that cannot extend this prefix.
This algorithm naturally corresponds to a prefix tree (Trie).
For example:
fn_
│
┌──────────┼──────────┐
│ │ │
a g r
│ │ │
d r e
│ │ │
d e v
│ │ │
numbers et erse
A Trie makes prefix lookup much more efficient than repeatedly scanning every function name.
A production implementation could therefore replace:
for name in self.func_names:with a Trie traversal.
This is one of the first optimizations worth making.
Suppose the decoder has already emitted:
"a":
and the schema says:
"a": {
"type": "number"
}The decoder now maintains:
partial = ""
It examines candidate tokens.
Suppose:
token 10 -> "2"
token 11 -> "3"
token 12 -> "abc"
token 13 -> "-"
token 14 -> "."
The decoder asks:
If I append this token to the current partial string, does the result remain a valid number?
For example:
partial = "12"
candidate = "3"
gives:
"123"
which is valid.
But:
partial = "12"
candidate = "abc"
gives:
"12abc"
which is invalid.
Therefore:
"3" -> allowed
"abc" -> forbidden
The number constraint can be understood formally.
Suppose the accepted language is:
-?(\d+\.?\d*|\.\d+)
This describes strings such as:
0
1
2
10
123
1.5
3.14159
-2
-10.5
.5
-.5
and rejects:
abc
1abc
--2
..
1.2.3
The decoder effectively asks:
where:
is the language accepted by the numeric constraint.
This is a much deeper concept than simply "checking a regex".
It is incremental language recognition.
There is an important distinction:
A generated string does not necessarily need to be a complete valid number after every token.
For example:
"-"
is not a complete number under many definitions.
But it can be a valid prefix of:
-12
Similarly:
"12."
may be a valid prefix of:
12.5
Therefore, a robust constrained decoder should distinguish:
complete
from:
valid prefix
Conceptually:
"1"
│
├── complete
└── can continue
"1."
│
├── maybe incomplete
└── can continue
"1.5"
│
└── complete
This is precisely the type of state tracking that formal automata provide.
Strings are more difficult because almost arbitrary characters may be valid.
The simple implementation therefore creates:
str_safe_idscontaining tokens that do not contain:
"
This prevents the generated string from accidentally terminating the JSON string.
For example:
hello
is safe.
But a token containing:
"
could terminate:
"name": "hello"prematurely.
The implementation therefore masks such tokens while generating a string and later injects the closing quote.
This is a deliberately simplified string constraint.
A fully correct JSON string constraint would also need to reason about:
\"
\\
\n
\t
\uXXXX
and other JSON escaping rules.
One of the most important architectural ideas in ConstrainCall is the distinction between:
These are known ahead of time:
{"name": "
", "parameters": {"
":
, "
"}
The decoder can tokenize these strings and directly append the resulting token IDs.
These require the model to make a decision:
fn_add_numbers
2
3
shrek
hello
Therefore:
Structure
│
└── deterministic
Semantic values
│
└── model-driven + constrained
This reduces the amount of freedom exposed to the model.
Input:
What is the sum of 2 and 3?
Available function:
fn_add_numbers
with:
a: number
b: number
The target is:
{"name":"fn_add_numbers","parameters":{"a":2,"b":3}}The decoder creates a system description similar to:
Functions:
fn_add_numbers(a: number, b: number):
Add two numbers together and return their sum.
...
Output JSON:
{"name":"<fn>","parameters":{<args>}}
User:
What is the sum of 2 and 3?
Assistant:
The purpose is to provide the model with semantic context.
The prompt tells the model:
These are the available operations.
This is their meaning.
This is the desired output structure.
This is the user's request.
Conceptually:
prompt_ids = model.encode(full_prompt)[0].tolist()The result is:
prompt text
│
▼
tokenizer
│
▼
[101, 542, 17, 982, ...]
prompt_ids represents the complete textual context given to the model.
Initially:
gen_ids = []Then:
gen_ids.extend(
self._enc('{"name": "')
)Suppose the tokenizer produces:
"{" -> 500
"name" -> 1200
":" -> 91
" " -> 32
Then:
gen_ids =
[
500,
1200,
91,
32
]
gen_ids is therefore the sequence of tokens that the decoder has committed to generating.
It is not the KV cache.
It is not the model's hidden state.
It is not the logits.
It is simply token history.
This distinction is extremely important.
[500, 1200, 91, 32, ...]
contains token IDs.
contains internal Transformer tensors such as:
K layer 0
V layer 0
K layer 1
V layer 1
...
K layer N
V layer N
The KV cache exists to avoid recomputing attention states for the entire sequence.
Therefore:
gen_ids ≠ KV cache
The _enc() function is also not the KV cache.
The helper:
def _enc(self, text):
if text not in self._enc_cache:
self._enc_cache[text] = \
self.model.encode(text)[0].tolist()
return self._enc_cache[text]is simply an encoding cache.
If:
'{"name": "'
has already been encoded once:
_encode_cache['{"name": "']
stores its token IDs.
The next time it is needed, tokenization is avoided.
Therefore:
_enc()
means:
text
↓
tokenizer
↓
token IDs
↓
cache
It has nothing inherently to do with KV caching.
Now:
_generate_func_name(prompt_ids, gen_ids)is called.
The function's purpose is:
Generate a function name, but never allow a token that would make the final name invalid.
The model produces logits.
Suppose:
fn_add_numbers -> 12.4
fn_greet -> 8.2
hello -> 14.1
dog -> 10.2
Without constraints:
hello
would win.
With constraints:
hello -> forbidden
dog -> forbidden
Only tokens capable of extending a valid function name remain.
Initially:
partial = ""
Every function matches.
Suppose the model selects:
fn_
Then:
partial = "fn_"
Every function still starts with:
fn_
Suppose the next selected token is:
add
Now:
partial = "fn_add"
Only:
fn_add_numbers
matches.
Therefore the candidate space collapses.
The returned IDs:
fidare appended:
gen_ids.extend(fid)Now gen_ids represents:
{"name": "fn_add_numbers
in token-ID form.
The decoder checks:
if fname not in self.func_map:func_map is effectively:
function name
↓
function schema
For example:
fn_add_numbers
│
▼
{
a: number,
b: number
}
Therefore once:
fname = "fn_add_numbers"
is known, the decoder knows exactly which parameters must be generated.
The decoder appends:
", "parameters": {"
This is deterministic structure.
The model does not need to decide whether to emit:
,
or:
}
or:
"parameters"
The decoder already knows.
This is an important design principle:
Do not spend model inference on information the program already knows.
The schema says:
"a": {
"type": "number"
}The decoder injects:
"a":
Then calls:
_generate_number(...)Suppose the model produces:
"2" -> logit 15.3
"3" -> logit 4.2
"hello" -> logit 14.7
Without constraints:
"2" wins
which is fine.
But imagine:
"hello" -> 20
"2" -> 15
Ordinary greedy decoding would produce:
hello
which violates the schema.
The constrained decoder creates:
valid = {
token IDs that can continue a number
}
and applies:
m[i] = -np.inffor every invalid token.
The logits might become:
"hello" -> -inf
"2" -> 15
"3" -> 4
Then:
argmax()selects:
2
A subtle but important point:
gen_ids is not the set of valid candidate tokens.
It is the sequence of tokens that have actually been selected and committed to the generated output.
For example:
gen_ids:
[
token("{"),
token('"name"'),
token(":"),
token('"'),
token("fn_add_numbers"),
token('"'),
token(","),
token('"parameters"'),
token(":"),
token("{"),
token('"a"'),
token(":"),
token("2")
]
The set:
valid
is different.
valid means:
Tokens that are legal candidates at the current generation step.
gen_ids means:
Tokens that have already been selected.
This distinction is fundamental.
The decoder injects:
, "b":
Then:
_generate_number(...)runs again.
The current sequence now semantically represents:
{
"name": "fn_add_numbers",
"parameters": {
"a": 2,
"b":The model sees the entire context.
The constraint engine says:
b must be a number
Suppose:
3 -> 16.1
5 -> 7.2
hello -> 19.4
After masking:
3 -> 16.1
5 -> 7.2
hello -> -inf
The selected token is:
3
The decoder now knows:
function = fn_add_numbers
a = 2
b = 3
and constructs:
{
"name": "fn_add_numbers",
"parameters": {
"a": 2,
"b": 3
}
}A useful visualization is:
State 0
Prompt
+
"{"name": ""
│
▼
State 1
Generate function name
│
▼
"fn_add_numbers"
│
▼
State 2
Inject:
", "parameters": {"
│
▼
State 3
Generate parameter a
│
▼
2
│
▼
State 4
Inject:
, "b":
│
▼
State 5
Generate parameter b
│
▼
3
│
▼
State 6
Close JSON
│
▼
FINAL
This is already very similar to a finite-state machine.
The difference is that the current implementation encodes the states procedurally in Python rather than explicitly representing them as a formal automaton.
The decoder can be viewed as a transition system.
Let:
be the current decoding state.
Let:
be the vocabulary.
Let:
be the set of valid tokens from that state.
Then:
After selecting token
where:
is the state-transition function.
This gives:
Current State
│
│ token
▼
Next State
Many simple constraints can be represented by regular languages.
For example:
[0-9]+
can be recognized by a finite automaton.
The same applies to:
[a-z]+
or:
-?[0-9]+(\.[0-9]+)?
This means a regex constraint can often be compiled into an automaton.
Then constrained decoding becomes:
LLM logits
+
automaton state
│
▼
legal token IDs
│
▼
masked logits
This is the conceptual foundation behind FSM-based structured generation systems. Outlines, for example, describes structured generation using automata derived from regular expressions or grammars.
Regular expressions are not sufficient for every structured language.
JSON contains nested structures:
{
"a": {
"b": {
"c": 1
}
}
}Nested structures naturally require stack-like state.
A context-free grammar can represent this structure.
For example:
object
::= "{" members "}"
members
::= pair
| pair "," members
pair
::= string ":" value
value
::= string
| number
| object
| array
This is much more expressive than a simple regex.
llama.cpp, for example, provides GBNF grammar support for constrained generation and can convert supported JSON schemas into grammars.
A useful theoretical model for context-free grammars is a Pushdown Automaton (PDA).
A finite-state machine has:
state
A PDA additionally has:
stack
For nested JSON:
{
"a": {
"b": {
...
}
}
}
the stack can conceptually remember:
OBJECT
OBJECT
OBJECT
and pop them as:
}
}
}
are generated.
This is why general JSON constraints are more complicated than merely checking whether a character is allowed.
A general grammar-constrained decoder works approximately like:
Grammar
│
▼
Parser / Automaton
│
▼
Current State
│
▼
Valid Token Set
│
▼
LLM ───────► Logits ───────► Mask
│
▼
Token Selection
│
▼
State Update
The parser is effectively synchronized with generation.
This is one of the hardest parts of constrained decoding.
The grammar may operate on:
characters
while the model operates on:
tokens
Suppose the grammar allows:
abc
but the tokenizer contains:
"abc"
as one token.
The decoder must determine whether that token corresponds to a valid grammar transition.
More complicated:
"abc"
could tokenize as:
["ab", "c"]
or:
["a", "bc"]
or:
["a", "b", "c"]
depending on the tokenizer.
Therefore:
Grammar alphabet ≠ model vocabulary
A correct constrained decoder must bridge these two representations.
This tokenizer/grammar alignment problem is a major area of research in grammar-constrained decoding. Recent work explicitly focuses on efficiently computing token masks while preserving soundness with subword tokenizers.
A naive implementation might do:
for token_id in vocabulary:
text = decode(token_id)
if grammar_accepts(text):
allow(token_id)But if:
then every generation step potentially examines approximately 152k tokens.
If generation requires:
100 tokens
the naive approach can perform millions of grammar/token checks.
Therefore production systems optimize:
token classification
+
grammar execution
+
mask generation
XGrammar, for example, specifically addresses this problem by separating context-independent and context-dependent token processing and optimizing grammar execution.
XGrammar 2 further explores dynamic dispatch, JIT compilation, caching, and parser-based techniques for dynamic agentic structured generation.
Constrained decoding can be categorized into several levels.
Example:
Only allow token IDs:
{1, 5, 9, 20}
Example:
Must begin with:
fn_
Example:
-?[0-9]+(\.[0-9]+)?
The decoder maintains an explicit automaton state.
The decoder maintains parser/grammar state.
Example:
{
"name": "string",
"age": "integer"
}The schema can be compiled into a grammar or automaton.
These are more difficult.
For example:
a must be smaller than b
A JSON grammar alone cannot necessarily enforce this.
You may need:
grammar constraint
+
semantic validator
This distinction is extremely important:
Syntax
≠
Semantics
A decoder can guarantee:
{"a": 100, "b": 2}is syntactically valid while still violating:
a < b
ConstrainCall:
Procedural
+
Schema-aware
+
Manual state transitions
+
Dynamic token masking
General grammar engine:
Schema / Grammar
│
▼
Compiler
│
▼
Automaton / Parser
│
▼
Runtime state
│
▼
Token mask
The latter is more general.
The former is easier to understand and experiment with.
This makes ConstrainCall useful as an educational implementation.
| Approach | Structure | Generality | Complexity |
|---|---|---|---|
| Free generation | None | Very high | Low |
| Prompt-only JSON | Weak | High | Low |
| Manual masking | Medium | Medium | Medium |
| ConstrainCall procedural decoder | Strong for known schema | Medium | Medium |
| Regex → FSM | Strong | Medium | Medium |
| CFG/PDA | Very strong | High | High |
| Full grammar engine | Very strong | Very high | High |
| Optimized grammar engine | Very strong | Very high | Very high |
The model forward pass normally dominates inference cost.
However, constrained decoding introduces additional work.
A naive decoder might perform:
constraint checks per generation step.
For a vocabulary of:
that can become expensive.
Therefore useful optimizations include:
Precompute:
token -> character properties
token -> grammar transitions
token -> numeric compatibility
Use a Trie for:
function names
Compile regexes into finite-state machines.
Compile a schema once and reuse the compiled structure.
Cache masks for repeated grammar states.
Do not reparse the complete generated output every time.
Avoid recomputing the entire Transformer context.
It is important not to confuse:
token history
with:
KV cache
Token history:
[101, 203, 501, 91]
KV cache:
Layer 0:
K
V
Layer 1:
K
V
...
Layer 28:
K
V
gen_ids belongs to the first category.
The inference engine internally maintains the second.
A production C++ implementation should exploit the KV cache so that each new token does not require recomputing the complete prefix.
ConstrainCall's decoding is designed around greedy selection:
Therefore, if:
- model execution is deterministic,
- the tokenizer is deterministic,
- the constraint state is deterministic,
- no random sampling is used,
then the output is deterministic.
This is stronger than merely saying:
"The output is usually valid."
The decoder's goal is to make the generation path itself deterministic.
However, deterministic decoding does not automatically mean semantically correct decoding.
For example:
The model could deterministically choose the wrong function.
The constraint guarantees can be structural while semantic accuracy remains model-dependent.
This distinction is crucial.
A constrained decoder can potentially guarantee:
valid JSON syntax
while not guaranteeing:
correct interpretation of the user's request
For example:
User:
What is the sum of 2 and 3?
A decoder could produce:
{
"name": "fn_get_square_root",
"parameters": {
"a": 2
}
}and this could still be perfectly valid according to the JSON grammar.
Therefore:
ConstrainCall addresses the first problem directly and relies on the model plus prompt/schema information for the second.
The procedural implementation is intentionally simpler than a full grammar engine.
Important limitations include:
The decoder knows the JSON layout procedurally rather than compiling arbitrary JSON Schema.
A production JSON string recognizer must correctly handle escaping.
The numeric regex does not represent every possible JSON-number edge case.
A naive token mask can require inspecting many vocabulary entries.
A token can contain multiple characters, meaning character-level constraints cannot always be implemented by inspecting only the first character.
Structural constraints cannot guarantee semantic correctness.
A 0.6B model has substantially less semantic capacity than larger instruction-tuned models.
An important architectural distinction should be maintained between:
constraint enforcement
and:
post-processing repair
Constraint enforcement happens before token selection:
logits
│
▼
mask
│
▼
selection
Post-processing happens after generation:
generated output
│
▼
repair / regex / replacement
These are fundamentally different.
A future version of ConstrainCall should ideally move as many correctness rules as possible into the decoding constraint itself rather than relying on post-generation repair.
That gives the system a stronger property:
Invalid outputs are never generated in the first place.
A more general architecture for ConstrainCall would be:
Function Schema
│
▼
Schema Compiler
│
▼
Grammar / Automaton
│
▼
Token Matcher
│
▼
Valid Token Set
│
│
Prompt ──► Tokenizer ──► LLM ──► Logits
│
▼
Logit Processor
│
▼
Token Selection
│
▼
Parser Update
│
└───────► next step
This would transform ConstrainCall from a procedural decoder into a reusable structured-generation engine.
A production-quality C++ implementation could be organized as:
ConstrainCall/
│
├── tokenizer/
│ ├── tokenizer.hpp
│ └── tokenizer.cpp
│
├── model/
│ ├── model.hpp
│ └── model.cpp
│
├── grammar/
│ ├── grammar.hpp
│ ├── parser.hpp
│ ├── dfa.hpp
│ └── pda.hpp
│
├── constraints/
│ ├── constraint.hpp
│ ├── json_constraint.hpp
│ ├── number_constraint.hpp
│ └── string_constraint.hpp
│
├── decoding/
│ ├── decoder.hpp
│ ├── greedy.hpp
│ ├── sampler.hpp
│ └── logits_processor.hpp
│
└── function_calling/
├── schema.hpp
└── function_decoder.hpp
The important abstraction would be:
class Constraint {
public:
virtual bool accepts(int token_id) const = 0;
virtual void advance(int token_id) = 0;
virtual bool accepting() const = 0;
virtual ~Constraint() = default;
};Then the decoder does:
auto logits = model.forward(input_ids);
constraint->apply_mask(logits);
int next_token = argmax(logits);
constraint->advance(next_token);
input_ids.push_back(next_token);This is much closer to the architecture of a general structured-generation engine.
A particularly interesting architecture for ConstrainCall is:
Function names
│
▼
Trie
│
▼
Parameter schema
│
▼
Grammar
│
▼
DFA/PDA
│
▼
Token compatibility
│
▼
Logit mask
For function names:
Trie
is ideal.
For simple lexical constraints:
DFA
is ideal.
For nested JSON:
CFG / PDA
is appropriate.
For semantic constraints:
external validator / semantic predicate
may be required.
This gives a hierarchy:
Token filter
↓
Prefix automaton
↓
DFA
↓
CFG / PDA
↓
Grammar + semantic validation
The general idea explored by ConstrainCall is not unique to this project.
Modern structured-generation systems use similar principles.
For example:
- Outlines uses automata-based structured generation.
- llama.cpp provides GBNF grammar-constrained generation and JSON-schema-to-grammar support.
- XGrammar focuses on efficient CFG-based structured generation and token-mask computation.
- XGrammar 2 extends this work toward dynamic agentic structured generation, including JIT compilation and cross-grammar caching.
- Recent research continues to investigate faster grammar-mask computation and methods that reduce dependence on vocabulary-size-linear work.
This places ConstrainCall within a broader research area:
Structured generation / constrained decoding for autoregressive language models.
The ordinary language model defines a distribution:
where
Constrained decoding introduces a language:
representing all valid outputs.
At every step, we want:
where:
is the set of prefixes that can still be extended into a valid string in
Therefore the decoder does not merely ask:
Is this output valid?
It asks:
Can this prefix still become valid?
This is one of the most important ideas in constrained decoding.
Suppose the valid language is:
L = {"cat", "car"}
The valid prefixes are:
""
"c"
"ca"
"cat"
"car"
At:
"ca"
the valid next characters are:
t
r
At:
"cat"
the sequence is complete.
This is exactly what the function-name decoder does.
Given:
fn_add_numbers
fn_greet
fn_reverse_string
the current prefix determines which continuations are legal.
An LLM may have:
possible tokens.
The constraint can reduce this to:
500 tokens
or:
20 tokens
or:
2 tokens
or even:
1 token
Therefore, constrained decoding can be interpreted as:
The model ranks candidates.
The constraint eliminates impossible candidates.
The decoder does not tell the model:
"Generate this exact answer."
Instead, it tells the model:
"Choose whatever you think is best,
but only among answers that are structurally legal."
Mathematically:
Model:
"What do I prefer?"
Constraint:
"What is allowed?"
Decoder:
"Choose the highest-preference legal option."
This separation is the core idea behind structured generation.
This becomes especially interesting when building LLM agents.
An agent might have:
Filesystem tools
Database tools
HTTP tools
Compiler tools
Search tools
The model should not be allowed to produce arbitrary calls.
Instead:
User request
│
▼
LLM
│
▼
Function selection
│
▼
Schema constraint
│
▼
Valid tool call
│
▼
Tool execution
│
▼
Tool result
│
▼
LLM
For example:
{
"name": "read_file",
"parameters": {
"path": "/tmp/example.txt"
}
}The structured decoder can ensure that the call conforms to the expected interface before the tool receives it.
This is one reason structured generation is particularly important for agentic systems.
ConstrainCall can evolve into a serious research project by investigating questions such as:
These questions lead directly toward current structured-generation research.
A serious benchmark should measure:
Useful metrics include:
tokens/sec
milliseconds/token
constraint overhead/token
mask construction time
grammar compilation time
memory usage
KV-cache memory
A useful comparison is:
Unconstrained generation
vs
Constrained generation
with identical model and hardware.
Attention itself is not simply "quadratic at every generation step".
For a prompt of length (n), processing the full prompt has substantial (O(n^2))-style attention interaction.
During autoregressive decoding, a KV cache allows previously computed keys and values to be reused, so each new token attends to the existing context rather than recomputing the entire prefix from scratch.
Therefore, a precise performance discussion should distinguish:
Prefill
from:
Decode
and should account for:
KV cache
rather than simply stating:
LLM generation = O(N²)
for the entire process.
Structured generation is also relevant to security engineering.
A downstream system often assumes:
parser(input)
receives a valid structure.
If the model can emit arbitrary text, the application may accidentally create ambiguous parsing paths.
Constraining the output reduces the attack surface associated with malformed model-generated structures.
However:
Constrained decoding is not a complete security boundary.
A valid JSON object can still contain:
malicious paths
dangerous SQL
unexpected URLs
invalid business logic
Therefore:
Grammar validation
+
schema validation
+
semantic validation
+
authorization
+
sandboxing
may all be necessary.
The long-term architecture envisioned for ConstrainCall is:
┌───────────────┐
│ User Prompt │
└───────┬───────┘
│
▼
┌───────────────┐
│ Tokenizer │
└───────┬───────┘
│
▼
Token IDs
│
▼
┌───────────────┐
│ Transformer │
└───────┬───────┘
│
▼
Logits
│
▼
┌───────────────────────┐
│ Constraint Engine │
│ │
│ Trie │
│ DFA │
│ CFG/PDA │
│ Schema │
│ Semantic predicates │
└───────────┬───────────┘
│
Valid Token IDs
│
▼
Logit Masking
│
▼
Token Selection
│
▼
State Update
│
└──────────┐
│
▼
Next iteration
ConstrainCall explores a fundamental idea:
Structured generation
The language model provides a probability distribution:
while the constraint engine defines a legal set:
The decoder combines them:
or, for probabilistic constrained sampling:
The key engineering insight is that the model does not need to be trusted to produce the correct structure by itself.
Instead:
Model
↓
"Here are the tokens I prefer."
Constraint engine
↓
"Here are the tokens that are legal."
Decoder
↓
"Choose the best legal token."
That simple interaction transforms unconstrained probabilistic generation into a form of guided formal-language generation.
The current procedural implementation is intentionally specialized. Its next evolutionary step is to replace manually encoded phases with compiled formal constraints:
Function names
↓
Trie
Regex
↓
DFA
JSON Schema
↓
CFG / PDA
Grammar
↓
Token compatibility
Token compatibility
↓
Logit mask
Logit mask
↓
Constrained decoding
That architecture would turn ConstrainCall from a project-specific function-call generator into a general-purpose structured-generation engine.
- Vaswani et al., Attention Is All You Need Paper
The Qwen3-0.6B configuration currently specifies a vocabulary size of 151,936, hidden size 1024, 28 transformer layers, 16 attention heads, and KV caching.
- Outlines structured generation explanation
- llama.cpp GBNF Grammar Guide
- XGrammar paper
- XGrammar 2 paper
- Flexible and Efficient Grammar-Constrained Decoding
XGrammar describes CFG-based constrained decoding and optimization of token-mask computation, while XGrammar 2 extends the approach toward dynamic agentic structured generation.
The LogitsProcessor abstraction is conceptually very close to the interface used by a custom constrained decoder: modify the model's logits before token selection.
A particularly interesting direction is the ongoing optimization of grammar-constrained decoding. Recent work investigates how to reduce the cost of constructing vocabulary masks, including approaches that aim to avoid work proportional to the entire vocabulary at every decoding step.
ConstrainCall is not merely an attempt to make an LLM "output JSON."
The deeper objective is to understand the boundary between:
probabilistic machine learning
and:
deterministic formal computation
The language model provides learned probabilistic behavior.
The tokenizer provides a discrete representation.
The transformer provides contextual inference.
The logits represent the model's preferences.
The constraint engine provides formal validity.
The decoder combines the two.
That boundary is where modern LLM inference systems, tool-calling engines, structured generation systems, and agent runtimes increasingly converge.
ConstrainCall is an exploration of that boundary.