This project has been created as part of the 42 curriculum by afomin.

Call Me Maybe

Description

A function calling system that translates natural language prompts into structured JSON function calls using a small 0.6B parameter language model (Qwen3-0.6B). The key challenge is achieving near-perfect reliability with a model that would otherwise produce valid JSON only ~30% of the time. This is solved through constrained decoding — guiding the model's output token-by-token to guarantee valid structure.

Given a prompt like "What is the sum of 2 and 3?", the system produces:

{
    "prompt": "What is the sum of 2 and 3?",
    "name": "fn_add_numbers",
    "parameters": {"a": 2.0, "b": 3.0}
}

Instructions

Installation

# Clone the repository
git clone <your_repo_url>
cd call-me-maybe

# Copy llm_sdk into the project root
cp -r /path/to/llm_sdk ./llm_sdk

# Install dependencies
make install
# or manually:
uv sync

Running

# Default paths (data/input/ → data/output/)
make run

# Custom paths
uv run python -m src \
    --functions_definition data/input/functions_definition.json \
    --input data/input/function_calling_tests.json \
    --output data/output/function_calling_results.json

Other commands

make debug    # Run with pdb debugger
make lint     # Run flake8 and mypy
make clean    # Remove caches

Algorithm Explanation

Constrained Decoding

The system generates output token-by-token, restricting which tokens are allowed at each step:

1. Function selection: Only tokens that are valid continuations of known function names are allowed. The model picks the most likely token from this restricted set, progressively narrowing candidates until one function remains — then the rest of the name is appended without any LLM call.

2. Argument generation — options-based approach: For string arguments, complete phrases are extracted from the prompt and tokenized. At each step, only the first token of each remaining option is allowed. The model picks one — all options not starting with that token are eliminated. This repeats until one option remains, which is then appended in full without further LLM calls.

   For "Greet shrek": options are ["Greet", "shrek"] → model picks "shrek" as a complete unit, not character by character.

3. Typed arguments:

   • number / integer: extracted from the prompt via options, validated and converted to float
   • boolean: restricted to true / false only
   • regex: resolved deterministically via keyword matching (see below)

4. Regex arguments: The prompt is scanned for known keywords (vowels, digits, spaces, etc.) and mapped to the corresponding regex pattern ([aeiouAEIOU], \d+, \s+, etc.). For literal word replacements (e.g. "cat"), the word is extracted directly from the prompt.

Chat Template

The model is prompted using Qwen's native tool-calling format:

<|im_start|>system
You are provided with function signatures within <tools></tools> XML tags:
<tools>
{"name": "fn_add_numbers", ...}
</tools>
For each function call, return a json object within <tool_call></tool_call> tags:
<tool_call>
{"name": <function-name>, "arguments": <args-json-object>}
</tool_call>
<|im_end|>
<|im_start|>user
What is the sum of 2 and 3?
<|im_end|>
<|im_start|>assistant
<tool_call>
{"name": "fn_add_numbers", "arguments": {"a": 2, "b": 3}}

This matches Qwen3's training format, significantly improving argument extraction quality compared to a custom JSON format.

Design Decisions

• Pydantic everywhere: All classes (Encoder, LLM, Function, CallMeMaybe) use pydantic BaseModel for validation and type safety. Private attributes use PrivateAttr() to bypass validation on large internal structures.
• Separate Encoder class: Tokenization is decoupled from LLM inference. Custom BPE encoder uses a trie structure for O(n) encoding instead of O(n²) substring search.
• Options-based argument extraction: Instead of character-level masking, string arguments are generated by selecting from complete tokenized phrases extracted from the prompt. This prevents early stopping on uncommon words and guarantees the output is always a real value from the input.
• Short instruction during arg generation: When generating arguments, only the current function's schema is kept in the instruction context (not all functions). This reduces token count and speeds up logit computation — which scales O(n²) with context length.
• Keyword-based regex resolution: Rather than asking the LLM to generate regex patterns (unreliable for 0.6B models), patterns are resolved deterministically from prompt keywords.
• Native tool-call format: Using Qwen's documented tool-calling template instead of a custom JSON format leverages the model's existing training.

Performance Analysis

• Accuracy: 90%+ on the test suite — both on provided examples and unseen prompts with new function sets.
• JSON validity: 100% — constrained decoding guarantees parseable output on every run.
• Speed: ~3.5 minutes for 11 prompts on CPU. Depends on prompt and function complexity.
• Key bottleneck: get_logits_from_input_ids is called once per token — O(n²) attention over the full context. Mitigated by switching to a single-function instruction during argument generation.

Challenges Faced

• Early stopping on uncommon words: Character-level constrained decoding caused the model to stop mid-word on rare names ("Greet shrek" → "shr"). Solved by switching to options-based generation — extracting complete phrases from the prompt and letting the model select among them token-prefix by token-prefix until one candidate remains.
• LLM generating regex: Early attempts to have the LLM generate regex patterns directly failed — the 0.6B model would mix keyword descriptions with pattern syntax (e.g. numbers\d+). Replaced with deterministic keyword mapping.
• Pydantic and large structures: Pydantic validation on the trie dictionary (150k tokens) during __init__ was slow. Solved by moving trie and vocab to PrivateAttr(), bypassing validation entirely. This cut initialization time significantly.
• Context length and speed: Including all function definitions in every logit call was slow. Solved by switching to a short instruction (single function schema) during argument generation.
• Chat template: Without Qwen's native tool-calling template, the model had no context for what format to continue. Adding the correct <|im_start|> / <|im_end|> structure and <tool_call> tags dramatically improved output quality.

Testing Strategy

• Ran the provided function_calling_tests.json and manually verified each output against expected values.
• Tested with unseen prompts and different function sets (compound interest, SQL queries, file reading, template formatting) to verify generalization.
• Verified JSON validity by parsing all outputs with json.loads().
• Tested CLI arguments with custom paths to ensure defaults and overrides work correctly.
• Tested error handling: missing input files, malformed JSON in input, empty function definitions.

Example Usage

# Run with default test files
uv run python -m src

# Run with custom files
uv run python -m src \
    --functions_definition data/input/functions_definition.json \
    --input data/input/function_calling_tests.json \
    --output data/output/function_calling_results.json

Example output (data/output/function_calling_results.json):

[
    {
        "prompt": "What is the sum of 2 and 3?",
        "name": "fn_add_numbers",
        "parameters": {"a": 2.0, "b": 3.0}
    },
    {
        "prompt": "Greet john",
        "name": "fn_greet",
        "parameters": {"name": "john"}
    }
]

Resources

• Qwen3 documentation
• Qwen tool calling format
• Pydantic documentation
• BPE tokenization explained

AI Usage

Claude (Anthropic) was used throughout this project for:

• Debugging pydantic initialization patterns (PrivateAttr, super().__init__())
• Identifying the chat template format for Qwen3 tool calling
• Keyword-mapping approach for regex pattern resolution
• Code review and architecture cleanup