feat(server): expose POST /v1/tokenize endpoint for client-side token counting

[rafaeldrincon]

Problem

OpenAI-compatible clients have no way to pre-validate prompt size against max_model_len. An oversized prompt is only rejected at generation time (dropped with Input sequence length exceeds max_seq_len) — the client learns nothing until the request fails.

llama.cpp exposes POST /tokenize for exactly this; clients (e.g. Understory, llama-swap) use it to count tokens and chunk prompts before submission. FreeToken already has the machinery (TokenizeManager, count_prompt_tokens for /v1/messages/count_tokens) but no OpenAI-path endpoint.

Solution

POST /v1/tokenize accepting both shapes:

• {"input": "text"} — raw encode via the frontend tokenizer, no chat template (llama.cpp-compatible)
• {"messages": [...]} — rendered through the SAME chat template a real generation would use (via count_prompt_tokens), so the count matches usage.prompt_tokens of a real request

Response: {"tokens": int, "token_ids": list[int]}

Reuses the existing frontend-side TokenizeManager (state.frontend_tokenizer()) — the same read-only instance /v1/messages/count_tokens uses, kept off the generation path.

Tests

tests/server/test_tokenize_endpoint.py — 3 tests, all passing on real hardware (RTX 3080 Ti, Python 3.13):

• raw text → tokens + token_ids
• empty request → 400
• pydantic model defaults

Closes #473

Test environment

ft version: 0.1.2
model: nvidia/Qwen3.6-35B-A3B-NVFP4
gpu: RTX 3080 Ti Laptop (16GB, sm_86)
cuda: 13.1, driver: 580
python: 3.13

[jason-fxz]

@rafaeldrincon

Tested on an H100 with Qwen3.5-2B (--max-seq-len-override 2048), comparing every answer with the usage.prompt_tokens the same server reports for the same prompt. The raw-text path matches /v1/completions usage exactly on three inputs (2, 201 and 40002 tokens). The messages path matches only when the request carries nothing but messages.

This adds a public endpoint, so the shape and the semantics need to be settled before merge.

What diverges today

request	/v1/tokenize	/v1/chat/completions usage.prompt_tokens
2 messages	29	29
2 messages + 1 tool	29	283
2 messages + reasoning_effort: "high"	29	27

TokenizeRequest has extra="allow", so tools, chat_template_kwargs, reasoning_effort and thinking are accepted and ignored, and count_prompt_tokens is always called with tools=None, chat_template_kwargs={}. The agent requests #473 describes all carry tools, so this is the case that has to be right. /v1/messages/count_tokens gets 283 here because it runs the full Anthropic conversion.

Target design

A single TokenizeRequest model with prompt and messages as mutually exclusive fields; both present or both absent is a 400.

{"prompt": "text", "add_special_tokens": true}
{"messages": [...], "tools": [...], "tool_choice": "auto", "chat_template_kwargs": {}, "reasoning_effort": "high", "thinking": {...}}

One response, identical for both paths:

{"tokens": [9707, 1879], "count": 2, "max_model_len": 32768}

• Same request and response fields as vLLM and SGLang. tokens is the id array, count its length, max_model_len the same number /v1/models reports. No length rejection: the client compares against max_model_len, which is the pre-validation feat(server): expose /v1/tokenize endpoint for client-side token validation #473 asks for. Every existing consumer (SillyTavern's remote tokenizer, AIBrix, Agent Router) reads tokens as an array; tokens: int breaks all of them, and a public shape is hard to change afterwards.

• The messages path reuses the chat request pipeline up to tokenization. Convert the request into a ChatCompletionRequest, run it through chat_request_to_genspec, and count the resulting spec with count_prompt_tokens, so tools, tool_choice, chat_template_kwargs, reasoning_effort and thinking behave exactly as on /v1/chat/completions. count_prompt_tokens has to return the ids so this path fills tokens too; today token_ids is always [] on it.

• A list prompt is either rejected or returned per item (tokens a list of lists, count a list, as SGLang does). Flattening loses the per-item counts that chunking needs.

• add_special_tokens applies to the raw path only. The template path never adds them, which is what TokenizeManager already does for list input.

• Mount /v1/tokenize and /tokenize on the same handler. The root path is what vLLM, llama.cpp and their clients call, just alias.