feat(rl): Add SFTTrainer for supervised fine-tuning

[gspeter-max]

Summary

Implements #752: Adds SFTTrainer and SFTConfig classes to provide a consistent API for supervised fine-tuning, matching the pattern of the existing RLTrainer.

Changes

Core Implementation

• SFTConfig: Configuration class extending TrainingArguments with 40 fields for model loading, LoRA, batch parameters, training parameters, and optional vLLM integration
• SFTTrainer: Training class extending Trainer with simple cross-entropy loss (no PPO), optional vLLM sampling, and same logging/metrics patterns as RLTrainer

Key Features

• ✅ Consistent API with RLTrainer (same config structure, same initialization)
• ✅ Reuses existing utilities (model loading, logging, LoRA setup)
• ✅ Optional vLLM integration for sample monitoring (not required)
• ✅ Enables easy SFT → RL workflows
• ✅ CLI interface: vf-sft @ path/to/config.toml

Files Modified

• packages/verifiers-rl/verifiers_rl/rl/trainer/config.py - Added SFTConfig
• packages/verifiers-rl/verifiers_rl/rl/trainer/trainer.py - Added SFTTrainer
• packages/verifiers-rl/verifiers_rl/rl/trainer/__init__.py - Added exports
• packages/verifiers-rl/verifiers_rl/__init__.py - Added exports
• verifiers/__init__.py - Added lazy imports
• packages/verifiers-rl/verifiers_rl/scripts/sft.py - Created SFT script
• packages/verifiers-rl/pyproject.toml - Added vf-sft entry point

Usage Example

import verifiers as vf
from datasets import load_dataset

# Load dataset
dataset = load_dataset("willcb/V3-wordle", split="train")

# Create config
config = vf.SFTConfig(
    run_name="wordle-sft",
    max_steps=500,
    learning_rate=2e-5,
    batch_size=512,
    micro_batch_size=8,
)

# Create trainer
trainer = vf.SFTTrainer(
    model="Qwen/Qwen3-4B-Instruct",
    train_dataset=dataset,
    args=config,
)

# Train
trainer.train()

Or via CLI:

vf-sft @ path/to/config.toml

Example TOML config:

model = "Qwen/Qwen3-4B-Instruct"
dataset = "willcb/V3-wordle"

[sft]
run_name = "wordle-sft"
max_steps = 500
learning_rate = 2e-5
batch_size = 512
micro_batch_size = 8
use_lora = true
lora_rank = 8

Design Philosophy

• Similar Structure: SFTTrainer follows the same class structure as RLTrainer
• Simpler Internals: SFT is inherently simpler (no rollouts, no advantages, no orchestrator)
• Optional vLLM: For monitoring sample quality during training (not required)
• Reuses Existing Utilities: Leverages all existing model loading, logging, and utilities

Closes #752

---

Note

Medium Risk
Introduces new training-path code and a new CLI entrypoint, plus touches shared config/LoRA initialization; issues would mainly affect training runs and logging rather than core runtime behavior.

Overview
Adds supervised fine-tuning support alongside the existing RL trainer by introducing SFTConfig (a TrainingArguments-based config that auto-computes gradient_accumulation_steps from batch_size/micro_batch_size and reuses the LoRA setup) and SFTTrainer (a Trainer subclass using standard cross-entropy loss, with optional vLLM-based sample generation and logging).

Exposes the new APIs via verifiers_rl and verifiers lazy imports, adds a new vf-sft TOML-driven CLI script, and updates documentation (docs/reference.md, docs/training.md, and the training skill guide) with usage examples and an SFT → RL workflow.

^{Written by Cursor Bugbot for commit 2475bf7. This will update automatically on new commits. Configure here.}

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.}

[cursor]

vLLM sampling triggers multiple times per global step

High Severity

The training_step method is called once per micro-batch by HuggingFace's Trainer, but self.state.global_step only increments after gradient_accumulation_steps micro-batches. The check self.state.global_step % self.vllm_sample_every_n_steps == 0 evaluates to True for every micro-batch in that gradient accumulation window. With default settings (batch_size=512, micro_batch_size=8, 1 GPU), gradient_accumulation_steps=64, so _generate_and_log_samples() fires 64 times per trigger — each calling blocking asyncio.run() with vLLM API requests and logging duplicate samples.

 

[cursor]

Gradient accumulation steps set after parent initialization

Medium Severity

gradient_accumulation_steps is computed after super().__post_init__() to access world_size, but TrainingArguments.__post_init__() uses gradient_accumulation_steps internally for setup (e.g., DeepSpeed config, logging effective batch size). This means the parent class initializes using the default value of 1 rather than the correct computed value, which can cause misconfiguration in distributed setups or when DeepSpeed is enabled.