bug(deps): ai-stack/npu provisioning fails — tokenizers>=0.23.1 unsatisfiable with transformers (ResolutionImpossible) #10331

Summary

A fresh ai-stack venv provision fails at pip install -r requirements-ai.txt:

ERROR: Cannot install -r requirements-ai.txt (line 19/20), tokenizers>=0.23.1 and transformers==5.12.1
  because these have conflicting dependencies:
  transformers 5.12.1 depends on tokenizers<=0.23.0,>=0.22.0
  The user requested tokenizers>=0.23.1
ERROR: ResolutionImpossible

failed=1 → provisioning fails → the /slm/setup wizard can't complete (blocks steps 8/9).

Root cause

autobot-infrastructure/shared/docker/ai-stack/requirements-ai.txt pins transformers>=5.10.2 (resolves to the latest, 5.12.1) and tokenizers>=0.23.1. But no available transformers (latest 5.12.1) allows tokenizers > 0.23.0 — every 5.x caps tokenizers<=0.23.0. So tokenizers>=0.23.1 (set by #9424, a dependency bump) is fundamentally unsatisfiable with the current transformers ecosystem. It only surfaces on a clean resolve (rebuilt 3.12 venv); pre-populated venvs masked it.

Same conflicting pair is in autobot-infrastructure/autobot-npu-worker/docker/requirements-npu.txt.

Fix

Constrain tokenizers to the transformers-compatible range: tokenizers>=0.22.0,<0.24 (pip then picks 0.23.0, which transformers 5.12.1 accepts). Verified via pip install --dry-run — resolves cleanly. When a transformers release supports tokenizers>=0.23.1, the floor can be raised again.

Impact

Install-blocking for ai-stack + npu provisioning. The original >=0.23.1 bump (#9424) outran transformers. Discovered during live wizard provisioning. Part of #9919.