ToolCanary measures what 200 steps of multi-turn tool-use RL in a fully simulated environment do to a 4B policy, and ships the pipeline that produced the measurement.
- 🔑 TL;DR
- 🚀 Quick Start
- 📊 Results
- 📁 Repository Layout
- train/ — configuration and the diff against the reference
- eval/ — the BFCL pipeline and its silent failure modes
- analysis/ — raw artifacts to figures and statistics
- analysis/data/ — the two CSVs every number comes from
- audit/ — contamination audit, with its report
- trustee/ — the training code the run used, verbatim
⚠️ Notes and Limitations- 📚 Citation
- 📄 License
One Qwen3-8B plays task author, tool, user and verifier; Qwen3-4B learns
against it with GRPO. 8xA100-40G, 200 steps, 25,600 multi-turn trajectories.
Every 10th checkpoint was scored on BFCL V4 — 21 points across the run — with
three repeated evaluations at steps 0, 150 and 200 to measure how much the
evaluation moves on its own.
pip install -r requirements.txt
python analysis/make_figures.py --table # reproduces the table below
python analysis/make_figures.py --fig f2 # reproduces the figure below step 0 step 200 rel delta pp/10 steps t repeat s.d.
Abstention required 77.64 73.90 -4.81% -3.74 -0.199 -4.98 0.46 0.47 0.14
Tool call required 85.85 85.01 -0.98% -0.84 -0.015 -1.24 0.14 0.32 0.11
Non-Live 87.67 88.24 0.65% +0.57 +0.015 +0.68 0.40 0.17 0.28
Live 82.36 80.68 -2.04% -1.68 -0.025 -1.35 0.04 0.52 0.00
Multi-Turn 32.96 31.00 -5.94% -1.96 -0.127 -2.51 0.51 1.09 1.47
The two rates of loss differ by 4.45x. Questions whose correct answer is to decline lose 3.74 points; questions that require a call lose 0.84. In item counts the two groups lose 42.0 and 21.0 — two thirds of the loss from one third of the items. The groups also differ in composition, so holding the Live/Non-Live split fixed is the control that matters: within Live the two declines are 4.03 and 1.68, a ratio of 2.40; within Non-Live they are 2.64 against an increase of 0.14, a difference in sign rather than in rate.
Meanwhile the simulated environment reports steady progress: the curriculum reaches its difficulty ceiling by step 79 and holds it for the remaining 121 steps, and none of the three BFCL aggregates follows it.
The trade-off itself is a known failure mode. ToolRL
(arXiv:2504.13958) reports Relevance Detection and Irrelevance Detection as two of the seven columns in its main
table (on BFCL V3), and its numbers show the same direction far more
violently — Qwen2.5-1.5B
loses 64 points of irrelevance detection under PPO cold start while relevance
detection rises to 100%. We did not discover this effect. What this
repository adds is a trajectory instead of two endpoints, a measured noise
floor, and a contamination audit published with the result.
Three environments, because two of them cannot coexist: bfcl-eval pins
numpy<2 in a way that conflicts with the training stack. The evaluation
client speaks plain HTTP and never touches a GPU, so the split costs nothing.
cp .env.example .env && $EDITOR .env
set -a && . ./.env && set +a# (a) analysis only -- all you need to reproduce the table and the figure
conda create -n toolcanary-analysis python=3.12 -y && conda activate toolcanary-analysis
pip install -r requirements.txt
# (b) training and serving
conda create -n toolcanary-train python=3.12 -y && conda activate toolcanary-train
pip install -r trustee/requirements.txt
# (c) evaluation client
conda create -n toolcanary-eval python=3.12 -y && conda activate toolcanary-eval
pip install bfcl-eval==2026.3.23bash train_grpo.sh # 8 GPUs, about 44 hours
bash eval_bfcl.sh "$TOOLCANARY_BASE/models/Qwen3-4B" base # one GPU per jobStep 0 must be scored through this same pipeline. Comparing our checkpoints against a number printed in someone else's paper would fold their setup differences into every delta.
Everything above is recomputed from two CSVs that ship with this repository.
analysis/data/bfcl_by_category.csv holds 27 evaluation passes x 17 BFCL
categories as raw correct/total counts, so any regrouping can be recomputed
without the 1 GB of per-question output. analysis/data/train_metrics.csv
holds 200 steps x 19 metrics.
From BFCL's own output for our step-0 evaluation:
data_non_live.csv Non-Live Overall Acc 87.62% == AST Summary 87.62%
Irrelevance Detection 83.75% (its own column)
data_live.csv Live Overall Acc 82.31% == AST Summary 82.31%
Irrelevance Detection 76.13% (its own column)
data_overall.csv Overall Acc ... Relevance 81.25 | Irrelevance 79.94
BFCL reports irrelevance detection as its own column and includes it in
Overall Acc. What excludes it is the pair Non-Live Overall and
Live Overall, which are AST-only by construction. Our two groups aggregate
BFCL's existing categories: abstention = irrelevance + live_irrelevance
(1124 items, 31% of the single-turn bank); tool call = the ten AST categories
(2501 items).
Every delta is measured against a step-0 checkpoint scored through this pipeline. As a check that the pipeline is comparable, the untrained policy scores 87.67 / 82.36 / 32.96 against 87.77 / 82.16 / 32.12 reported by Tang et al. (2026) — differences of -0.10, +0.20 and +0.84.
Inside the training distribution the policy calls tools less often as a fraction of what it does. Assistant turns that call a tool go from 1.73 to 2.38, but turns that do not go from 1.08 to 3.51, so the share that calls a tool falls from 0.616 to 0.403.
That fall is complete by step 40 and the series is flat afterwards: over steps 41-200 its slope is slightly positive. A line fitted to the whole run reports -0.053 per 100 steps with t = -6.71, but that describes a step, not a trend, so we report the levels instead. Meanwhile the benchmark series takes its level shift between steps 60 and 80 and has no trend after that, though it swings by up to three points between checkpoints. The two are opposite in sign and their windows do not overlap. We report this without a mechanism.
ToolCanary/
├── train_grpo.sh # 8 GPUs: 4 train, 4 serve the simulator
├── eval_bfcl.sh # score one checkpoint on BFCL V4
├── examples/ # the bare invocations both wrappers call
├── train/ # configuration and the diff against the reference
├── eval/ # the BFCL pipeline and its silent failure modes
├── analysis/ # raw artifacts -> two CSVs -> figures and statistics
│ └── data/ # the two CSVs
├── audit/ # contamination audit, with its report
├── assets/ # figures
└── trustee/ # the training code the run used, verbatim
- train/ — configuration, the one change needed to fit on a 40 GB card, and a by-value diff against the reference script: 91 overrides identical, 12 different, one of them substantive and it is a memory-layout change.
- eval/ — the pipeline, and three ways it can report a wrong number without failing.
- analysis/ — the definitions that are ours rather than the benchmark's, and the mistakes that are guarded in code.
- analysis/data/ — CSV schemas and headline numbers.
- audit/ — overlap between the training tool library and the evaluation sets, item by item, and what the audit cannot see.
- trustee/ — what is vendored, why, and what we changed in it (nothing).
- One training run, one base model. Nothing here is a property of the method.
- The mechanism is not established. The task generator floors its
expected-call parameter at 1 in three places, but two code paths can still
yield an empty
expected_tool_calls, and the generated tasks were not persisted. See audit/. - The t statistics assume independent residuals, and they are not. The 21 checkpoints come from one trajectory; on the abstention series the lag-1 residual correlation is +0.54. The endpoint difference against the repeat standard deviation does not depend on that assumption.
- Repeated evaluation of a fixed checkpoint is not reproducible in the multi-turn categories. At step 200 three passes gave 29.38 / 31.38 / 32.25 while Live gave 80.68 three times. We did not establish the source, and the whole of our noise floor comes from these categories.
- tau2-bench and
web_searchwere not run. The first needs GPT-4.1 as the user simulator (Tang et al. 2026, D.3); the second needs a paid search backend and enters none of the three aggregates.
The per-directory READMEs carry the rest: the long-context items above the model's window, what the audit cannot see, why no memory saving is claimed in gigabytes, and the 7% disagreement between the two wall-clock accountings.
@misc{toolcanary2026,
title = {Measuring a Fully Simulated Tool-Use RL Run: Abstention Loss,
Evaluation Noise, and a Direction Mismatch},
author = {Su, Zidong},
year = {2026},
howpublished = {\url{https://github.com/ThomasSu1/ToolCanary}}
}Training runs on verl; evaluation uses
BFCL. The recipe under study is
TRUSTEE, vendored in trustee/ under its own licence:
@article{tang2026trustee,
title = {Democratizing Tool Learning with Environments Fully Simulated by a
Free 8B Language Model},
author = {Tang, Chenming and Huang, Hsiu-Yuan and Liu, Weijie and
Zheng, Junqiang and Yang, Saiyong and Wu, Yunfang},
journal = {arXiv preprint arXiv:2604.17739},
year = {2026}
}Apache-2.0 for everything we wrote; see LICENSE and NOTICE.
trustee/ is third-party code, redistributed unmodified under its own licence
(trustee/LICENSE, trustee/verl/LICENSE, trustee/verl/Notice.txt). Nothing
in it is relicensed; trustee/PROVENANCE.md records what was copied and what
was left out.