bfcl

Specification for a reproducible, provenance-bound multi-lane LLM benchmark suite.

What 200 steps of fully simulated multi-turn tool-use RL do to a 4B policy: every 10th checkpoint scored on BFCL v4, with the pipeline that produced the measurement.

Public benchmark harnesses and reproducible evaluations from PixelSpaceAI

Canonical IR, schema validation, and deterministic delivery for function calls.

A bf16 LoRA fine-tune of Qwen 3.5 4B for function calling on xLAM. v1.0 ships below the BFCL gate with full per-category failure analysis.

Task-conditioned tail reliability for tool-using agents under equivalent interfaces

The first Turkish-native tool-calling benchmark: BFCL-style AST scoring, Turkish difficulty layer, HF leaderboard

Self-hosted LLM evaluation lab: MMLU-Pro / BFCL / IFEval / LiveCodeBench / HumanEval+ runs on Tesla V100 and RTX 3090, llama.cpp serving (incl. Volta sm_70), patched API benchmark clients

Paired metamorphic evaluation of tool-calling robustness under realistic user phrasing, built on BFCL.

Post-training, BFCL evaluation, and single-L4 vLLM serving benchmarks for Qwen3.5-2B.

Independent audit of a fine-tuned LLM tool-calling PoC — BFCL regression decomposition, inference stack risk assessment, and production recommendation for a FinTech client. Qwen-2.5, LoRA, SGLang, H100.

Single-GPU trajectory SFT for Qwen3-1.7B: +30 pp in-domain BFCL task success, with open data, LoRA weights and reproducible evaluation.

Does quantization break tool calling? 10,800 graded agent tasks: it depends which Gemma 4 you compress. The 26B MoE stays flat down to 2 bits, the dense 12B collapses, and no static benchmark reveals it.

Post-train a small instruction-tuned LLM for reliable multi-step tool use

面向工具调用的大模型后训练与泛化评测系统。 End-to-end tool-calling LLM post-training pipeline with data auditing, QLoRA SFT, grounded on-policy DPO, strict evaluation and BFCL.

QLoRA fine-tune of Qwen2.5-1.5B for tool calling on one 8 GiB GPU: +8.5% on BFCL call categories, -55 points on the ability to decline. Refusal data recovers ~49 of them, reproduced across 3 seeds.

OpenEuroLLM snapshot of the Berkeley Function Calling Leaderboard evaluation harness and OLMo evaluation orchestration.

A compact, model-first notation for LLM tool definitions. Re-encodes JSON Schema with a median ~30% input-token reduction and behavior-preserving fallback. Includes the specification, a reference converter, a deployment protocol, and the full evaluation.

Applying Jev to zero-shot tool calling on the Berkeley Function Calling Leaderboard (BFCL)

Fine-tuning, serving and evaluating Qwen3.8-27B on AMD MI300X: LoRA adapters, BFCL v4 and tau3-bench results, and the nine harness bugs found along the way.