[bug] Several docs/DX papercuts: removed client<llm> syntax, missing baml.deep_equals, '::' in test names, reduce() seed typing, api_key_env fallback message, .env not auto-loaded

[tha-hammer]

Product

BAML

Describe the bug

While building a small but complete BAML program (an HTTP server: routing, JSON persistence, an embedded LLM client, baml pack), several things didn't match the CLI's own bundled docs/skill content (baml agent install / .claude/skills/baml-core/SKILL.md) or produced confusing error messages. Filing as one issue since each is small — happy to split into separate issues if that's preferred.

1. client<llm> Name { provider: ..., options: {...} } config-block syntax is removed, but is exactly what the bundled skill doc (and, presumably, older getting-started docs) show as the way to declare a client. The compiler error is genuinely excellent and tells you the fix (client Name = openai.OpenAiClient.new(...)), but the docs a new user is pointed to (baml agent install) still teach the old syntax.

2. baml.deep_equals(a, b) is documented (bundled skill doc) as "the bool form" of assert.equal's structural comparison, but the symbol doesn't exist:

   $ baml describe baml.deep_equals
   no symbol found: deep_equals
   

   == on arrays/classes turns out to already do structural (not reference) comparison, which works as a drop-in replacement, but this isn't documented anywhere I found, and "Classes are reference types" (also in the bundled docs) reads as if == would be identity comparison.

3. A test name containing a literal :: — e.g. test "llm::make_cloze does X" { ... }, which the bundled docs show as the idiomatic way to namespace/group real-LLM-call tests so they can be selected separately from the fast suite (-i "::llm*") — throws at test discovery time, not compile time:

   uncaught throw: testing.InvalidTestName {message: "test name may not contain reserved separator `::`: llm::make_cloze produces exactly one blank and a non-empty answer"}
   

   The actual supported mechanism turned out to be a real testset "llm" { test "..." { ... } } block, selected with -i "*llm*" (a bare "llm::*" didn't match once a namespace prefixes the testset in the canonical id, e.g. root.mymodule::llm::my test). Worth either fixing the docs, or making the ::-in-name case a baml check-time compile error instead of a runtime throw during test discovery.

4. Array.reduce(fn, seed) with a bare integer literal seed (e.g. xs.reduce((acc, x) -> { acc + x.n }, 0)) infers the accumulator's type as the literal type 0, not widened to int, and fails on the first accumulation:

   error[E0001]: mismatched types
     primary: expected `0`, found `int`
   

   Workaround: bind the seed to an explicitly-typed let first (let zero: int = 0; xs.reduce(fn, zero)). This feels like it should either infer int from the closure's own return type, or the literal-type narrowing shouldn't apply to a value used as a mutable accumulator.

5. openai.OpenAiClient.new(api_key_env = "MY_KEY", ...), when MY_KEY is unset in the process environment, panics with a message naming a different, apparently-hardcoded fallback variable name rather than the one actually configured:

   uncaught throw: baml.panics.UserPanic {message: "env var not found: OPENAI_API_KEY"}
   

   even though api_key_env was set to "OPENROUTER_API_KEY". Confusing to debug — I initially assumed api_key_env was being ignored entirely, until I exported the correctly-named var and it started working. If there's an intentional "fall back to OPENAI_API_KEY" behavior for the generic OpenAI-compatible client, it'd help to say so in the panic message (e.g. "env var not found: OPENROUTER_API_KEY (nor fallback OPENAI_API_KEY)").

6. .env is not auto-loaded by baml test / baml run / baml check — env vars referenced via env.NAME or api_key_env must already be exported in the process environment. This may well be intentional (avoids silently reading arbitrary files), but it wasn't obvious from the CLI's own docs, and it bit me specifically when spawning baml run some.entrypoint as a subprocess from Playwright's webServer config, which doesn't inherit a shell that's sourced .env. A one-line doc note ("BAML does not read .env files — export variables yourself, e.g. via direnv, dotenv-cli, or your test runner's env-loading hook") would save the debugging loop I went through.

7. Smaller: baml --help's Commands list is not stable run-to-run on the same installed version/selector — I saw a grep (semantic code search) subcommand listed at one point in this session, then not listed a bit later (auth/feedback appeared instead) while baml --version reported the same baml wrapper 0.2.0 / baml toolchain 0.16.0 throughout. I don't have a clean repro for this one (may be a canary-channel artifact I don't fully understand), but flagging in case it's a real inconsistency in how the CLI resolves its own command table.

Reproduction Steps

Each numbered item above includes its own minimal snippet/command and exact error output. Happy to split any of these into their own issue with a fuller standalone repro if that's more useful for triage.

BAML Version

canary 0.16.0 — wrapper 0.2.0

Language/Framework

Other

Operating System

Linux (Debian 13 "trixie", kernel 6.12.85, x86_64)

[aaronvg]

yep we are in the middle of some big changes, thanks for the report! expect more stability by the end of today / monday