Summary
Three of four audio example test files now use the same 3 generic behavioral LLM-judge criteria (aligned via #612 and #655). The fourth — python/examples/test_audio_to_audio.py — still uses the old style.
Current state of test_audio_to_audio.py
criteria=[
"The agent identifies or guesses the voice is male",
"The agent acknowledges the input was audio",
...
]
These are over-specific (voice-gender classification is non-deterministic and not the capability under test).
The generic criteria now used by the other 3 siblings
criteria=[
"The agent's response demonstrates it processed the audio content (e.g. it addresses what was in the audio, attempts to answer the audio question, or acknowledges what it heard)",
"The agent provides a coherent, on-topic response — not an error message, refusal, or unrelated reply",
"The agent's response indicates it received input in a non-text format, or that the question came via audio rather than text (exact phrasing does not matter)",
]
Fix
Either:
- Update
test_audio_to_audio.py to use the same 3 generic criteria as the siblings
- Or document explicitly why this file intentionally differs (in a code comment at the criteria block)
Option 1 is preferred unless this test has a genuinely different behavioral contract.
Context
Surfaced during review of #655 by [fowler] and [hygiene]. Out of scope for #655 (which targeted two specific files). Align in a dedicated PR.
Summary
Three of four audio example test files now use the same 3 generic behavioral LLM-judge criteria (aligned via #612 and #655). The fourth —
python/examples/test_audio_to_audio.py— still uses the old style.Current state of test_audio_to_audio.py
These are over-specific (voice-gender classification is non-deterministic and not the capability under test).
The generic criteria now used by the other 3 siblings
Fix
Either:
test_audio_to_audio.pyto use the same 3 generic criteria as the siblingsOption 1 is preferred unless this test has a genuinely different behavioral contract.
Context
Surfaced during review of #655 by [fowler] and [hygiene]. Out of scope for #655 (which targeted two specific files). Align in a dedicated PR.