Skip to content

chore(audio-examples): align python/examples/test_audio_to_audio.py criteria with generic set #682

Description

@drewdrewthis

Summary

Three of four audio example test files now use the same 3 generic behavioral LLM-judge criteria (aligned via #612 and #655). The fourth — python/examples/test_audio_to_audio.py — still uses the old style.

Current state of test_audio_to_audio.py

criteria=[
    "The agent identifies or guesses the voice is male",
    "The agent acknowledges the input was audio",
    ...
]

These are over-specific (voice-gender classification is non-deterministic and not the capability under test).

The generic criteria now used by the other 3 siblings

criteria=[
    "The agent's response demonstrates it processed the audio content (e.g. it addresses what was in the audio, attempts to answer the audio question, or acknowledges what it heard)",
    "The agent provides a coherent, on-topic response — not an error message, refusal, or unrelated reply",
    "The agent's response indicates it received input in a non-text format, or that the question came via audio rather than text (exact phrasing does not matter)",
]

Fix

Either:

  1. Update test_audio_to_audio.py to use the same 3 generic criteria as the siblings
  2. Or document explicitly why this file intentionally differs (in a code comment at the criteria block)

Option 1 is preferred unless this test has a genuinely different behavioral contract.

Context

Surfaced during review of #655 by [fowler] and [hygiene]. Out of scope for #655 (which targeted two specific files). Align in a dedicated PR.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P3 - lowLow priority, nice to haverefactorCode restructuring, no behavior changetech-debt

    Type

    No type

    Projects

    Status
    Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions