Video/audio attachments silently degrade to text stubs even for models that accept them natively

[0thernet]

What problem are you trying to solve?

eve stages every inbound FilePart to the framework sandbox and re-inlines bytes at model-call time through
shouldInlineSandboxRefAsBytes (packages/eve/src/harness/attachment-staging.ts), a hard two-case whitelist:
image/* ≤3 MiB and application/pdf ≤20 MiB. Everything else — every video, every audio file — hydrates as an
Attached file <path> (<mediaType>) text stub. Silently: the send succeeds, the turn runs, no error and no
stream event mark the downgrade. A user who attaches a screen recording to a Gemini session — a model that
natively understands video — gets a model that can only see the file's path.

The whitelist's conservatism is justified where it guards: a blind widening would regress Anthropic sessions,
because @ai-sdk/anthropic's converter throws UnsupportedFunctionalityError on video/audio file parts —
today's harmless stub would become a failed model call (the same failure class as #497/#399). But the AI SDK is
otherwise ready: ai passes any mediaType through file parts untouched, and @ai-sdk/google's converter turns
any bytes file part into inlineData with no media-type whitelist (verified against the versions eve pins:
@ai-sdk/google 4.0.0, @ai-sdk/anthropic 4.0.0). The provider spec exposes no inline-media capability
introspection (supportedUrls covers URL passthrough only), so the missing piece is a small, honest per-family
rule in eve.

Proposed solution

Model-aware hydration, the same shape as the existing detectPromptCachePath (harness behavior keyed on the
already-resolved model):

export interface ModelMediaSupport {
  /** `video/*` file parts hydrate as inline bytes. */
  readonly video: boolean;
  /** `audio/*` file parts hydrate as inline bytes. */
  readonly audio: boolean;
}

export function detectModelMediaSupport(model: LanguageModel | undefined): ModelMediaSupport;

// signature change, default-preserving (model omitted = today's behavior):
export async function hydrateSandboxAttachments(
  messages: readonly ModelMessage[],
  model?: LanguageModel,
): Promise<ModelMessage[]>;

• The family map is deliberately one entry: "google" → { video: true, audio: true } (gateway ids like
  google/gemini-3-pro, direct google.generative-ai / vertex instances). Every Gemini language model natively
  takes video/audio and the converter inlines them.
• Everything else stays a text stub. OpenAI is deliberately excluded (only the audio-preview/realtime subset
  accepts input_audio, so a family-level rule would fail mainline openai/* calls); Anthropic stays excluded
  because its converter throws.
• Video/audio inline under a 20 MiB cap (matching the Gemini inline-data request budget); the existing image/PDF
  cases and caps are byte-for-byte unchanged, as is the missing-bytes degrade path from fix(eve): degrade missing attachment bytes on resume instead of failing the turn #325.
• One production call-site change in the tool loop, where the resolved model is already in scope two statements
  above hydration.
• Session history is untouched: hydration output was already transient (history stays ref-only), so bytes ride
  one model call and re-read from the staged file each step.
• Failure modes strictly improve: no family ever inlines a part its provider converter would reject.

We have this implemented and tested against main — a DCO-signed patch (+279/−17 across 4 files: the widened
predicate + detector, the call site, unit tests covering every model-classification branch, and four integration
cases on the real stage-then-hydrate round trip, including the stub-by-default and oversized-video paths). On the
patched tree: unit 4384 passed / 1 skipped (419 files), integration 388 passed (50 files),
lint/typecheck/guard:invariants/fmt clean; it still applies cleanly on current main. The full design writeup
lives in our public SDK mirror: [design/proposals/eve-hydrate-model-aware-media.md](https://github.com/zocomput
er/agent-sdk/blob/main/design/proposals/eve-hydrate-model-aware-media.md), with the patch beside it. Happy to
open the PR if the direction fits.

Alternatives considered

• A general per-model capability table — context windows already flow from the gateway catalog; media
  modalities could too. Better long-term; the hard-coded one-entry family map could become the fallback rather than
  the source. We kept the patch minimal and left this as an open question for you.
  • Widening the whitelist unconditionally — regresses Anthropic sessions from a harmless stub to a thrown
    UnsupportedFunctionalityError mid-turn.
  • Probing supportedUrls — it only describes URL passthrough, not inline-media acceptance; there is no
    capability introspection for this in the provider spec today.
  • Emitting a downgrade signal instead of fixing hydration — observability for stubbed parts is worth having
    regardless (today's silent downgrade is how this gap went unnoticed), but it doesn't get the video to the model