fix(openai_mllm_python): migrate realtime client to the GA interface

[leepokai]

This extension is currently broken against the live API. Verified against
api.openai.com: connecting with the header main sends today returns
The Realtime Beta API is no longer supported. Please use /v1/realtime for the GA API.

Summary

openai_mllm_python speaks the beta shape of the Realtime API on three axes:

• realtime/connection.py sends the OpenAI-Beta: realtime=v1 header, which the GA interface requires callers to drop
• realtime/struct.py recognises only beta server event names — GA renamed these to response.output_audio.delta, response.output_text.delta and response.output_audio_transcript.delta
• session.update is serialised flat, where GA requires session.type: "realtime", audio nested under session.audio.input / session.audio.output, output_modalities and max_output_tokens

This migrates the client to GA, corrects two model defects, and adds the extension's first tests.

Approach

The beta/GA difference is confined to a serialisation boundary in realtime/struct.py, which imports only the standard library.

Inbound. The EventType enum values change to the GA strings. The dispatch in parse_server_message compares against those members, so it follows automatically.

Outbound. A session_update_to_ga_dict() mapper converts the flat internal params into the GA nested payload; to_json() dispatches to it, so connection.py's send_request() path is untouched.

extension.py matches on dataclasses rather than event strings, so the rename does not touch its 20 match/case arms.

The mapper is an allowlist: a field with no GA mapping raises. GA rejects the entire session.update when it carries an unknown key, which would take instructions, tools, VAD, transcription and voice down with it, so a field added to SessionUpdateParams without a mapping has to fail loudly rather than silently invalidate the session.

Audio format

audio.*.format takes an object ({"type": "audio/pcm", "rate": 24000}), not the string "pcm16" the published reference still documents — the GA implementation answers the string form with Invalid type for 'session.audio.input.format': expected an object, but got a string instead.

The extension leaves the format unset today and to_json drops None, so it inherits the GA default of PCM16 24 kHz mono, matching its own sample_rate: 24000. Behaviour is unchanged; the mapping only applies when a format is explicitly set.

Other GA gaps fixed

• assistant conversation items used the beta content-part type text, which GA renamed to output_text (Literal["output_text", "output_audio"] in openai-python). Every assistant item was rejected, and since SessionCreated replays stored history on each reconnect, the model came back with only the user half of its context
• the OpenAI branch authenticated with aiohttp.BasicAuth, sending Authorization: Basic. GA documents a Bearer token, which is also what stepfun_mllm_python sends for the same protocol. Note this one is alignment, not a fix — Basic is still accepted by the service today (see verification below)
• case ErrorMessage() only logged. GA leaves the socket open when it rejects an event, so nothing surfaced the failure: the graph waited on mllm_server_session_ready against a connection that looked healthy. Errors now route through the base class's send_mllm_error

Model configuration

• the config default was gpt-4o, which is not a realtime model — any deployment not overriding the property failed to connect. Defaults now track gpt-realtime-2.1
• new optional reasoning_effort property exposing that model's reasoning budget. It defaults to empty, which omits the field, so existing deployments are unaffected

Scope

Limited to openai_mllm_python.

azure_mllm_python, glm_mllm_python and stepfun_mllm_python carry their own copies of realtime/ — stepfun's struct.py was byte-identical to this one — but they target independent endpoints (AZURE_AI_FOUNDRY_BASE_URI with its own api_version, wss://open.bigmodel.cn, wss://api.stepfun.com) and none of them sends the beta header. They cloned the beta event schema but are separate services on their own release schedules, so propagating GA event names to them would break them.

Relationship to #2167

#2167 performs the same protocol migration using the same mapper approach and has been open without review since 2026-05-21. This supersedes it by adding the model defaults, the reasoning_effort control, the GA gaps listed above, and a test suite. Happy to close this in favour of #2167 plus a follow-up if maintainers prefer — the overlap is the protocol layer only.

Tests

52 tests, the extension's first. They target the wire boundary, so they need no credentials, no network and no built runtime.

Two deliberate deviations from how other extensions lay out tests/: there is no tests/__init__.py and there is a tests/pytest.ini. Making tests a subpackage causes pytest to import openai_mllm_python/__init__.py, which imports addon, which imports the native ten_runtime — so those suites only run where a runtime is installed. Keeping the extension package out of the collection tree lets this suite run anywhere, which is the point of confining the wire format to a layer that depends on nothing.

Source-level checks use ast rather than substring matching, so a comment naming a symbol cannot decide the result.

tests/test_connection.py         .......                  (7)
tests/test_ga_events.py          ...................      (19)
tests/test_ga_session_update.py  ..........................(26)
52 passed

Verified via ./tests/bin/start, the path task test-extension invokes. black --line-length 80 --check is clean — note that the files in this extension satisfy 80, while 79 and 88 both report drift against unmodified files.

Live verification

Run against api.openai.com with model=gpt-realtime-2.1, driving the actual
to_json() from this branch:

Check	Result
OpenAI-Beta: realtime=v1 (what main sends)	error — The Realtime Beta API is no longer supported. Please use /v1/realtime for the GA API.
header absent (this branch)	session.created
this branch's session.update	session.updated
audio.input.format as object	accepted
audio.input.format as string "pcm16"	error — Invalid type for 'session.audio.input.format': expected an object, but got a string instead.
flat beta session.update	error — Missing required parameter: 'session.type'.
Authorization: Basic (what main sends)	session.created — still accepted

Two things worth calling out. The published reference still documents
format as a string; the service rejects it, so the object mapping in
_GA_AUDIO_FORMATS is required and now has empirical backing rather than
resting on the docs. And Basic auth is still accepted, so the switch to
Bearer is alignment with the documented scheme and the sibling clients, not
a fix for a live failure — reviewers should weigh it on those terms.

Not covered

• No audio round-trip. Verification covers the handshake and session
  negotiation, not streaming PCM in and audio out. The event renames are
  covered offline against captured payload shapes rather than live deltas.
• temperature and max_tokens are declared on the config, in manifest.json and in property.json, but are read nowhere in extension.py — pre-existing dead config. Left alone: temperature may not be accepted on a reasoning model, so wiring it up needs its own verification. The mapper now raises rather than leaking it, so the gap is visible instead of silent.
• Shipped graphs pin their own model. voice-assistant-realtime, voice-assistant-companion and demo all set model explicitly, so this default bump reaches only new graphs. Worth a separate pass — gpt-realtime enters deprecation on 2026-08-28.
• ResponseCreateParams is not migrated. Nothing sets it today (send_client_create_response sends a bare ResponseCreate()), so it is unmigrated rather than wrong; to_json now raises on the first caller that populates it.
• The README documents a retired Azure model and a preview URL shape. Pre-existing and wider than this change.