feat(client): add self-hosted orchestrator session support

[a1anfan]

Adds self-hosted orchestrator session support to @elevenlabs/client, matching the Python SDK's OnPremInitiationData (named orchestratorConfig here to avoid the on* callback-handler naming convention). Passing orchestrator to Conversation.startSession (or createConnection) routes the conversation WebSocket to a self-hosted orchestrator. The orchestrator is stateless, so the client sends an enclave_setup_config message (agent config, tools, optional Bedrock inference profile and post-call webhooks) before the standard conversation_initiation_client_data message; after that the protocol matches the cloud path. Orchestrator sessions are websocket-only.

Two connection details differ from the cloud path: the configured URL is used verbatim, and no WebSocket subprotocols are requested, because the orchestrator accepts the socket without subprotocol negotiation and browsers fail a connection whose requested subprotocol is not echoed back.

const conversation = await Conversation.startSession({
  orchestrator: {
    url: "wss://your-host/sagemaker/convai/conversation",
    agentConfig: config.agent_config,
    tools: config.tools_config,
  },
});

The React SDK picks this up through its session config passthrough with no changes; the widget and React Native can follow separately.

Testing

• unit tests
• stub orchestrator e2e (Node WebSocket)
• browser e2e and manual testing against a deployed orchestrator

---

Note

Medium Risk
New connection path and session bootstrap protocol for private deployments; misconfiguration could fail sessions or expose webhook HMAC secrets in client-visible config (documented as test-only).

Overview
Adds experimental self-hosted orchestrator sessions to @elevenlabs/client via an orchestrator option on Conversation.startSession / createConnection, aligned with the Python SDK’s on-prem initiation flow.

Sessions use websocket only: the client opens the configured URL as-is (no convai / bearer subprotocols) and, on connect, sends an enclave_setup_config message (agent config, overrides, tools, optional Bedrock profile and post-call webhooks) before the usual conversation_initiation_client_data. createConnection rejects mixing orchestrator with WebRTC, signedUrl, conversationToken, or authorization.

uploadFile is blocked for orchestrator sessions so file uploads are not sent to the ElevenLabs cloud API.

^{Reviewed by Cursor Bugbot for commit c5c55d2. Bugbot is set up for automated code reviews on this repo. Configure here.}

[a1anfan]

cursor review

[Copilot]

Pull request overview

Adds an on-premises conversation session path to @elevenlabs/client by allowing callers to provide onPremConfig to route the session WebSocket to a self-hosted orchestrator and send an enclave_setup_config payload before the normal initiation message.

Changes:

• Add OnPremConfig / OnPremSessionConfig types and export them from the client entrypoint.
• Update WebSocket session bootstrap to support on-prem URLs and omit WebSocket subprotocol negotiation on that path.
• Add helpers + tests for constructing and sending the enclave_setup_config wire message, plus WebSocketConnection on-prem tests.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
packages/client/src/utils/WebSocketConnection.ts	Adds on-prem WebSocket URL handling and sends enclave_setup_config before initiation; omits subprotocols on-prem.
packages/client/src/utils/WebSocketConnection.test.ts	Adds on-prem connection tests (URL usage, message ordering).
packages/client/src/utils/onPrem.ts	Introduces constructEnclaveSetupConfig mapping to wire format.
packages/client/src/utils/onPrem.test.ts	Unit tests for on-prem wire mapping and null defaults.
packages/client/src/utils/ConnectionFactory.ts	Forces on-prem sessions onto WebSocket and rejects WebRTC.
packages/client/src/utils/BaseConnection.ts	Adds exported on-prem config types to the session config union.
packages/client/src/index.ts	Re-exports new on-prem types.
.changeset/thick-clouds-hang.md	Declares a minor release for on-prem session support.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit be3b753. Configure here.}

[a1anfan]

cursor review

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit ac53cdb. Configure here.}

[kraenhansen]

I would like to have a conversation about your thoughts around this API. Specially to see if any of these on-prem specifics could be easily expressed in the existing configuration options or if we should instead take these "top-level" and discriminate based on a separate boolean or string-literal field. I also don't particularly like that the onPremConfig starts with on* as this is a pattern we use for callback handlers already.

In any case these should be marked with the @experimental tsdoc tag to allow us changing these without the need for a major version bump.

[lagercat]

For us to figure out how to generate these automatically with OpenAPI so we don't have to keep updating it with new parameters.

[a1anfan]

cursor review

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 05417b5. Configure here.}

[a1anfan]

@kraenhansen, added @experimental tags (513365a), and went ahead and renamed "onPremConfig" to "orchestratorConfig" (05417b5) so you can see it concretely — the docs define the orchestrator as the gateway exposing the conversation WebSocket, and the wire types here already use ClientToOrchestratorEvent, so it felt like the natural term. also renamed the webhook type to PostCallWebhookConfig to match the Python SDK. happy to change it if you have a different direction in mind

re folding the fields into existing options: these fields change the connection itself rather than the conversation config since the URL is used verbatim, no convai subprotocol can be requested (i.e. browsers fail the connection if the server doesn't echo it, and self-hosted orchestrators don't negotiate), and an extra enclave_setup_config message precedes the normal initiation. the grouped object keeps those co-dependent fields together and mirrors the Python SDK's OnPremInitiationData. however lmk you feel strongly about a top-level discriminator shape!

[kraenhansen]

I'd like some more time to dig into this in detail to see how this could potentially reuse part of the or replace the existing "top-level" config. Wondering if this should be encoded as a new connectionType?

[kraenhansen]

Small naming note: since everything in OrchestratorConfig is inherently config, the Config suffix on the field itself (orchestratorConfig) reads as redundant — the type name already carries that. We drop the suffix on the field elsewhere, e.g. connectionDelay: DelayConfig (BaseConnection.ts:65), so orchestrator: OrchestratorConfig would be more consistent with that pattern.

[kraenhansen]

Could you share more about the expected shape of entries in toolsConfigList? It's currently typed as Record<string, unknown>[] with no reference type, and the only shape hint is the test fixture ({ type: "webhook" }) — is this meant to match the platform's tool-export format exactly, and is there a schema we could reference or link to in the doc comment?

Same naming note as orchestratorConfig: Config and List both read as type-related rather than name-related (the [] and object shape already say "list of configs"), so tools: Record<string, unknown>[] would drop the redundancy — same logic would apply to overrideAgentConfigList.

[kraenhansen]

One more: what happens if a caller sets both overrides.agent.prompt (replaces the prompt) and orchestratorConfig.promptKnowledgeBase (appends to it)? Since these act on the same underlying prompt via two different mechanisms/messages (conversation_initiation_client_data vs enclave_setup_config), it'd be good to spell out the interaction in the doc comments.

We have a precedent for a similar conflict with textOnly vs overrides.conversation.textOnly (BaseConversation.ts:97-113), though note that one only warns and picks a winner rather than failing — worth deciding whether prompt/promptKnowledgeBase should follow that same soft pattern, or fail fast client-side, since "replace" vs "append" feels like a more genuine logical conflict than two conflicting booleans.

[a1anfan]

cursor review

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit c5c55d2. Configure here.}

[a1anfan]

thanks for the thorough review @kraenhansen! all points addressed:

• naming: field is now orchestrator, plus tools and agentConfigOverrides (64b1adc). open to a better name for the last one.
• tools shape: the platform's tool export format, same as the Python SDK's tools_config_list. the orchestrator validates entries server-side and rejects the session on mismatch. now in the doc comment; no public schema yet.
• promptKnowledgeBase vs overrides.agent.prompt: checked the orchestrator source, they compose rather than conflict: one replaces the knowledge-base section, the other sets the prompt text. documented (64b1adc), so no warn/fail needed.
• hmacSecret: visible to the end user by design, like the rest of the session config. it signs deliveries to the customer's own webhook receiver, and whoever can set the secret can also set the URL, so nothing privileged leaks. doc note added (c5c55d2).
• client SDK vs server SDK: the browser is the real target: a page where a customer's evaluators call an agent inside their own VPC, with no ElevenLabs cloud in the path. Python parity is only because that was the sole reference implementation.
• client_secrets: agreed that's the right end state. the orchestrator is WebSocket-only today, no HTTP surface or storage layer to mint against, which is exactly why everything is @experimental, so we can migrate without a major bump.
• connectionType (your earlier question): kept it as a session option rather than a new connectionType because orchestrator sessions still use the websocket transport; what changes is the endpoint and the setup handshake. presence-of-exactly-one of agentId / signedUrl / conversationToken / orchestrator also matches the existing union.

also fixed two Bugbot findings on #922: file upload UI is hard-off for orchestrator sessions so file bytes can't leave the customer network (3c47153), and a language attribute honored against a declared supported set now behaves like the picker (fe7e674). both verified on the wire against a stub orchestrator.