livekit-server — VP9/AV1 Simulcast Fork (SpeakNow)

A minimal fork of livekit/livekit v1.11.0 that makes VP9 and AV1 simulcast work end-to-end. LiveKit treats VP9/AV1 as SVC-only codecs — the simulcast field is ignored for them, and rid-based simulcast (ONE_SPATIAL_LAYER_PER_STREAM, a separate RTP stream per spatial resolution) is unimplemented. This fork adds the missing SFU pieces for both VP9 and AV1.

✅ Status: running in production since 2026-06-13. Single and multiple subscribers, per-viewer resolution adaptation (a weak viewer drops to a lower-res rid while others stay higher), clean up/down layer switches, contentHint text and motion both fine.

This branch	vp9-av1-simulcast-server — the SFU (server) fork
Client branch	vp9-av1-simulcast-client — the client-sdk-js fork
Base	v1.11.0 (8ccad68)
Image	ghcr.io/speaknow06/livekit-server:v1.11.0-vp9simulcast-fix3-dynacast2-av1
Turkish notes	FORK-CHANGES.md

Everything for review, in one place

For livekit/livekit#4594 — both forks plus the write-up live in this repo across two branches:

• VP9_AV1_SIMULCAST_PR_REPORT.md — full, line-by-line write-up of every change (client + server), with file:line, old→new, and why.
• SFU fork — this branch (vp9-av1-simulcast-server) — the server-side changes; Turkish notes in FORK-CHANGES.md.
• Client SDK fork — branch vp9-av1-simulcast-client — our livekit/client-sdk-js v2.19.0 changes (full diff + changed files + prebuilt UMD bundle).
• AV1 works as real simulcast via the same mechanism — one extra line (skipReferenceTS in the AV1 branch), no client change. Verified in production.

Why VP9/AV1 simulcast (vs SVC)?

Per-viewer resolution adaptation at constant fps (weak viewer → lower-res rid; fps stays), plus Android hardware decode of single-spatial rids. VP9 SVC doesn't give this (screen-content/text collapses spatial to one; Android falls back to software decode); VP9 simulcast does, but it's unsupported upstream.

The bug

A VP9-simulcast subscriber works ~60 s (stable on its initial layer), then freezes the instant the allocator first switches layers: the forwarder enters a constant processSourceSwitch loop (~60/s), output timestamps freeze (extNextTS = extLastTS + 1), NACK/PLI storm follows. H.264/VP8 simulcast is unaffected.

Root causes & fixes

1. pkg/sfu/receiver_base.go — all rids collapse to spatial layer 0 (the main one)

spatialLayer := layer
if extPkt.Spatial >= 0 {            // SVC: take spatial from the packet
    spatialLayer = extPkt.Spatial
}

Each VP9 simulcast rid is single-spatial, so its Dependency-Descriptor SpatialId == 0. The override maps every rid to spatialLayer 0; the forwarder can no longer tell rids apart and alternates between their SSRCs on every packet. (H.264/VP8 are unaffected: extPkt.Spatial == -1, so the override never fires — which is why they work and VP9 doesn't.) Fix: apply the override only for SVC (single uptrack → layer == 0).

2. pkg/sfu/forwarder.go — cross-layer timestamp offset never establishes

getRefLayerRTPTimestamp derives the inter-layer RTP-timestamp offset from RTCP Sender Reports; for VP9 simulcast it never becomes valid (tsOffset stays 0), so every switch fails with switch point too far behind. Fix: treat VP9 simulcast like ONE_SPATIAL_LAYER_PER_STREAM_INCOMPLETE_RTCP_SR (set skipReferenceTS) and fall back to the elapsed-time extExpectedTS.

3. pkg/sfu/forwarder.go — wrong selector

VP9 simulcast was routed to the DependencyDescriptor selector (built for one-stream SVC), which reports a switch on nearly every frame. Fix: use the Simulcast selector (keyframe-based, built for separate-stream simulcast — same path as VP8/H.264).

4. pkg/sfu/buffer/dependencydescriptorparser.go — frame drops with frequent keyframes

structureExtFrameNum advances on every structure-bearing keyframe, even when StructureId is unchanged, so reordered/retransmitted frames are dropped as "earlier than current structure" (severe with screen-content / contentHint=text). Fix: advance the drop threshold (new structureChangeExtFrameNum) only on an actual StructureId change; leave ExtKeyFrameNum alone.

Plus (not VP9-specific)

• pkg/service/roommanager.go — ipv6 TURN-URL fix: stock v1.11.0 emits IPv6 into the TURN URL without brackets → client setConfiguration error. Use NodeIP.V4 only.
• Dockerfile — --platform=$BUILDPLATFORM for fast native cross-compile.

Client SDK also needs a fork

VP9 simulcast must be enabled on the client too — upstream livekit-client forces VP9 into SVC. A separate livekit-client fork (2.19.0, branch svc-patch) signals SimulcastCodec.videoLayerMode = ONE_SPATIAL_LAYER_PER_STREAM (without which the server assumes SVC), sets per-encoding scalabilityMode (required on Chrome M113+), and skips the SVC defaults in simulcast mode. App publish config: videoCodec:'vp9', simulcast:true, scalabilityMode:'L1T2'. The server fork (this repo) and the client SDK fork are both required.

Build

docker build --platform linux/amd64 -t ghcr.io/speaknow06/livekit-server:TAG .
docker push ghcr.io/speaknow06/livekit-server:TAG

Every LiveKit upgrade requires re-applying these 5 changes onto the new base. A spatial-drop-first allocator experiment was tried and reverted (stock allocator + the Simulcast selector are sufficient). Reported upstream: livekit/livekit#4594 (PR offered).

---

Below is the upstream LiveKit README.

LiveKit: Real-time video, audio and data for developers

LiveKit is an open source project that provides scalable, multi-user conferencing based on WebRTC. It's designed to provide everything you need to build real-time video audio data capabilities in your applications.

LiveKit's server is written in Go, using the awesome Pion WebRTC implementation.

Features

• Scalable, distributed WebRTC SFU (Selective Forwarding Unit)
• Modern, full-featured client SDKs
• Built for production, supports JWT authentication
• Robust networking and connectivity, UDP/TCP/TURN
• Easy to deploy: single binary, Docker or Kubernetes
• Advanced features including:
  • speaker detection
  • simulcast
  • end-to-end optimizations
  • selective subscription
  • moderation APIs
  • end-to-end encryption
  • SVC codecs (VP9, AV1)
  • webhooks
  • distributed and multi-region

Documentation & Guides

https://docs.livekit.io

Live Demos

• LiveKit Meet (source)
• Spatial Audio (source)
• Livestreaming from OBS Studio (source)
• AI voice assistant using ChatGPT (source)

Ecosystem

• Agents: build real-time multimodal AI applications with programmable backend participants
• Egress: record or multi-stream rooms and export individual tracks
• Ingress: ingest streams from external sources like RTMP, WHIP, HLS, or OBS Studio

SDKs & Tools

Client SDKs

Client SDKs enable your frontend to include interactive, multi-user experiences.

Language	Repo	Declarative UI	Links
JavaScript (TypeScript)	client-sdk-js	React	docs | JS example | React example
Swift (iOS / MacOS)	client-sdk-swift	Swift UI	docs | example
Kotlin (Android)	client-sdk-android	Compose	docs | example | Compose example
Flutter (all platforms)	client-sdk-flutter	native	docs | example
Unity WebGL	client-sdk-unity-web		docs
React Native (beta)	client-sdk-react-native	native
Rust	client-sdk-rust

Server SDKs

Server SDKs enable your backend to generate access tokens, call server APIs, and receive webhooks. In addition, the Go SDK includes client capabilities, enabling you to build automations that behave like end-users.

Language	Repo	Docs
Go	server-sdk-go	docs
JavaScript (TypeScript)	server-sdk-js	docs
Ruby	server-sdk-ruby
Java (Kotlin)	server-sdk-kotlin
Python (community)	python-sdks
PHP (community)	agence104/livekit-server-sdk-php

Tools

• CLI - command line interface & load tester
• Docker image
• Helm charts

Install

Tip

We recommend installing LiveKit CLI along with the server. It lets you access server APIs, create tokens, and generate test traffic.

The following will install LiveKit's media server:

MacOS

brew install livekit

Linux

curl -sSL https://get.livekit.io | bash

Windows

Download the latest release here

Getting Started

Starting LiveKit

Start LiveKit in development mode by running livekit-server --dev. It'll use a placeholder API key/secret pair.

API Key: devkey
API Secret: secret

To customize your setup for production, refer to our deployment docs

Creating access token

A user connecting to a LiveKit room requires an access token. Access tokens (JWT) encode the user's identity and the room permissions they've been granted. You can generate a token with our CLI:

lk token create \
    --api-key devkey --api-secret secret \
    --join --room my-first-room --identity user1 \
    --valid-for 24h

Test with example app

Head over to our example app and enter a generated token to connect to your LiveKit server. This app is built with our React SDK.

Once connected, your video and audio are now being published to your new LiveKit instance!

Simulating a test publisher

lk room join \
    --url ws://localhost:7880 \
    --api-key devkey --api-secret secret \
    --identity bot-user1 \
    --publish-demo \
    my-first-room

This command publishes a looped demo video to a room. Due to how the video clip was encoded (keyframes every 3s), there's a slight delay before the browser has sufficient data to begin rendering frames. This is an artifact of the simulation.

Deployment

Use LiveKit Cloud

LiveKit Cloud is the fastest and most reliable way to run LiveKit. Every project gets free monthly bandwidth and transcoding credits.

Sign up for LiveKit Cloud.

Self-host

Read our deployment docs for more information.

Building from source

Pre-requisites:

• Go 1.23+ is installed
• GOPATH/bin is in your PATH

Then run

git clone https://github.com/livekit/livekit
cd livekit
./bootstrap.sh
mage

Contributing

We welcome your contributions toward improving LiveKit! Please join us on Slack to discuss your ideas and/or PRs.

License

LiveKit server is licensed under Apache License v2.0.

LiveKit Ecosystem
Agents SDKs	Python · Node.js
LiveKit SDKs	Browser · Swift · Android · Flutter · React Native · Rust · Node.js · Python · Unity · Unity (WebGL) · ESP32 · C++
Starter Apps	Python Agent · TypeScript Agent · React App · SwiftUI App · Android App · Flutter App · React Native App · Web Embed
UI Components	React · Android Compose · SwiftUI · Flutter
Server APIs	Node.js · Golang · Ruby · Java/Kotlin · Python · Rust · PHP (community) · .NET (community)
Resources	Docs · Docs MCP Server · CLI · LiveKit Cloud
LiveKit Server OSS	LiveKit server · Egress · Ingress · SIP
Community	Developer Community · Slack · X · YouTube