A high-performance, universal serving framework for any-to-any models.

An open toolkit and public dataset hub for collecting, sanitizing, analyzing, and visualizing coding agent traces.

Learn the ins and outs of efficiently serving Large Language Models (LLMs). Dive into optimization techniques, including KV caching and Low Rank Adapters (LoRA), and gain hands-on experience with Predibase’s LoRAX framework inference server.

Discrete-event simulation of LLM request routing across multiple serving instances: round-robin, least-load, prefix-aware, and hybrid cache-aware routing with queue-depth threshold sweep.

Discrete-event simulation of SLO-aware autoscaling for multi-instance LLM serving — comparing reactive, predictive, and conservative policies across workload patterns and startup delays.

Simulating KV cache reuse across multi-turn LLM conversations using real ShareGPT traces. Compares sticky, least-load, hybrid, and cost-aware routing strategies for cache locality vs load balance.

Discrete-event simulator for studying throughput, TTFT, and KV-cache transfer tradeoffs in disaggregated LLM serving.

Comparing KV-cache-aware scheduling policies for LLM serving: greedy preempts 49% of requests under memory pressure while memory-first achieves 43% higher effective throughput with zero preemptions.

Benchmarking output length prediction quality and calibration for KV-cache-aware LLM serving scheduling. A noisy predictor with 1.5x safety margin captures 99.7% of oracle gains — calibration matters more than accuracy.

Step-level simulation of chunked prefill under KV cache pressure. Block policy wastes 89.6% of prefill chunks while achieving the same throughput as upfront rejection. Chunk waste rate is the metric standard throughput metrics miss.

Comparing LLM serving under real ShareGPT traces vs synthetic Poisson workloads. Real distributions produce 42x worse P99 TTFT and earlier SLO violations — scheduling policies only differentiate under heavy-tailed real traffic.