Artifact for "Marconi: Prefix Caching for the Era of Hybrid LLMs" [MLSys '25 Outstanding Paper Award, Honorable Mention]
-
Updated
Mar 5, 2025 - Python
#
prefix-caching
Here are 30 public repositories matching this topic...
Serving Qwen3.8-27B-FP8 on a single DGX Spark (GB10): 7.88 to 58.5 tok/s single-stream from decode strategy alone, weights untouched. Speculative decoding and prefix caching benchmarked, plus DFlash 2 — the only Qwen3.8-27B build that can serve it under vLLM.
benchmark nvidia fp8 llm-serving llama-cpp vllm llm-inference qwen speculative-decoding qwen3 prefix-caching gb10 multi-token-prediction dgx-spark dflash2
-
Updated
Aug 19, 2026 - Python
Reproducible llama.cpp CPU inference profiling and a deterministic LLM serving simulator with continuous batching, KV cache, prefix caching, and workload-driven latency analysis.
-
Updated
Jul 19, 2026 - C++
Models Take Notes at Prefill: KV Cache Can Be Editable and Composable (arXiv:2606.17107) — paper, code, results, and interactive companion.
-
Updated
Jun 29, 2026 - Python
Unified execution runtime for LLM and ML programs.
machine-learning deep-learning transformers pytorch agents execution-engine kv-cache dataflow-graph llm generative-ai ai-runtime workflow-optimization program-optimization prefix-caching agent-runtime llm-runtime compiler-runtime cross-call-caching
-
Updated
Jul 11, 2026 - C++
C++ inference runtime for llama.cpp that shares a single document KV-cache prefill across multiple analytical branches via snapshot fan-out. Eliminates redundant GPU compute and dramatically reduces TTFT in DAG-based multi-agent pipelines.
-
Updated
Jun 19, 2026 - C++
A prefix-cache advisor for LLM serving infrastructure that recommends KV-cache capacity and eviction policies from your request traces/logs.
-
Updated
Aug 7, 2026 - Python
Cache-aware router for OpenAI-compatible LLM servers, in Go. Per-worker radix trees route each request to the worker holding its KV prefix. Validated on 4x A100 + vLLM and Apple Silicon + llama.cpp.
-
Updated
May 24, 2026 - Go
LLM inference serving-layer study: a control plane over vLLM (session-affinity routing, EDF+floor scheduling, admission at a measured goodput knee) plus a from-scratch mini-engine (paged KV, hash-chain prefix cache, continuous batching) — with a measured 5x prefix-cache ablation.
-
Updated
Aug 9, 2026 - Python
Context engineering toolkit for LLMs — pack, cache, debug, red-team, and orchestrate context windows. Council of Experts, adversarial testing, immune system, context compiler, drift detection, multi-agent entanglement. TypeScript + Python.
python typescript ai multi-agent rag llm prompt-engineering llm-tools context-window prefix-caching context-engineering adversarial-testing token-budget council-of-experts context-packing
-
Updated
Aug 17, 2026 - Python
A production-grade, native Rust speculative inference engine for Apple Silicon with Metal GPU acceleration and paged attention.
metal gpu-acceleration systems-programming apple-silicon openai-api tensor-parallelism llm-inference speculative-decoding paged-attention continuous-batching prefix-caching disaggregated-serving
-
Updated
Jun 13, 2026 - Rust
Reproducible long-context inference benchmark comparing vLLM, SGLang, and TensorRT-LLM on NVIDIA GB10.
benchmarking cuda nvidia gpu-performance long-context vllm llm-inference tensorrt-llm sglang prefix-caching
-
Updated
Jul 20, 2026 - Python
KV-cache-aware inference router for self-hosted LLM pools. Envoy ext_proc Endpoint Picker + standalone proxy, in Go.
kubernetes golang simulation routing inference load-balancer envoy gateway-api kv-cache llm ext-proc llm-serving vllm llm-inference prefix-caching
-
Updated
Jul 30, 2026 - Go
An interactive simulator of the PagedAttention algorithm for KV cache management in LLMs, inspired by operating system virtual memory.
python memory-management systems-engineering discrete-event-simulation copy-on-write kv-cache vllm llm-inference paged-attention prefix-caching
-
Updated
Jul 15, 2026 - Python
Single-GPU LLM decode research prototype: paged KV cache, Triton attention, CUDA append, scheduling, shared prefixes, and multi-layer transactions.
cuda pytorch triton gpu-kernels kv-cache llm-inference paged-attention flashinfer prefix-caching systems-research inference-runtime request-scheduling
-
Updated
Aug 2, 2026 - Python
Event-driven simulation of concurrent KV-cache prefix sharing, copy-on-write, prefix retention, and scheduler economics inspired by RadixAttention-style LLM serving systems.
python performance-engineering benchmark simulation scheduler inference systems gpu-memory serving copy-on-write event-driven-simulation kv-cache llm prefix-caching radix-attention
-
Updated
Jul 23, 2026
KV-cache warming strategies for LLM serving: top-k adaptive, reserve-headroom, replay, and progressive warming under cold-start after scale-up.
python simulation cold-start autoscaling systems-engineering performance-modeling inference-optimization kv-cache llm-inference cache-warming prefix-caching serving-systems
-
Updated
Jul 26, 2026 - Python
Prefix KV sharing in disaggregated LLM serving: transfer vs recompute vs hotset/LRU replication under decode memory pressure.
python distributed-systems simulation scheduling memory-management systems-engineering performance-modeling inference-optimization kv-cache llm-inference prefix-caching disaggregated-serving
-
Updated
Jul 23, 2026 - Python
Multi-tenant prefix-cache side-channel auditor. Fixed a delimiter-injection collision where an attacker-controlled tenant id containing "|" could shift the cache-key namespace/prefix boundary and defeat tenant_isolated, fully recovering a victim's private value.
-
Updated
Aug 5, 2026 - Python
Benchmarking LLM prefix-cache eviction policies against Tree-Constrained Belady on ShareGPT traces.
radix-tree lru-cache page-replacement-algorithm kv-cache llm-serving vllm llm-inference sglang prefix-caching
-
Updated
May 2, 2026 - HTML
Improve this page
Add a description, image, and links to the prefix-caching topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the prefix-caching topic, visit your repo's landing page and select "manage topics."