fp8

A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit and 4-bit floating point (FP8 and FP4) precision on Hopper, Ada and Blackwell GPUs, to provide better performance with lower memory utilization in both training and inference.

cuDNN Frontend is NVIDIA's modern, open-source entry point to the cuDNN library and a growing collection of high-performance open-source kernels.

Microsoft Automatic Mixed Precision Library

An innovative library for efficient LLM inference via low-bit quantization

Kernel Library for Large Headdim Attention (64~1024, BF16/FP8/FP4), 1.5x~15x↑ vs PyTorch SDPA.

Flux diffusion model implementation using quantized fp8 matmul & remaining layers use faster half precision accumulate, which is ~2x faster on consumer devices.

Optimized SGLang runtime for Qwen3.8-27B FP8 with DFlash2 and Qwen3.8 Flash-Next NVFP4 with FR-Spec on one NVIDIA RTX PRO 6000 Blackwell 96 GB GPU (SM120): 524K context, HiCache and NIXL.

Serving Qwen3.8-27B-FP8 on a single DGX Spark (GB10): 7.88 to 58.5 tok/s single-stream from decode strategy alone, weights untouched. Speculative decoding and prefix caching benchmarked, plus DFlash 2 — the only Qwen3.8-27B build that can serve it under vLLM.

Measured SM121 compatibility recipe for MiniMax H3 FL2VA on one NVIDIA DGX Spark with vLLM-Omni and online FP8.

Qwen3.8-Flash-Next on 2× RTX 3090 + 128 GB RAM: 1,402 tok/s prefill, 135.2 tok/s decode, and full 256K context.

Lynn 原生 LLM 推理引擎 · W4A8/NVFP4 量化 · 自写 CUDA/Triton kernel · MoE · 投机解码 | Lynn-native LLM inference engine for NVIDIA Blackwell

FP8 Metal compute kernels for Apple Silicon MPS — fixing what PyTorch doesn't support yet. FLUX/SD3.5/ComfyUI on Mac.

LLM inference engine for AMD RDNA4 — Rust + Vulkan compute shaders, gguf & native FP8.

JAX Scalify: end-to-end scaled arithmetics

Benchmarks and notes for running modern LLMs with vLLM on 8x Tesla V100-32GB in 2026.

Automated Triton w8a8 block FP8 kernel tuning tool for vLLM. Auto-detects model architecture, supports Qwen3-Coder-30B-A3B-Instruct-FP8/DeepSeek-V3/custom models, multi-GPU parallel tuning, and generates optimized kernel configs for quantization.

Qwen3.8 Flash Next AWQ INT4 4x3090 v1: 262K, Vision, FP8 QSA KV and CUDA Graphs on 4x RTX 3090

Source-code level analysis of LLM RL training infra: async RL, weight sync, FP8, MoE routing | LLM RL 训练基础设施源码级分析

Docker compose configs for serving Qwen3.8-27B on a DGX Spark (GB10): vLLM+MTP current, SGLang+DSPARK + FP8 rollback stacks

Reproducible recipe: serve abliterated Gemma-4-12B (gemma4_unified) at 50-118 tok/s on no-NVLink Blackwell (SM120) via vLLM nightly + ModelOpt FP8/NVFP4 + MTP spec-decode.