SGLang is a structured generation language designed for large language models (LLMs). It makes your interaction with models faster and more controllable.

[irthomasthomas]

• sgl-project/sglang: SGLang is a structured generation language designed for large language models (LLMs). It makes your interaction with models faster and more controllable.

TITLE: sgl-project/sglang

SGLang is a structured generation language designed for large language models (LLMs). It makes your interaction with models faster and more controllable.

DESCRIPTION: SGLang

| Blog | Paper |

SGLang is a structured generation language designed for large language models (LLMs). It makes your interaction with LLMs faster and more controllable by co-designing the frontend language and the runtime system.

The core features of SGLang include:

• A Flexible Front-End Language: This allows for easy programming of LLM applications with multiple chained generation calls, advanced prompting techniques, control flow, multiple modalities, parallelism, and external interaction.
• A High-Performance Runtime with RadixAttention: This feature significantly accelerates the execution of complex LLM programs by automatic KV cache reuse across multiple calls. It also supports other common techniques like continuous batching and tensor parallelism.

News

• [2024/02] 🔥 SGLang enables 3x faster JSON decoding with compressed finite state machine (blog).
• [2024/01] 🔥 SGLang powers the serving of the official LLaVA v1.6 release demo (usage).
• [2024/01] SGLang provides up to 5x faster inference with RadixAttention (blog).
  URL: https://github.com/sgl-project/sglang/

Suggested labels

{ "label-name": "structured-generation", "description": "Language for large models with advanced control flow and parallelism", "repo": "sgl-project/sglang", "confidence": 91.99 }