Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SGLang is a structured generation language designed for large language models (LLMs). It makes your interaction with models faster and more controllable. #519

Open
1 task
irthomasthomas opened this issue Feb 6, 2024 · 0 comments
Labels
AI-Agents Autonomous AI agents using LLMs Algorithms Sorting, Learning or Classifying. All algorithms go here. Automation Automate the things code-generation code generation models and tools like copilot and aider llm Large Language Models llm-function-calling Function Calling with Large Language Models MachineLearning ML Models, Training and Inference ml-inference Running and serving ML models. Papers Research papers RAG Retrieval Augmented Generation for LLMs source-code Code snippets

Comments

@irthomasthomas
Copy link
Owner

TITLE: sgl-project/sglang

SGLang is a structured generation language designed for large language models (LLMs). It makes your interaction with models faster and more controllable.

DESCRIPTION: SGLang

| Blog | Paper |

SGLang is a structured generation language designed for large language models (LLMs). It makes your interaction with LLMs faster and more controllable by co-designing the frontend language and the runtime system.

The core features of SGLang include:

  • A Flexible Front-End Language: This allows for easy programming of LLM applications with multiple chained generation calls, advanced prompting techniques, control flow, multiple modalities, parallelism, and external interaction.
  • A High-Performance Runtime with RadixAttention: This feature significantly accelerates the execution of complex LLM programs by automatic KV cache reuse across multiple calls. It also supports other common techniques like continuous batching and tensor parallelism.

News

  • [2024/02] 🔥 SGLang enables 3x faster JSON decoding with compressed finite state machine (blog).
  • [2024/01] 🔥 SGLang powers the serving of the official LLaVA v1.6 release demo (usage).
  • [2024/01] SGLang provides up to 5x faster inference with RadixAttention (blog).
    URL: https://github.com/sgl-project/sglang/

Suggested labels

{ "label-name": "structured-generation", "description": "Language for large models with advanced control flow and parallelism", "repo": "sgl-project/sglang", "confidence": 91.99 }

@irthomasthomas irthomasthomas added Automation Automate the things llm Large Language Models ml-inference Running and serving ML models. New-Label Choose this option if the existing labels are insufficient to describe the content accurately Papers Research papers source-code Code snippets AI-Agents Autonomous AI agents using LLMs Algorithms Sorting, Learning or Classifying. All algorithms go here. llm-function-calling Function Calling with Large Language Models hosting-services llm model hosting services RAG Retrieval Augmented Generation for LLMs code-generation code generation models and tools like copilot and aider labels Feb 6, 2024
@irthomasthomas irthomasthomas changed the title sgl-project/sglang: SGLang is a structured generation language designed for large language models (LLMs). It makes your interaction with models faster and more controllable. SGLang is a structured generation language designed for large language models (LLMs). It makes your interaction with models faster and more controllable. Feb 13, 2024
@irthomasthomas irthomasthomas added MachineLearning ML Models, Training and Inference and removed New-Label Choose this option if the existing labels are insufficient to describe the content accurately hosting-services llm model hosting services labels Feb 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
AI-Agents Autonomous AI agents using LLMs Algorithms Sorting, Learning or Classifying. All algorithms go here. Automation Automate the things code-generation code generation models and tools like copilot and aider llm Large Language Models llm-function-calling Function Calling with Large Language Models MachineLearning ML Models, Training and Inference ml-inference Running and serving ML models. Papers Research papers RAG Retrieval Augmented Generation for LLMs source-code Code snippets
Projects
None yet
Development

No branches or pull requests

1 participant