From ec6077f1b6e16f8f6f156bf2b671443d628e7d6e Mon Sep 17 00:00:00 2001 From: kerthcet Date: Sun, 22 Sep 2024 09:51:40 +0800 Subject: [PATCH] Add NanoFlow Signed-off-by: kerthcet --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 0bde027..530feb8 100644 --- a/README.md +++ b/README.md @@ -58,6 +58,7 @@ | **[MInference](https://github.com/microsoft/minference)** | ![Stars](https://img.shields.io/github/stars/microsoft/minference.svg) | ![Release](https://img.shields.io/github/release/microsoft/minference) | ![Contributors](https://img.shields.io/github/contributors/microsoft/minference) | To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy. | | | **[MLC LLM](https://github.com/mlc-ai/mlc-llm)** | ![Stars](https://img.shields.io/github/stars/mlc-ai/mlc-llm.svg) | ![Release](https://img.shields.io/github/release/mlc-ai/mlc-llm) | ![Contributors](https://img.shields.io/github/contributors/mlc-ai/mlc-llm) | Universal LLM Deployment Engine with ML Compilation | | | **[MLServer](https://github.com/SeldonIO/MLServer)** | ![Stars](https://img.shields.io/github/stars/SeldonIO/MLServer.svg) | ![Release](https://img.shields.io/github/release/SeldonIO/MLServer) | ![Contributors](https://img.shields.io/github/contributors/SeldonIO/MLServer) | MLServer aims to provide an easy way to start serving your machine learning models through a REST and gRPC interface, fully compliant with KFServing's V2 Dataplane spec. | | +| **[Nanoflow](https://github.com/efeslab/Nanoflow)** | ![Stars](https://img.shields.io/github/stars/efeslab/nanoflow.svg) | ![Release](https://img.shields.io/github/release/efeslab/nanoflow) | ![Contributors](https://img.shields.io/github/contributors/efeslab/nanoflow) | A throughput-oriented high-performance serving framework for LLMs | | | **[Ollama](https://github.com/ollama/ollama)** | ![Stars](https://img.shields.io/github/stars/ollama/ollama.svg) | ![Release](https://img.shields.io/github/release/ollama/ollama) | ![Contributors](https://img.shields.io/github/contributors/ollama/ollama) | Get up and running with Llama 3, Mistral, Gemma 2, and other large language models. | edge | | **[OpenLLM](https://github.com/bentoml/OpenLLM)** | ![Stars](https://img.shields.io/github/stars/bentoml/openllm.svg) | ![Release](https://img.shields.io/github/release/bentoml/openllm) | ![Contributors](https://img.shields.io/github/contributors/bentoml/openllm) | Operating LLMs in production | | | **[OpenVINO](https://github.com/openvinotoolkit/openvino)** | ![Stars](https://img.shields.io/github/stars/openvinotoolkit/openvino.svg) | ![Release](https://img.shields.io/github/release/openvinotoolkit/openvino) | ![Contributors](https://img.shields.io/github/contributors/openvinotoolkit/openvino) | OpenVINO™ is an open-source toolkit for optimizing and deploying AI inference | |