A high-throughput and memory-efficient inference and serving engine for LLMs

SGLang is a fast serving framework for large language models and vision language models.

TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.

NumPy aware dynamic Python compiler using LLVM

NumPy & SciPy for GPU

Supercharge Your LLM with the Fastest KV Cache Layer

An interactive NVIDIA-GPU process viewer and beyond, the one-stop solution for GPU process management.

A Python framework for accelerated simulation, data generation and spatial computing.

A flexible framework of neural networks for deep learning

A PyTorch Library for Accelerating 3D Deep Learning Research

GPU cluster manager for optimized AI model deployment

Jittor is a high-performance deep learning framework based on JIT compiling and meta-operators.

A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit and 4-bit floating point (FP8 and FP4) precision on Hopper, Ada and Blackwell GPUs, to provide better performance with lower memory utilization in both training and inference.

PyTorch/TorchScript/FX compiler for NVIDIA GPUs using TensorRT

Minkowski Engine is an auto-diff neural network library for high-dimensional sparse tensors

PyTorch native quantization and sparsity for training and inference

Self-hosted, local only NVR and AI Computer Vision software. With features such as object detection, motion detection, face recognition and more, it gives you the power to keep an eye on your home, office or any other place you want to monitor.

Pytorch domain library for recommendation systems

RamaLama is an open-source developer tool that simplifies the local serving of AI models from any source and facilitates their use for inference in production, all through the familiar language of containers.

label-smooth, amsoftmax, partial-fc, focal-loss, triplet-loss, lovasz-softmax. Maybe useful