veRL: Volcano Engine Reinforcement Learning for LLM

Democratizing Reinforcement Learning for LLMs

Awesome-LLM: a curated list of Large Language Model

Fast and memory-efficient exact attention

OpenR: An Open Source Framework for Advanced Reasoning with Large Language Models

Ongoing research training transformer language models at scale, including: BERT & GPT-2

A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit floating point (FP8) precision on Hopper and Ada GPUs, to provide better performance with lower memory utilizatio…

Example models using DeepSpeed

A schedule language for large model training

CUDA Templates for Linear Algebra Subroutines

CLIP (Contrastive Language-Image Pretraining), Predict the most relevant text snippet given an image

Evals is a framework for evaluating OpenAI models and an open-source registry of benchmarks.

The model pre- and post-processing framework

A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.

Tensors and Dynamic neural networks in Python with strong GPU acceleration

A scalable eigensolver for dense, symmetric (hermitian) matrices (fork of https://gitlab.mpcdf.mpg.de/elpa/elpa.git)

Continuous builder and binary build scripts for pytorch

A lightweight parameter server interface

A high performance and general PS framework for distributed training

Unified Communication X (mailing list - https://elist.ornl.gov/mailman/listinfo/ucx-group)

Lingvo

Python bindings for UCX

DeepSpeed is a deep learning optimization library that makes distributed training easy, efficient, and effective.

Count the MACs / FLOPs of your PyTorch model.

gossip: Efficient Communication Primitives for Multi-GPU Systems

HugeCTR is a high efficiency GPU framework designed for Click-Through-Rate (CTR) estimating training