Distributed Keras Engine, Make Keras faster with only one line of code.

Chimera: bidirectional pipeline parallelism for efficiently training large-scale models.

sensAI: ConvNets Decomposition via Class Parallelism for Fast Inference on Live Data

SHADE: Enable Fundamental Cacheability for Distributed Deep Learning Training

🚨 Prediction of the Resource Consumption of Distributed Deep Learning Systems

Ok-Topk is a scheme for distributed training with sparse gradients. Ok-Topk integrates a novel sparse allreduce algorithm (less than 6k communication volume which is asymptotically optimal) with the decentralized parallel Stochastic Gradient Descent (SGD) optimizer, and its convergence is proved theoretically and empirically.

Decentralized Asynchronous Training on Heterogeneous Devices

Eager-SGD is a decentralized asynchronous SGD. It utilizes novel partial collectives operations to accumulate the gradients across all the processes.

WAGMA-SGD is a decentralized asynchronous SGD based on wait-avoiding group model averaging. The synchronization is relaxed by making the collectives externally-triggerable, namely, a collective can be initiated without requiring that all the processes enter it. It partially reduces the data within non-overlapping groups of process, improving the…

Distributed deep learning framework based on pytorch/numba/nccl and zeromq.

Collection of resources for automatic deployment of distributed deep learning jobs on a Kubernetes cluster

Horovod Tutorial for Pytorch using NVIDIA-Docker.

An implementation of a distributed ResNet model for classifying CIFAR-10 and MNIST datasets.

A foundational repository for setting up distributed training jobs using Kubeflow and PyTorch FSDP.

Auto-Tuned Scheduler Prototype for Heterogeneous GPU Clusters

A blockchain based neural architecture search project.

Implemented training strategies to help improve bottlenecks and to improve the training speed while maintaining the quality of our GANs.

Simultaneous Multi-Party Learning Framework