DL CookBook Progress

[▓▓▓░░░░░░░░░░░░░░] 10.86% (20/184)

Neural Network Architectures

Basic Architectures (4/4)

• MLP
• CNN
• ResNet
• DenseNet

Recurrent Networks (5/5)

• Vanilla RNN
• Bidirectional RNN
• GRU
• LSTM
• Memory Networks

Auto-Encoders (4/4)

• Sparse
• Variational
• Contractive
• VQ-VAE

Transformers (7/11)

• Vanilla
• Linear
• Sparse
• Gaussian
• Differential
• Universal
• Negative-Weights
• FlashAttention (v1/v2/v3)
• Transformers^2
• Titans
• Atlas

Diffusion Models (0/4)

• DDPM
• DDIM
• Latent Diffusion
• Large Language Diffusion

State Space Models (0/3)

• Mamba
• S4
• H3

Physics-Informed Neural Networks (0/5)

• Neural ODE
• Fourier Neural Operator
• Deep Operator Networks
• Hamiltonian Neural Networks
• Lagrangian Neural Networks

Graph Neural Networks (0/5)

• Graph Convolutional Networks (GCN)
• Graph Attention Networks (GAT)
• Message Passing Neural Networks
• Graph Transformers
• Graph Normalizing Flows

Generative Adversarial Networks (0/5)

• DCGAN
• WGAN
• CycleGAN
• StyleGAN
• BigGAN

Energy-Based Models (0/4)

• Restricted Boltzmann Machines
• Deep Belief Networks
• Deep Energy Networks
• Normalizing Flows

Hyperbolic Networks (0/5)

• H-Attention Network
• H-Graph Network
• H-Normalizing Flows
• H-VAE
• Mixed Geometry Networks

Specialized Architectures (0/8)

• Kolmogorov-Arnold Networks
• Mixture of Experts
• Fuzzy Neural Networks
• Jacobian Fields
• Spiking Neural Networks
• Free-Equivariance Neural Networks
• Neural Causal Models
• Large Concept Models

Reinforcement Learning Algorithms

Value-Based Methods (0/7)

• Deep Q-Networks (DQN)
• Double DQN
• Dueling DQN
• Rainbow DQN
• Categorical 51-Atom DQN (C51)
• Quantile Regression DQN (QR-DQN)
• Implicit Quantile Networks (IQN)

Policy Gradient Methods (0/8)

• Asynchronous Advantage Actor-Critic (A3C)
• Advantage Actor-Critic (A2C)
• Proximal Policy Optimization (PPO)
• Trust Region Policy Optimization (TRPO)
• Deep Deterministic Policy Gradient (DDPG)
• Twin Delayed DDPG (TD3)
• Soft Actor-Critic (SAC)
• Group Related Policy Optimization (GRPO)

Model-Based Methods (0/6)

• World Models
• Imagination-Augmented Agents (I2A)
• Model-Based RL with Model-Free Fine-Tuning (MBMF)
• Model-Based Value Expansion (MVE)
• Dreamer
• PlaNet

Multi-Agent RL (0/4)

• Multi-Agent DDPG (MADDPG)
• Counter-Factual Multi-Agent (COMA)
• Multi-Agent PPO (MAPPO)
• Multi-Agent SAC (MASAC)

Exploration Strategies (0/4)

• Hindsight Experience Replay (HER)
• Random Network Distillation (RND)
• Never Give Up (NGU)
• Go-Explore

Game-Playing and Planning (0/4)

• AlphaZero
• MuZero
• AlphaGo
• Monte Carlo Tree Search (MCTS)

Offline RL (0/4)

• Conservative Q-Learning (CQL)
• Behavior Regularized Actor Critic (BRAC)
• Implicit Q-Learning (IQL)
• Decision Transformer

Meta-RL (0/4)

• RL^2
• MAML for RL
• PEARL
• ProMP

Hierarchical RL (0/4)

• Option-Critic
• Hierarchical Actor-Critic (HAC)
• HIRO
• FUN

Optimization Algorithms

First-Order Methods (0/14)

• Stochastic Gradient Descent (SGD)
• SGD with Momentum
• SGD with Nesterov Momentum
• AdaGrad
• AdaDelta
• RMSProp
• Adam and Variants
• Adam
• AdamW
• NAdam
• RAdam
• AdaMomentum
• AdaBelief
• AdaFactor

Large-Scale/Distributed Training (0/5)

• LARS (Layer-wise Adaptive Rate Scaling)
• LAMB (Layer-wise Adaptive Moments for Batch training)
• Shampoo
• SOAP
• FTRL (Follow The Regularized Leader)

Recent Innovations (0/7)

• Lion
• Prodigy
• Sophia
• Muon
• DeMo
• Adan
• Ranger

Geometric Methods (0/3)

• Reimann SGD
• Natural Gradient Descent
• Mirror Descent

Adaptive Learning Rate Methods (0/4)

• Cyclical Learning Rates
• One Cycle Policy
• Cosine Annealing
• SGDR (Stochastic Gradient Descent with Restarts)

Hybrid Methods (0/5)

• AdaScale
• NovoGrad
• Apollo
• MADGRAD
• SAM (Sharpness-Aware Minimization)

Specialized Optimizers (0/6)

• Lookahead Optimizer
• Rectified Adam
• AGC (Adaptive Gradient Clipping)
• LBFGS (Limited-memory BFGS)
• AdaMax
• AMSGrad

Tokenization

Tokenizing Techniques (0/7)

• Byte Pair Encoding
• Character-level tokenization
• Character-aware tokenization via CNNs
• Raw byte-level end-to-end learned tokenization
• Byte Latent Transformers
• MrT5 Token Deletion gates
• EvaByte

Distributed Training

Distributed Training (0/14)

• MegatronLM
• GPipe
• Alpa
• Tenplex
• DeepSeed
• PipeDream
• ZeRO
• PyTorch DDP
• Horovod
• TensorFlow Distribution Strategy
• Ray Train
• FairScale
• Colossal-AI
• JAX pmap/pjit

Architectures Excluded

• Neural Turing Machines
• Differentiable Neural Computers
• Memory-Augmented Neural Networks
• Beta Variational Auto-encoders
• Convolutional Auto-encoders
• Conditional Auto-encoders
• Vision Transformers
• Performer
• Reformer
• Longformer
• FNet
• Routing Transformers
• Perceiver/Perceiver IO
• SinkFormers