Clean, minimal implementations of classic RL algorithms in PyTorch — from DQN to PPO.
Each algorithm is self-contained in a single file with an Agent class and a train() function. Designed for learning and experimentation, not production.
Blog Post: SimpleRL: Building DQN to PPO from Scratch
| Algorithm | Environment | Key Ideas |
|---|---|---|
| DQN | CartPole-v1 | Epsilon-greedy, experience replay, target network |
| REINFORCE | CartPole-v1 | Monte Carlo policy gradient |
| DDPG | Pendulum-v1 | Deterministic policy, OU noise, soft target update |
| TD3 | Pendulum-v1 | Twin Q-networks, delayed policy update, target smoothing |
| SAC | Pendulum-v1 | Maximum entropy RL, automatic temperature tuning |
| PPO | Pendulum-v1 | Clipped surrogate, GAE, truncation-aware bootstrapping |
| DQN | REINFORCE |
|---|---|
 |
 |
| DDPG | TD3 |
|---|---|
 |
 |
| SAC | PPO |
|---|---|
 |
 |
pip install -r requirements.txtRequirements: Python 3.10+, PyTorch 2.0+, Gymnasium, Matplotlib
Train a single algorithm:
python dqn.py
python reinforce.py
python ddpg.py
python td3.py
python sac.py
python ppo.pyTrain all algorithms and generate plots:
python train_all.pyResults are saved to assets/.
simpleRL/
├── common/
│ ├── buffer.py # ReplayBuffer, RolloutBuffer
│ ├── networks.py # MLP, QNetwork, Actor, Critic networks
│ └── utils.py # Seeding, plotting, smoothing
├── dqn.py # Deep Q-Network
├── reinforce.py # REINFORCE (vanilla policy gradient)
├── ddpg.py # Deep Deterministic Policy Gradient
├── td3.py # Twin Delayed DDPG
├── sac.py # Soft Actor-Critic
├── ppo.py # Proximal Policy Optimization
├── train_all.py # Run all algorithms
└── assets/ # Training result plots
DQN — Approximates Q*(s, a) with a neural network. Uses epsilon-greedy for exploration and a separate target network for stable TD targets.
REINFORCE — The simplest policy gradient: collect a full episode, compute discounted returns, and update the policy to increase the probability of high-return actions.
DDPG — Actor-critic for continuous control. The actor learns a deterministic policy μ(s), while the critic learns Q(s, a). Ornstein-Uhlenbeck noise provides exploration.
TD3 — Addresses DDPG's overestimation bias with: (1) twin Q-networks (take the min), (2) delayed policy updates, (3) target policy smoothing via noise.
SAC — Maximizes both return and entropy: π* = argmax E[Σ r + α·H(π)]. Learns the temperature α automatically. Generally the most sample-efficient off-policy method here.
PPO — On-policy method using a clipped surrogate objective to prevent destructive policy updates. Uses GAE for variance reduction and handles environment truncation correctly.
The continuous control algorithms (DDPG, TD3, SAC, PPO) work on any Gymnasium continuous environment. To switch to MuJoCo:
# In any continuous control file, change:
env = gym.make("Pendulum-v1")
# to:
env = gym.make("HalfCheetah-v4")You may need to adjust hyperparameters (more episodes, larger networks) for harder environments.
- Playing Atari with Deep Reinforcement Learning (DQN)
- Simple Statistical Gradient-Following Algorithms (REINFORCE)
- Continuous control with deep reinforcement learning (DDPG)
- Addressing Function Approximation Error in Actor-Critic Methods (TD3)
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep RL (SAC)
- Proximal Policy Optimization Algorithms (PPO)
MIT