Keras/TF implementation of AdamW, SGDW, NadamW, Warm Restarts, and Learning Rate multipliers
-
Updated
Jan 6, 2022 - Python
#
adamw
Here are 19 public repositories matching this topic...
Implements https://arxiv.org/abs/1711.05101 AdamW optimizer, cosine learning rate scheduler and "Cyclical Learning Rates for Training Neural Networks" https://arxiv.org/abs/1506.01186 for PyTorch framework
scheduler clr optimizer pytorch restarts triangular adamw cosine-annealing cyclical-learning-rate adamw-optimizer
-
Updated
Jul 14, 2019 - Python
Quasi Hyperbolic Rectified DEMON Adam/Amsgrad with AdaMod, Gradient Centralization, Lookahead, iterative averaging and decorrelated Weight Decay
optimizer ranger adam demon lookahead amsgrad adamw radam adamod gradient-centralization qhadam decay-momentum iterate-averaging qhranger adaptive-optimizer hypergradient-descent hd-adam hd-sgd nosadam nostalgic-adam
-
Updated
Sep 23, 2020 - Python
Nadir: Cutting-edge PyTorch optimizers for simplicity & composability! 🔥🚀💻
machine-learning optimization pytorch lion adam-optimizer adamax sgd-optimizer amsgrad adamw radam adamp adabelief
-
Updated
Jun 15, 2024 - Python
[ICLR 2026] LoFT: Low-Rank Adaptation That Behaves Like Full Fine-Tuning
-
Updated
Feb 28, 2026 - Python
SCAO is a sparse, second-order PyTorch optimizer designed as a high-throughput, drop-in replacement for AdamW.
-
Updated
May 22, 2026 - Python
GYRO is an optimizer for deep neural networks that augments Adam with a geometric rotation step applied to the gradient before momentum buffers are updated.
deep-learning optimizer pytorch gradient-descent optimization-algorithms adam adam-optimizer adamw adamw-optimizer
-
Updated
May 4, 2026 - Python
Clean-room GPT-2/GPT-3 implementation: tokenizers, architecture blocks, training loop with AdamW + cosine decay, CLI scripts, inference tools, and pytest suite. Covers OpenWebText-10k & WikiText-103 workflows. Designed as an academic reference for understanding and scaling decoder-only transformers
nlp transformers pytorch gpu-acceleration language-model adamw gpt2 gpt3 cosine-decay decoder-only educational-implementation
-
Updated
Feb 18, 2026 - Python
🕳️ BlackHole Optimizer - A General Relativity approach to optimization. Beats Adam/AdamW via Schwarzschild metric, Hawking radiation, Kerr frame dragging, Penrose process, and Bekenstein-Hawking entropy.
machine-learning deep-learning optimization optimizer pytorch neural-networks gradient-descent general-relativity adam schwarzschild adamw superradiance kaluza-klein penrose-process hawking-radiation physics-inspired kerr-metric blackhole-optimizer bekenstein-hawking
-
Updated
Jul 13, 2026 - Python
reproduce Adam, AdamW, Adafactor optimizors with pytorch, and introduce popular optimizers in the training of the LLMs.
-
Updated
Mar 24, 2025 - Python
Drop-in PyTorch optimizer that beats AdamW with lower variance
-
Updated
Feb 11, 2026 - Python
11th place solution for the U-Tokyo Deep Learning Course MLP Competition (Top 0.8%). High-performance MLP implemented from scratch in NumPy, featuring AdamW, EMA, SWA, and MC Dropout.
deep-learning mlp from-scratch fashion-mnist adamw monte-carlo-dropout implementation-from-scratch competition-solutions numpy-only
-
Updated
Nov 9, 2025 - Python
A repo that contains source code for my blog "Deep Learning Optimizers: A Comprehensive Guide for Beginners (2024)"
-
Updated
Jul 26, 2024 - Python
Super-Convergence on CIFAR10
sophia cifar10 lion second-order-optimization adamw super-convergence weight-decay sharpness-aware-minimization madgrad large-batch-optimization lion-optimizer
-
Updated
Jun 17, 2024 - Python
Finsler-Adam: Asymmetric-metric optimizer with critical scaling gradient clipping for PyTorch. Drop-in AdamW replacement.
machine-learning deep-learning optimizer pytorch navier-stokes gradient-clipping adamw finsler-geometry
-
Updated
Apr 2, 2026 - Python
-
Updated
Sep 11, 2018 - Python
Provide a collection of flexible learning rate schedulers for PyTorch with presets, warmup wrappers, and research-based implementations.
deep-learning scheduler clr sam optimizer transformer muon large-scale-learning scion multinode lr triangular learning-rate-decay adamw cyclical-learning-rate radam learning-rate-warmup gradient-centralization adamp madgrad
-
Updated
Jul 13, 2026 - Python
Improve this page
Add a description, image, and links to the adamw topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the adamw topic, visit your repo's landing page and select "manage topics."