Release notes from algorithmic-efficiency

V 1.0.1

2026-02-10T04:50:12Z

Summary

Small fixes.

Fixes

Small fixes in documentation
Fix in dataset setup for fineweb edu 10B

V 1.0.0

2026-02-07T05:46:26Z

Summary

Stable release for AlgoPerf benchmark
Added language model workload.
Migrated from 8xV100 to 4xA100 setup.

What's Changed

Improved and streamlined version of the benchmark, which includes a new workload and hardware migration.

Added

[Code, Rules] Migrated from 8xV100 to 4xA100 (40GB) and calibrated the workload runtime budgets for this new hardware (PR).
[Code, Rules] Added LM finewebedu_lm workload with decoder-only language model with Fineweb-Edu 10B dataset (PR).
[Code] Support for changing dropout with the model_fn (PR).

Full Changelog: algoperf-benchmark-0.5.0...algoperf-benchmark-1.0.0

v0.6.0

2025-09-08T13:26:12Z

Summary

A rolling leaderboard to support continuous submissions.
Rule updates aimed at cost efficiency, e.g., removing held-out workloads, limiting to 3 repetition studies, and adjusting workload runtime budgets based on competition results.
JIT-sharding for JAX workloads.
Important bug fixes (e.g., batch norm behavior) and a more flexible API (e.g., a new prepare_for_eval function).

What's Changed

Improved and streamlined version of the benchmark, which includes important bug fixes, API improvements, and benchmark protocol changes following the lessons learned from the first competition.

Added

[Code, Rules] Updated API to allow for prepare_for_eval function (PR/Issue).
[Docs] Document default dropout values for each workload (PR/Issue).
[Docs] Unified versioning policy section (PR).
[Code] Add the ability to change dropout values during training (PR/Issue).

Changed/Removed

[Code, Docs] Rename package to algoperf (PR).
[Code, Docs] Switch to ruff for linting and formatting(PR).
[Code, Rules] Pass train_state to update_params function (PR/Issue).
[Code, Rules] Reduced number of studies from 5 to 3 (PR). See also Section 5.1 in our results paper.
[Code, Rules] Remove held-out workloads from the benchmark (PR). See also Section 5.1 in our results paper.
[Code] Remove sacrebleu dependency (PR).
[Code] Switch to pyproject.toml for package management (PR).
[Code] Update Python version to 3.11 and dependencies accordingly (PR/Issue).
[Rules] Modify the runtime budgets and step hints for each workload (PR/Issue). See also Section 5.1 in our results paper.
[Code] Automatically determine the package version via the latest GitHub tag (PR).
[Code, Docs] Move all algorithms into a dedicated algorithms directory (PR).
[Code] Migrate from pmap to jit in JAX for better performance and scalability (PR).

Fixed

[Code] Batch norm bug (PR/PR/Issue).
[Code] Fix bug of potentially giving a free evaluation to a submission that goes out of max_runtime (PR/Issue).
[Code] Fix that models in the self-tuning ruleset will always be initialized with default dropout (PR/PR/Issue).

Full Changelog: algoperf-benchmark-0.5.0...algoperf-benchmark-0.6.0

v0.5.0

2025-06-24T12:40:12Z

Summary

Finalized variant workload targets.
Fix in random_utils helper function.
For conformer PyTorch Dropout layers set inplace=True.
Clear CUDA cache at begining of each trial for PyTorch.

What's Changed

update speech variants target setting points by @priyakasimbeg in #727
set num_workers for librispeech back to 4 by @priyakasimbeg in #736
[fix] random_utils.py to _signed_to_unsigned by @tfaod in #739
Fix path in helper config for running experiments in bulk. by @priyakasimbeg in #740
Finalize variants targets by @priyakasimbeg in #738
Aiming to Fix Conformer OOM by @pomonam in #710
Lint fixes by @priyakasimbeg in #742
Add warning for PyTorch data loader num_workers flag. by @priyakasimbeg in #726

Full Changelog: algoperf-benchmark-0.1.4...algoperf-benchmark-0.1.5

v0.0.4

2025-06-24T12:39:11Z

Upgrade CUDA version to CUDA 12.1:

Upgrade CUDA version in Dockerfiles that will be used for scoring.
Update Jax and PyTorch package version tags to use local CUDA installation.

Add flag for completely disabling checkpointing.

Note that we will run with checkpointing off at scoring time.

Update Deepspeech and Conformer variant target setting configurations.

Note that variant targets are not final.

Fixed bug in scoring code to take best trial in a study for external-tuning ruleset.

Added instructions for submission.

Changed default number of workers for PyTorch data loaders to 0. Running imagenet workloads with >0 may lead to incorrect eval results see #732.
Update: for speech workloads the pytorch_eval_num_workers flag to submission_runner.py has to be set to >0, to prevent data loader crash in jax code.

v0.0.3

2025-06-24T12:38:35Z

Update technical documentation.

Bug fixes:

Fix workload variant names in Dockerfile.
Fix VIT GLU OOM by reducing batch size.
Fix submission_runner stopping condition.
Fix dropout rng in ViT and WMT.

v0.0.2

2025-06-24T12:37:59Z

Add workload variants.

Add prize qualification logs for external tuning ruleset.
Note: FastMRI trials with dropout are not yet added due to #664.

Add functionality to Docker startup script for self_tuning ruleset.
Add self_tuning ruleset option to script that runs all workloads for scoring.

Data setup fixes.

Fix tests that check training differences in PyTorch and JAX on GPU.

v0.0.1

2025-06-24T12:37:29Z

First release of the AlgoPerf: Training algorithms benchmarking code.