-
Notifications
You must be signed in to change notification settings - Fork 9
Open
Description
Summary
Implement Lottery Ticket Hypothesis (LTH) pruning based on the foundational paper by Frankle & Carlin (2018).
Paper: https://arxiv.org/abs/1803.03635
Title: The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
Background
The Lottery Ticket Hypothesis proposes that dense neural networks contain sparse subnetworks ("winning tickets") that:
- Are typically 10-20% of original network size
- Achieve comparable test accuracy when trained from scratch
- Train faster and achieve higher accuracy than full networks above minimum threshold
- Depend critically on their initialization weights
Implementation Tasks
- Add
LotteryTicketPrunerstruct tosrc/pruning/mod.rs - Implement iterative magnitude pruning with weight rewinding
- Add winning ticket identification algorithm
- Support for rewinding to initialization (IMP - Iterative Magnitude Pruning)
- Add late rewinding variant (rewind to early training, not init)
- Benchmark against existing magnitude pruning
API Design
use aprender::pruning::{LotteryTicketPruner, RewindStrategy};
let pruner = LotteryTicketPruner::builder()
.target_sparsity(0.9) // 90% sparse (10% weights remaining)
.pruning_rounds(10) // Iterative pruning rounds
.rewind_strategy(RewindStrategy::Init) // Rewind to initialization
.build();
let winning_ticket = pruner.find_ticket(&model, &train_data)?;Acceptance Criteria
- LTH pruner achieves <20% parameter count with <1% accuracy drop on MNIST
- Winning tickets train faster than dense baseline
- Property tests for weight rewinding correctness
- Book chapter:
examples/lottery-ticket-pruning.md - Benchmark comparison with magnitude pruning
References
- Original Paper: Frankle & Carbin (2018) - https://arxiv.org/abs/1803.03635
- Stabilizing LTH: https://arxiv.org/abs/1903.01611
- Linear Mode Connectivity: https://arxiv.org/abs/1912.05671
Metadata
Metadata
Assignees
Labels
No labels