Awesome-Early-Exiting-Dynamic-Neural-Network

A paper list for early-exiting dynamic neural networks (EDNN).

The idea of early exiting is firsly proposed in ICPR16-BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks.

EDNN inserts intermediate classifiers at various network depths, allows the early termination of inference for simple inputs with high-confidence predictions and encourages efficient inference.

EDNN is also named multi-exit NN in some papers.

We categorized EDNN into three types and collected corresponding papers.

• Architecture
• Training
• Inference

A detailed mindmap of paper collected is illustrated as follow

More items will be added to the repository. Please feel free to suggest other key resources by opening an issue report, submitting a pull request, or dropping me an email @ (boyliu@buaa.edu.cn). 🥳

0. Survey

Survey

Title	Abbreviation	Venue	Year	Materials
Split computing and early exiting for deep learning applications: Survey and research challenges	N/A	CSUR	2022	[PAGE] [PDF]
Dynamic neural networks: A survey	N/A	TPAMI	2021	[PAGE] [PDF]
Adaptive inference through early-exit networks: Design, challenges and directions	N/A	EMDL@MobiSys	2021	[PAGE] [PDF]

1. Architecture

EE Design

Title	Abbreviation	Venue	Year	Materials
Multi-Scale Dense Networks for Resource Efficient Image Classification	MSDNet	ICLR	2018	[PDF]

EE Configuration

Title	Abbreviation	Venue	Year	Materials
SLEXNet: Adaptive Inference Using Slimmable Early Exit Neural Networks	SLEXNet	TECS	2024	[PAGE] [PDF]
HADAS: Hardware-Aware Dynamic Neural Architecture Search for Edge Performance Scaling	HADAS	DATE	2023	[PAGE]
Enabling Low Latency Edge Intelligence based on Multi-exit DNNs in the Wild	ME-DNN	ICDCS	2021	[PAGE]

2. Training

Exits Augmentation

Title	Abbreviation	Venue	Year	Materials
Harmornized Dense Knowledge Distiilation Training for Multi-Exit Architecture	HDKD		2021	[PAGE] [PDF]
GAML-BERT: Improving BERT Early Exiting by Gradient Aligned Mutual Learning	GAML-BERT	EMNLP	2021	[PAGE] [PDF]
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference	DeeBERT	ACL	2020	[PAGE] [PDF]
Distillation-Based Training for Multi-Exit Architectures	N/A	ICCV	2019	[PAGE] [PDF]

Conflict among Exits

Title	Abbreviation	Venue	Year	Materials
Deep Feature Surgery: Towards Accurate and Efficient Multi-Exit Networks	DFS	ECCV	2024	[PDF]
Meta-GF: Training Dynamic-Depth Neural Networks Harmoniously	Meta-GF	ECCV	2022	[PAGE] [PDF] [CODE]

Train-test Mismatch

Title	Abbreviation	Venue	Year	Materials
Jointly-learned Exit and Inference for a Dynamic Neural Network: JEI-DNN	JEI-DNN	ICLR	2024	[PAGE] [PDF] [CODE]
Boosted Dynamic Neural Networks	BoostNet	AAAI	2023	[PAGE] [PDF] [CODE]
Learning to Weight Samples for Dynamic Early-exiting Networks	L2W-DEN	ECCV	2022	[PAGE] [PDF]

Training Efficiency

Title	Abbreviation	Venue	Year	Materials
Hierarchical Training of Deep Neural Networks Using Early Exiting	N/A	TNNLS	2024	[PAGE]
EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism	EE-LLM	ICML	2024	[PAGE] [PDF] [CODE]

3. Inference

Exit Policy

Title	Abbreviation	Venue	Year	Materials
Elastic DNN Inference with Unpredictable Exit in Edge Computing	EINet	TMC	2024	[PAGE]
FrameExit: Conditional Early Exiting for Efficient Video Recognition	FrameExit	CVPR	2021	[PAGE] [PDF]
Empowering Adaptive Early-Exit Inference with Latency Awareness	N/A	AAAI	2021	[PAGE]
Shallow-Deep Networks: Understanding and Mitigating Network Overthinking	SDN	ICML	2019	[PAGE] [PDF]
Adaptive Neural Networks for Efficient Inference	N/A	ICML	2017	[PAGE] [PDF]

Exits Prediction

Title	Abbreviation	Venue	Year	Materials
Predictive Exit: Prediction of Fine-Grained Early Exits for Computation- and Energy-Efficient Inference	N/A	AAAI	2023	[PAGE]
FIANCEE: Faster Inference of Adversarial Networks via Conditional Early Exits	FIANCEE	CVPR	2023	[PAGE] [PDF]

Inference Efficiency

Title	Abbreviation	Venue	Year	Materials
Temporal Decisions: Leveraging Temporal Correlation for Efficient Decisions in Early Exit Neural Networks	N/A		2024	[PAGE] [PDF]
Unlocking the Non-deterministic Computing Power with Memory-Elastic Multi-Exit Neural Networks	MEEdge	WWW	2024	[PAGE]
EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism	EE-LLM	ICML	2024	[PAGE] [PDF] [CODE]
Scheduling Inputs in Early Exit Neural Networks	N/A	TC	2023	[PAGE]
ClassyNet: Class-Aware Early-Exit Neural Networks for Edge Devices	ClassyNet	IOTJ	2023	[PAGE]

Last Update: Sep 27, 2024 00:21:46