Awesome-Early-Exiting-Neural-Network

A paper list for early-exiting neural networks (ENN).

The idea of early exiting is firsly proposed in ICPR16-BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks.

ENN inserts intermediate classifiers at various network depths, allows the early termination of inference for simple inputs with high-confidence predictions and encourages efficient inference.

ENN is also named multi-exit NN in some papers.

We categorized ENN into four types and collected corresponding papers.

• Architecture
• Training
• Tuning
• Inference

A detailed mindmap of paper collected is illustrated as follow

More items will be added to the repository. Please feel free to suggest other key resources by opening an issue report, submitting a pull request, or dropping me an email @ (boyliu@buaa.edu.cn). 🥳

0. Survey

Survey

Title	Abbreviation	Venue	Year	Materials
Dynamic Neural Network Structure: A Review for Its Theories and Applications	N/A	TNNLS	2024	[PAGE]
Dynamic neural networks: A survey	N/A	TPAMI	2022	[PAGE] [PDF]
Split computing and early exiting for deep learning applications: Survey and research challenges	N/A	CSUR	2022	[PAGE] [PDF]
Adaptive inference through early-exit networks: Design, challenges and directions	N/A	EMDL@MobiSys	2021	[PAGE] [PDF]

1. Architecture

EEN Design

Title	Abbreviation	Venue	Year	Materials
Unlocking the Non-deterministic Computing Power with Memory-Elastic Multi-Exit Neural Networks	MEEdge	WWW	2024	[PAGE]
Resolution Adaptive Networks for Efficient Inference	RANet	CVPR	2020	[PDF]
Multi-Scale Dense Networks for Resource Efficient Image Classification	MSDNet	ICLR	2018	[PDF]

EEN Configuration

Title	Abbreviation	Venue	Year	Materials
SLEXNet: Adaptive Inference Using Slimmable Early Exit Neural Networks	SLEXNet	TECS	2024	[PAGE] [PDF]
HADAS: Hardware-Aware Dynamic Neural Architecture Search for Edge Performance Scaling	HADAS	DATE	2023	[PAGE]
Multi-Exit Semantic Segmentation Networks	MESS	ECCV	2022	[PAGE]
Enabling Low Latency Edge Intelligence based on Multi-exit DNNs in the Wild	ME-DNN	ICDCS	2021	[PAGE]
HAPI: Hardware-Aware Progressive Inference	HAPI	ICCAD	2020	[PAGE]
FlexDNN: Input-Adaptive On-Device Deep Learning for Efficient Mobile Vision	FlexDNN	SEC	2020	[PAGE]

2. Training

KD for Training

Title	Abbreviation	Venue	Year	Materials
Unlocking the Non-deterministic Computing Power with Memory-Elastic Multi-Exit Neural Networks	MEEdge	WWW	2024	[PAGE]
Harmornized Dense Knowledge Distiilation Training for Multi-Exit Architecture	HDKD	AAAI	2021	[PAGE] [PDF]
Distillation-Based Training for Multi-Exit Architectures	N/A	ICCV	2019	[PAGE] [PDF]

Conflict among Exits

Title	Abbreviation	Venue	Year	Materials
Deep Feature Surgery: Towards Accurate and Efficient Multi-Exit Networks	DFS	ECCV	2024	[PDF]
Early Exit with Disentangled Representation and Equiangular Tight Frame	DREE	ACL	2023	[PDF] [CODE]
Meta-GF: Training Dynamic-Depth Neural Networks Harmoniously	Meta-GF	ECCV	2022	[PAGE] [PDF] [CODE]

Train-test Mismatch

Title	Abbreviation	Venue	Year	Materials
Jointly-learned Exit and Inference for a Dynamic Neural Network: JEI-DNN	JEI-DNN	ICLR	2024	[PAGE] [PDF] [CODE]
Boosted Dynamic Neural Networks	BoostNet	AAAI	2023	[PAGE] [PDF] [CODE]
Learning to Weight Samples for Dynamic Early-exiting Networks	L2W-DEN	ECCV	2022	[PAGE] [PDF]

Training Efficiency

Title	Abbreviation	Venue	Year	Materials
Hierarchical Training of Deep Neural Networks Using Early Exiting	N/A	TNNLS	2024	[PAGE]
EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism	EE-LLM	ICML	2024	[PAGE] [PDF] [CODE]

3. Tuning

KD for Tuning

Title	Abbreviation	Venue	Year	Materials
GAML-BERT: Improving BERT Early Exiting by Gradient Aligned Mutual Learning	GAML-BERT	EMNLP	2021	[PAGE] [PDF]
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference	DeeBERT	ACL	2020	[PAGE] [PDF]
FastBERT: a Self-distilling BERT with Adaptive Inference Time	FastBERT	ACL	2020	[PAGE]

Test-time Adaptation

Title	Abbreviation	Venue	Year	Materials
Test-time Specialization of Dynamic Neural Networks	N/A	CVPR Workshop	2024	[PAGE] [PDF]

Tuning Efficiency

Title	Abbreviation	Venue	Year	Materials
EE-Tuning: An Economical yet Scalable Solution for Tuning Early-Exit Large Language Models	N/A	arXiv	2024	[PDF]

4. Inference

Exit Policy

Title	Abbreviation	Venue	Year	Materials
Elastic DNN Inference with Unpredictable Exit in Edge Computing	EINet	TMC	2024	[PAGE]
Scheduling Inputs in Early Exit Neural Networks	N/A	TC	2023	[PAGE]
FrameExit: Conditional Early Exiting for Efficient Video Recognition	FrameExit	CVPR	2021	[PAGE] [PDF]
Empowering Adaptive Early-Exit Inference with Latency Awareness	N/A	AAAI	2021	[PAGE]
EPNet: Learning to Exit with Flexible Multi-Branch Network	EPNet	CIKM	2020	[PAGE]
Shallow-Deep Networks: Understanding and Mitigating Network Overthinking	SDN	ICML	2019	[PAGE] [PDF]
Adaptive Neural Networks for Efficient Inference	N/A	ICML	2017	[PAGE] [PDF]

Exits Prediction

Title	Abbreviation	Venue	Year	Materials
Predictive Exit: Prediction of Fine-Grained Early Exits for Computation- and Energy-Efficient Inference	N/A	AAAI	2023	[PAGE]
FIANCEE: Faster Inference of Adversarial Networks via Conditional Early Exits	FIANCEE	CVPR	2023	[PAGE] [PDF]

Inference Ensemble

Title	Abbreviation	Venue	Year	Materials
Boosted Dynamic Neural Networks	BoostNet	AAAI	2023	[PAGE] [PDF] [CODE]
Towards Anytime Classification in Early-Exit Architectures by Enforcing Conditional Monotonicity	N/A	NIPS	2023	[PAGE] [PDF]

Inference Efficiency

Title	Abbreviation	Venue	Year	Materials
Temporal Decisions: Leveraging Temporal Correlation for Efficient Decisions in Early Exit Neural Networks	N/A		2024	[PAGE] [PDF]
Unlocking the Non-deterministic Computing Power with Memory-Elastic Multi-Exit Neural Networks	MEEdge	WWW	2024	[PAGE]
EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism	EE-LLM	ICML	2024	[PAGE] [PDF] [CODE]
ClassyNet: Class-Aware Early-Exit Neural Networks for Edge Devices	ClassyNet	IOTJ	2023	[PAGE]

40 papers are included

Last Update: Oct 14, 2024 20:47:37