Distillation Applications

• Large Language Models
• Foundation Models
• Self-Supervised Learning
• Diffusion Models
• Visual Recognition

---

Large Language Models

• TinyBERT: Distilling BERT for Natural Language Understanding, arXiv 2019, 🔗
• Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, arXiv 2019, 🔗
• Patient Knowledge Distillation for BERT Model Compression, arXiv 2019, 🔗
• Well-read students learn better: On the importance of pre-training compact models, arXiv 2019, 🔗
• Xtremedistil: Multi-stage distillation for massive multilingual models, arXiv 2020, 🔗
• MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices, ACL 2020, 🔗
• MINILM: deep self-attention distillation for task-agnostic compression of pre-trained transformers, NeurIPS 2020, 🔗
• xtremedistiltransformers: Task transfer for task-agnostic distillation, arXiv 2021, 🔗
• Explanations from Large Language Models Make Small Reasoners Better, arXiv 2022, 🔗
• BERT Learns to Teach: Knowledge Distillation with Meta Learning, ACL 2022, 🔗
• Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes, arXiv 2023, 🔗
• Large language models are reasoning teachers, ACL 2023, 🔗
• Less is More: Task-aware Layer-wise Distillation for Language Model Compression, ICML 2023, 🔗
• MCC-KD: Multi-CoT Consistent Knowledge Distillation, EMNLP 2023, 🔗
• Teaching Small Language Models to Reason, ACL 2023, 🔗
• Symbolic Chain-of-Thought Distillation: Small Models Can Also “Think” Step-by-Step, ACL 2023, 🔗
• Gkd: Generalized knowledge distillation for auto-regressive sequence models, arXiv 2023, 🔗
• SCOTT: Self-consistent chain-of-thought distillation, NeurIPS 2023, 🔗
• Dialogue Chain-of-Thought Distillation for Commonsense-aware Conversational Agents, EMNLP 2023, 🔗
• MiniLLM: Knowledge Distillation of Large Language Models, ICLR 2024, 🔗
• On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes, ICLR 2024, 🔗
• PaD: Program-aided Distillation Specializes Large Models in Reasoning, ACL 2024, 🔗
• Turning dust into gold: Distilling complex reasoning capabilities from llms by leveraging negative data, AAAI 2024, 🔗
• PC-LoRA: Low-Rank Adaptation for Progressive Model Compression with Knowledge Distillation, arXiv 2024, 🔗
• Reverse Thinking Makes LLMs Stronger Reasoners, arXiv 2024, 🔗
• UniCoTT: A Unified Framework for Structural Chain-of-Thought Distillation, ICLR 2025, 🔗
• MiniPLM: Knowledge Distillation for Pre-Training Language Models, ICLR 2025, 🔗

Foundation Models

Prominent Foundation Models

• CLIP : Learning Transferable Visual Models From Natural Language Supervision, PMLR 2021, 🔗
• GPT
  • GPT-1: Improving Language Understanding by Generative Pre-Training, OpenAI 2018, 🔗
  • Gpt-3: Language Models are Few-Shot Learners, NeurIPS 2020, 🔗
  • Gpt-4: A Review on Advancements and Opportunities in Natural Language Processing, arXiv 2023, 🔗
• DALL-E: Zero-Shot Text-to-Image Generation, ICML 2021, 🔗
• SAM: Segment Anything, ICCV 2023, 🔗
• ViT: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, arXiv 2020, 🔗
• DeiT: Training data-efficient image transformers & distillation through attention, ICML 2021, 🔗
• DINO: Emerging Properties in Self-Supervised Vision Transformers, ICCV 2021, 🔗
• DETR: End-to-End Object Detection with Transformers, ECCV 2020, 🔗
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows, ICCV 2021, 🔗

Vision-Language Models (VLMs)

• Simple Open-Vocabulary Object Detection with Vision Transformers, ECCV 2022, 🔗

• Learning to Detect and Segment for Open Vocabulary Object Detection, CVPR 2023, 🔗

• PracticalDG: Perturbation Distillation on Vision-Language Models for Hybrid Domain Generalization, CVPR 2024, 🔗

• CLIP

  • Open-Vocabulary Object Detection

    • Open-vocabulary Object Detection via Vision and Language Knowledge Distillation, ICLR 2021, 🔗
    • Detecting Twenty-thousand Classes using Image-level Supervision, ECCV 2022, 🔗
    • Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model, CVPR 2022, 🔗
    • Open-Vocabulary DETR with Conditional Matching, ECCV 2022, 🔗
    • F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models, arXiv 2022, 🔗
    • Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detection, NeurIPS 2022, 🔗
    • PromptDet: Towards Open-vocabulary Detection using Uncurated Images, ECCV 2022, 🔗
    • DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detection, NeurIPS 2022, 🔗
    • Exploiting Unlabeled Data with Vision and Language Models for Object Detection, ECCV 2022, 🔗
    • Learning Object-Language Alignments for Open-Vocabulary Object Detection, arXiv 2022, 🔗
    • Open-Vocabulary One-Stage Detection with Hierarchical Visual-Language Knowledge Distillation, CVPR 2022, 🔗
    • Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labeling, CVPR 2022, 🔗
    • Open Vocabulary Object Detection with Pseudo Bounding-Box Labels, ECCV 2022, 🔗
    • Fine-grained Visual-Text Prompt-Driven Self-Training for Open-Vocabulary Object Detection, TNNLS 2022, 🔗
    • Zero-shot Object Detection Through Vision-Language Embedding Alignment, ICDM 2022, 🔗
    • Aligning Bag of Regions for Open-Vocabulary Object Detection, CVPR 2023, 🔗
    • Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformers, CVPR 2023, 🔗
    • Object-Aware Distillation Pyramid for Open-Vocabulary Object Detection, CVPR 2023, 🔗

  • Open-Vocabulary Semantic Segmentation

    • Language-driven Semantic Segmentation, ICLR 2021, 🔗
    • Semantic Segmentation In-the-Wild Without Seeing Any Segmentation Examples, arXiv 2021, 🔗
    • Extract Free Dense Labels from CLIP, ECCV 2022, 🔗
    • Image Segmentation Using Text and Image Prompts, CVPR 2022, 🔗
    • Scaling Open-Vocabulary Image Segmentation with Image-Level Labels, ECCV 2022, 🔗
    • Decoupling Zero-Shot Semantic Segmentation, CVPR 2022, 🔗
    • A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model, ECCV 2022, 🔗
    • ReCo: Retrieve and Co-segment for Zero-shot Transfer, NeurIPS 2022, 🔗
    • Open-vocabulary Semantic Segmentation with Frozen Vision-Language Models, arXiv 2022, 🔗
    • Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP, CVPR 2023, 🔗
    • ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation, CVPR 2023, 🔗
    • FreeSeg: Unified, Universal and Open-Vocabulary Image Segmentation, CVPR 2023, 🔗
    • Exploring Open-Vocabulary Semantic Segmentation from CLIP Vision Encoder Distillation Only, ICCV 2023, 🔗
    • CLIP-DIY: CLIP Dense Inference Yields Open-Vocabulary Semantic Segmentation For-Free, WACV 2024, 🔗

  • Open-Vocabulary Part Segmentation

    • Understanding Multi-Granularity for Open-Vocabulary Part Segmentation, NeurIPS 2025, 🔗

  • Open-Vocabulary Customization

    • Open-Vocabulary Customization from CLIP via Data-Free Knowledge Distillation, ICLR 2025, 🔗

  • Weakly Supervised Semantic Segmentation

    • Cross Language Image Matching for Weakly Supervised Semantic Segmentation, CVPR 2022, 🔗
    • CLIP is Also an Efficient Segmenter: A Text-Driven Approach for Weakly Supervised Semantic Segmentation, CVPR 2023, 🔗
    • Learning Multi-Modal Class-Specific Tokens for Weakly Supervised Dense Object Localization, CVPR 2023, 🔗
    • Question-Answer Cross Language Image Matching for Weakly Supervised Semantic Segmentation, ACM MM 2023, 🔗
    • Prompting classes: Exploring the Power of Prompt Class Learning in Weakly Supervised Semantic Segmentation, WACV 2024, 🔗
    • Frozen CLIP: A Strong Backbone for Weakly Supervised Semantic Segmentation, CVPR 2024, 🔗
    • DIAL: Dense Image-text ALignment for Weakly Supervised Semantic Segmentation, ECCV 2024, 🔗
    • WeakCLIP: Adapting CLIP for Weakly-Supervised Semantic Segmentation, IJCV 2025, 🔗

  • Prompt Learning

    • Learning to Prompt for Vision-Language Models, IJCV 2022, 🔗
    • Conditional Prompt Learning for Vision-Language Models, CVPR 2022, 🔗
    • DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting, CVPR 2022, 🔗
    • Domain Adaptation via Prompt Learning, T-NNLS 2023, 🔗
    • PromptKD: Unsupervised Prompt Distillation for Vision-Language Models, CVPR 2024, 🔗

  • Generation and Editing

    • StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery, ICCV 2021, 🔗
    • StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generators, ACM SIGGRAPH 2022, 🔗
    • Zero-Shot Text-Guided Object Generation with Dream Fields, CVPR 2022, 🔗
    • CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields, CVPR 2022, 🔗
    • Text2Mesh: Text-Driven Neural Stylization for Meshes, CVPR 2022, 🔗
    • Decomposing NeRF for Editing via Feature Field Distillation, NeurIPS 2022, 🔗
    • AvatarCLIP: Zero-Shot Text-Driven Generation and Animation of 3D Avatars, ACM Trans. Graph 2022, 🔗
    • CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation, CVPR 2022, 🔗
    • CLIP2StyleGAN: Unsupervised Extraction of StyleGAN Edit Directions, ACM SIGGRAPH 2022, 🔗
    • Text and Image Guided 3D Avatar Generation and Manipulation, WACV 2023, 🔗
    • Local 3D Editing via 3D Distillation of CLIP Knowledge, CVPR 2023, 🔗

  • Person Re-Identification

    • Distilling CLIP with Dual Guidance for Learning Discriminative Human Body Shape Representation, CVPR 2024, 🔗

  • Zero-Shot Human-Object Interaction (HOI) Detection

    • Exploiting CLIP for Zero-shot HOI Detection Requires Knowledge Distillation at Multiple Levels, WACV 2024, 🔗

  • Open-Vocabulary Out-of-Distribution Classification

    • Distilling Large Vision-Language Model with Out-of-Distribution Generalizability, ICCV 2023, 🔗

  • Domain Generalization

    • A Sentence Speaks a Thousand Images: Domain Generalization through Distilling CLIP with Language Guidance, ICCV 2023, 🔗

  • Video-Language Retrieval

    • CLIPPING: Distilling CLIP-Based Models with a Student Base for Video-Language Retrieval, CVPR 2023, 🔗

  • Affordance Detection

    • What does CLIP know about peeling a banana?, CVPR 2024, 🔗

  • Video Highlight Detection

    • Unleash the Potential of CLIP for Video Highlight Detection, CVPR 2024, 🔗

  • Monocular Depth Estimation

    • CaBins: CLIP-based Adaptive Bins for Monocular Depth Estimation, CVPR 2024, 🔗

  • Multiple Tasks (Image Classification, Object Detection, Semantic Segmentation, Instance Segmentation, Image-Text Retrieval)

    • MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretraining, CVPR 2023, 🔗
    • TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritance, ICCV 2023, 🔗
    • CLIP-KD: An Empirical Study of CLIP Model Distillation, CVPR 2024, 🔗
    • CLIP-Embed-KD: Computationally Efficient Knowledge Distillation Using Embeddings as Teachers, HPEC 2024, 🔗

Segment Anything (SAM)

• Road Network Graph Extraction

  • Segment Anything Model for Road Network Graph Extraction, CVPR 2024, 🔗

• Polyp Segmentation

  • PP-SAM: Perturbed Prompts for Robust Adaptation of Segment Anything Model for Polyp Segmentation, CVPR 2024, 🔗

• Image Restoration

  • Distilling Semantic Priors from SAM to Efficient Image Restoration Models, CVPR 2024, 🔗

• Weakly Supervised Semantic Segmentation

  • Segment Anything Model (SAM) Enhanced Pseudo Labels for Weakly Supervised Semantic Segmentation, arXiv 2023, 🔗
  • An Alternative to WSSS? An Empirical Study of the Segment Anything Model (SAM) on Weakly-Supervised Semantic Segmentation Problems, arXiv 2023, 🔗
  • Segment Anything is A Good Pseudo-label Generator for Weakly Supervised Semantic Segmentation, arXiv 2023, 🔗
  • Weakly-Supervised Semantic Segmentation with Image-Level Labels: from Traditional Models to Foundation Models, arXiv 2023, 🔗
  • From SAM to CAMs: Exploring Segment Anything Model for Weakly Supervised Semantic Segmentation, CVPR 2024, 🔗

Multi-model Applications

• SAM + CLIP

  • Foundation Model Assisted Weakly Supervised Semantic Segmentation, WACV 2023, 🔗
  • SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understanding, CVPR 2024, 🔗
  • Test-Time Adaptation with SaLIP: A Cascade of SAM and CLIP for Zero shot Medical Image Segmentation, CVPR 2024, 🔗

• DINO + CLIP

  • LERF: Language Embedded Radiance Fields, ICCV 2023, 🔗
  • CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentation, ECCV 2025, 🔗

• DETR + CLIP

  • T-Rex2: Towards Generic Object Detection via Text-Visual Prompt Synergy, ECCV 2024, 🔗

• SAM + GDINO

  • Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks, arXiv 2024, 🔗
  • Prompting Foundational Models for Omni-supervised Instance Segmentation, CVPR 2024, 🔗

• Others

  • FeatureNeRF: Learning Generalizable NeRFs by Distilling Foundation Models, ICCV 2023, 🔗
  • DIME-FM: DIstilling Multimodal and Efficient Foundation Models, ICCV 2023, 🔗
  • AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into One, CVPR 2024, 🔗
  • Theia: Distilling Diverse Vision Foundation Models for Robot Learning, arXiv 2024, 🔗
  • Strategies to Leverage Foundational Model Knowledge in Object Affordance Grounding, CVPR 2024, 🔗
  • Enrich Distill and Fuse: Generalized Few-Shot Semantic Segmentation in Remote Sensing Leveraging Foundation Model's Assistance, CVPR 2024, 🔗
  • Exploring the Benefits of Vision Foundation Models for Unsupervised Domain Adaptation, CVPR 2024, 🔗
  • Distilling Knowledge from Multiple Foundation Models for Zero-shot Image classification, PlosOne 2024, 🔗

Vision Transformers

• Vision Transformer (ViT)

  • Supervised Masked Knowledge Distillation for Few-Shot Transformers, CVPR 2023, 🔗
  • Masked Autoencoders Enable Efficient Knowledge Distillers, CVPR 2023, 🔗
  • Distilling Self-Supervised Vision Transformers for Weakly-Supervised Few-Shot Classification & Segmentation, CVPR 2023, 🔗
  • TinyMIM: An Empirical Study of Distilling MIM Pre-trained Models, CVPR 2023, 🔗
  • Incrementer: Transformer for Class-Incremental Semantic Segmentation With Knowledge Distillation Focusing on Old Class, CVPR 2023, 🔗
  • Cumulative Spatial Knowledge Distillation for Vision Transformers, ICCV 2023, 🔗
  • Feature-based Knowledge Distillation for Vision Transformers, CVPR 2024, 🔗

• Distillation with No Labels (DINO)

  • Unsupervised Object Discovery

    • Localizing Objects with Self-Supervised Transformers and no Labels, BMVC 2021, 🔗
    • Self-Supervised Transformers for Unsupervised Object Discovery using Normalized Cut, CVPR 2022, 🔗
    • Unsupervised Object Localization: Observing the Background to Discover Objects, CVPR 2023, 🔗
    • Cut and Learn for Unsupervised Object Detection and Instance Segmentation, CVPR 2023, 🔗
    • Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection, ECCV 2024, 🔗
    • DIOD: Self-Distillation Meets Object Discovery, CVPR 2024, 🔗

  • Open-world Counting

    • CountGD: Multi-Modal Open-World Counting, NeurIPS 2025, 🔗

  • Zero-shot Semantic Segmentation

    • Diffusion Models for Open-Vocabulary Segmentation, arXiv 2023, 🔗

• DEtection TRansformer (DETR)

  • Knowledge Distillation for Detection Transformer with Consistent Distillation Points Sampling, CVPR 2024, 🔗
  • DETRDistill: A Universal Knowledge Distillation Framework for DETR-families, ICCV 2023, 🔗

• Swin Transformer

  • Per-Pixel Classification is Not All You Need for Semantic Segmentation, NeurIPS 2021, 🔗
  • Dynamic Feature Regularized Loss for Weakly Supervised Semantic Segmentation, arXiv 2021, 🔗
  • SwinNet: Swin Transformer Drives Edge-Aware RGB-D and RGB-T Salient Object Detection, TCSVT 2021, 🔗
  • SwinTrack: A Simple and Strong Baseline for Transformer Tracking, NeurIPS 2022, 🔗
  • SwinF: Swin Transformer with feature fusion in target detection, JPCS 2022, 🔗
  • Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis, CVPR 2022, 🔗
  • SSformer: A Lightweight Transformer for Semantic Segmentation, MMSP 2022, 🔗
  • Weakly Supervised Intracranial Hemorrhage Segmentation using Head-Wise Gradient-Infused Self-Attention Maps from a Swin Transformer in Categorical Learning, arXiv 2023, 🔗
  • HiFormer: Hierarchical Multi-scale Representations Using Transformers for Medical Image Segmentation, WACV 2023, 🔗
  • Swin-Fusion: Swin-Transformer with Feature Fusion for Human Action Recognition, NPL 2023, 🔗
  • Pyramid Swin Transformer: Different-Size Windows Swin Transformer for Image Classification and Object Detection, VISIGRAPP 2023, 🔗
  • SCUNet++: Swin-UNet and CNN Bottleneck Hybrid Architecture with Multi-Fusion Dense Skip Connection for Pulmonary Embolism CT Image Segmentation, WACV 2024, 🔗
  • Leveraging Swin Transformer for Local-to-Global Weakly Supervised Semantic Segmentation, MVIP 2024, 🔗

Self-Supervised Learning

• A simple framework for contrastive learning of visual representations, NeurIPS 2020, 🔗
• Momentum contrast for unsupervised visual representation learning, CVPR 2020, 🔗
• Improved Baselines with Momentum Contrastive Learning, arXiv 2020, 🔗
• Bootstrap your own latent-a new approach to self-supervised learning, NeurIPS 2020, 🔗
• Unsupervised learning of visual features by contrasting cluster assignments, NeurIPS 2020, 🔗
• Self-Supervised Learning of Pretext-Invariant Representations, CVPR 2020, 🔗
• CompRess: Self-Supervised Learning by Compressing Representations, NeurIPS 2020, 🔗
• SimReg: Regression as a Simple Yet Effective Tool for Self-supervised Knowledge Distillation, BMVC 2021, 🔗
• ISD: Self-Supervised Learning by Iterative Similarity Distillation, ICCV 2021, 🔗
• SEED: Self-supervised Distillation For Visual Representation, ICLR 2021, 🔗
• Simple Distillation Baselines for Improving Small Self-supervised Models, ICCV 2021, 🔗
• Emerging Properties in Self-Supervised Vision Transformers, ICCV 2021, 🔗
• Bag of Instances Aggregation Boosts Self-supervised Distillation, ICLR 2022, 🔗
• DisCo: Remedying Self-supervised Learning on Lightweight Models with Distilled Contrastive Learning, ECCV 2022, 🔗
• Auxiliary Learning for Self-Supervised Video Representation via Similarity-Based Knowledge Distillation, CVPR 2022, 🔗
• Masked Video Distillation: Rethinking Masked Feature Modeling for Self-Supervised Video Representation Learning, CVPR 2023, 🔗
• Multi-Mode Online Knowledge Distillation for Self-Supervised Visual Representation Learning, CVPR 2023, 🔗
• DINOv2: Learning Robust Visual Features without Supervision, TMLR 2024, 🔗

Diffusion Models

• Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed, arXiv 2021, 🔗
• Progressive Distillation for Fast Sampling of Diffusion Models, ICLR 2022, 🔗
• Accelerating diffusion sampling with classifier-based feature distillation, ICME 2023, 🔗
• Consistency Models, PMLR 2023, 🔗
• Fast Sampling of Diffusion Models via Operator Learning, PMLR 2023, 🔗
• Diffusion-GAN: Training GANs with Diffusion, ICLR 2023, 🔗
• ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation, NeurIPS 2023, 🔗
• Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Models, NeurIPS 2023, 🔗
• Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference, arXiv 2023, 🔗
• Adversarial Diffusion Distillation, ECCV 2024, 🔗
• Improved Techniques for Training Consistency Models, ICLR 2024, 🔗
• Relational diffusion distillation for efficient image generation, ACM MM 2024, 🔗
• Multistep Distillation of Diffusion Models via Moment Matching, NeurIPS 2024, 🔗
• EM Distillation for One-step Diffusion Models, NeurIPS 2024, 🔗
• Simple and Fast Distillation of Diffusion Models, NeurIPS 2024, 🔗
• One-step Diffusion with Distribution Matching Distillation, CVPR 2024, 🔗
• Simplifying, Stabilizing and Scaling Continuous-Time Consistency Models, arXiv 2024, 🔗
• Consistency Models Made Easy, arXiv 2024, 🔗
• TLCM: Training-efficient Latent Consistency Model for Image Generation with 2-8 Steps, arXiv 2024, 🔗

Knowledge Distillation in Visual Recognition

Object Detection

• Learning Efficient Object Detection Models with Knowledge Distillation, NIPS 2017, 🔗
• Distilling Object Detectors with Fine-grained Feature Imitation, CVPR 2019, 🔗
• An end-to-end architecture for class-incremental object detection with knowledge distillation, ICME 2019, 🔗
• LabelEnc: A New Intermediate Supervision Method for Object Detection, ECCV 2020, 🔗
• MimicDet: Bridging the Gap Between One-Stage and Two-Stage Object Detection, ECCV 2020, 🔗
• Improve object detection with feature-based knowledge distillation: Towards accurate and efficient detectors, ICLR 2021, 🔗
• Distilling Object Detectors via Decoupled Features, CVPR 2021, 🔗
• General Instance Distillation for Object Detection, CVPR 2021, 🔗
• Instance-Conditional Knowledge Distillation for Object Detection, NIPS 2021, 🔗
• Improving Object Detection by Label Assignment Distillation, WACV 2022, 🔗
• Distilling image classifiers in object detectors, NIPS 2021, 🔗
• G-DetKD: Towards General Distillation Framework for Object Detectors via Contrastive and Semantic-guided Feature Imitation, ICCV 2021, 🔗
• Distilling Object Detectors with Feature Richness, NIPS 2021, 🔗
• Prediction-Guided Distillation for Dense Object Detection, ECCV 2022, 🔗
• Focal and Global Knowledge Distillation for Detectors, CVPR 2022, 🔗
• GLAMD: Global and Local Attention Mask Distillation for Object Detectors, ECCV 2022, 🔗
• Masked Generative Distillation, ECCV 2022, 🔗
• LGD: Label-guided Self-distillation for Object Detection, AAAI 2022, 🔗
• Knowledge Distillation for Object Detection via Rank Mimicking and Prediction-Guided Feature Imitation, AAAI 2022, 🔗
• Task-balanced distillation for object detection, Pattern Recognition 2023, 🔗
• Structured Knowledge Distillation for Accurate and Efficient Object Detection, TPAMI 2023, 🔗
• Spatial Self-Distillation for Object Detection with Inaccurate Bounding Boxes, ICCV 2023, 🔗
• Dual Relation Knowledge Distillation for Object Detection, IJCAI 2023, 🔗
• ScaleKD: Distilling Scale-Aware Knowledge in Small Object Detector, CVPR 2023, 🔗
• Multi-level Logit Distillation, CVPR 2023, 🔗
• Bridging Cross-task Protocol Inconsistency for Distillation in Dense Object Detection, ICCV 2023, 🔗
• Distilling DETR with Visual-Linguistic Knowledge for Open-Vocabulary Object Detection, ICCV 2023, 🔗
• UniKD: Universal Knowledge Distillation for Mimicking Homogeneous or Heterogeneous Object Detectors, ICCV 2023, 🔗
• Texture-Guided Saliency Distilling for Unsupervised Salient Object Detection, CVPR 2023, 🔗
• Object-Aware Distillation Pyramid for Open-Vocabulary Object Detection, CVPR 2023, 🔗
• Localization distillation for object detection, IEEE TPAMI 2023, 🔗
• CrossKD: Cross-Head Knowledge Distillation for Dense Object Detection, CVPR 2024, 🔗
• MSSD: multi-scale self-distillation for object detection, Visual Intelligence 2024, 🔗
• CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Prediction, ICLR 2024, 🔗
• Gradient-Guided Knowledge Distillation for Object Detectors, WACV 2024, 🔗
• Efficient Feature Distillation for Zero-Shot Annotation Object Detection, WACV 2024, 🔗

Super Resolution

• Image Super-Resolution Using Knowledge Distillation, ACCV 2018, 🔗
• Feature-Affinity Based Knowledge Distillation for Efficient Image Super-Resolution, ICIP 2020, 🔗
• Learning with Privileged Information for Efficient Image Super-Resolution, ECCV 2020, 🔗
• Data-Free Knowledge Distillation for Image Super-Resolution, CVPR 2021, 🔗
• Towards Compact Single Image Super-Resolution via Contrastive Self-distillation, IJCAI 2021, 🔗
• DIPNet: Efficiency Distillation and Iterative Pruning for Image Super-Resolution, CVPRW 2023, 🔗
• Generative Adversarial Super-Resolution at the edge with knowledge distillation, Engineering Applications of Artificial Intelligence 2023, 🔗
• Hybrid knowledge distillation from intermediate layers for efficient Single Image Super-Resolution, Neurocomputing 2023, 🔗
• Dual cross knowledge distillation for image super-resolution, Journal of Visual Communication and Image Representation 2023, 🔗
• Pairwise Distance Distillation for Unsupervised Real-World Image Super-Resolution, ECCV 2024, 🔗
• Knowledge Distillation for Single Image Super-Resolution via Contrastive Learning, ICMR 2024, 🔗
• MTKD: Multi-Teacher Knowledge Distillation for Image Super-Resolution, ECCV 2024, 🔗
• You Only Need One Step: Fast Super-Resolution with Stable Diffusion via Scale Distillation, ECCV 2024, 🔗
• Attention Guidance Distillation Network for Efficient Image Super-Resolution, CVPRW 2024, 🔗
• SinSR: Diffusion-Based Image Super-Resolution in a Single Step, CVPR 2024, 🔗
• OSFFNet: Omni-Stage Feature Fusion Network for Lightweight Image Super-Resolution, AAAI 2024, 🔗
• Semantic Super-Resolution via Self-Distillation and Adversarial Learning, IEEE Access 2024, 🔗

Image Segmentation

• Structured knowledge distillation for semantic segmentation, CVPR 2019, 🔗
• Knowledge Adaptation for Efficient Semantic Segmentation, CVPR 2019, 🔗
• Intra-class Feature Variation Distillation for Semantic Segmentation, ECCV 2020, 🔗
• Domain Adaptive Knowledge Distillation for Driving Scene Semantic Segmentation, WACVW 2020, 🔗
• Channel-wise Knowledge Distillation for Dense Prediction, ICCV 2021, 🔗
• Double Similarity Distillation for Semantic Image Segmentation, IEEE Transactions on Image Processing (TIP) 2021, 🔗
• Robust Semantic Segmentation With Multi-Teacher Knowledge Distillation, IEEE Access 2021, 🔗
• Cross-Image Relational Knowledge Distillation for Semantic Segmentation, CVPR 2022, 🔗
• Structural and Statistical Texture Knowledge Distillation for Semantic Segmentation, CVPR 2022, 🔗
• Adaptive Perspective Distillation for Semantic Segmentation, TPAMI 2022, 🔗
• Masked Generative Distillation, ECCV 2022, 🔗
• Class Similarity Weighted Knowledge Distillation for Continual Semantic Segmentation, CVPR 2022, 🔗
• Cross-Domain Correlation Distillation for Unsupervised Domain Adaptation in Nighttime Semantic Segmentation, CVPR 2022, 🔗
• DiGA: Distil to Generalize and then Adapt for Domain Adaptive Semantic Segmentation, CVPR 2023, 🔗
• A Good Student is Cooperative and Reliable: CNN-Transformer Collaborative Learning for Semantic Segmentation, ICCV 2023, 🔗
• Endpoints Weight Fusion for Class Incremental Semantic Segmentation, CVPR 2023, 🔗
• Multi-Task Learning with Knowledge Distillation for Dense Prediction, ICCV 2023, 🔗
• Hierarchical Dense Correlation Distillation for Few-Shot Segmentation, CVPR 2023, 🔗
• Distilling Self-Supervised Vision Transformers for Weakly-Supervised Few-Shot Classification & Segmentation, CVPR 2023, 🔗
• Incrementer: Transformer for Class-Incremental Semantic Segmentation With Knowledge Distillation Focusing on Old Class, CVPR 2023, 🔗
• FAKD: Feature Augmented Knowledge Distillation for Semantic Segmentation, WACV 2024, 🔗
• Knowledge Distillation for Efficient Instance Semantic Segmentation with Transformers, CVPRW 2024, 🔗
• CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Prediction, ICLR 2024, 🔗
• P2Seg: Pointly-supervised Segmentation via Mutual Distillation, ICLR 2024, 🔗
• Rethinking Knowledge Distillation With Raw Features for Semantic Segmentation, WACV 2024, 🔗
• BPKD: Boundary Privileged Knowledge Distillation for Semantic Segmentation, WACV 2024, 🔗
• Guided Distillation for Semi-Supervised Instance Segmentation, WACV 2024, 🔗
• Distilling efficient Vision Transformers from CNNs for semantic segmentation, Pattern Recognition 2025, 🔗
• AICSD: Adaptive Inter-Class Similarity Distillation for Semantic Segmentation, Multimedia Tools and Applications 2025, 🔗

Depth Estimation

• Knowledge Distillation for Fast and Accurate Monocular Depth Estimation on Mobile Devices, CVPRW 2021, 🔗
• Realtime single image depth perception in the wild with handheld devices, Sensors 2021, 🔗
• Excavating the Potential Capacity of Self-Supervised Monocular Depth Estimation, ICCV 2021, 🔗
• Self-distilled Feature Aggregation for Self-supervised Monocular Depth Estimation, ECCV 2022, 🔗
• Attention-based depth distillation with 3d-aware positional encoding for monocular 3d object detection, AAAI 2022, 🔗
• Boosting Light-Weight Depth Estimation Via Knowledge Distillation, KSEM 2023, 🔗
• Urcdc-depth: Uncertainty rectified cross-distillation with cutflip for monocular depth estimation, IEEE Transactions on Multimedia 2023, 🔗
• Multi-task learning with knowledge distillation for dense prediction, CVPR 2023, 🔗
• Attention-Based Knowledge Distillation in Scene Recognition: The Impact of a DCT-Driven Loss, IEEE Transactions on Circuits and Systems for Video Technology 2023, 🔗
• Monocular Depth Estimation from a Fisheye Camera Based on Knowledge Distillation, Sensors 2023, 🔗
• 3D Distillation: Improving Self-Supervised Monocular Depth Estimation on Reflective Surfaces, ICCV 2023, 🔗
• Multi-Task Learning with Knowledge Distillation for Dense Prediction, ICCV 2023, 🔗
• GasMono: Geometry-Aided Self-Supervised Monocular Depth Estimation for Indoor Scenes, ICCV 2023, 🔗
• Depth Anywhere: Enhancing 360 Monocular Depth Estimation via Perspective Distillation and Unlabeled Data Augmentation, NIPS 2024, 🔗
• Structure-Centric Robust Monocular Depth Estimation via Knowledge Distillation, ACCV 2024, 🔗
• MonoProb: Self-Supervised Monocular Depth Estimation With Interpretable Uncertainty, WACV 2024, 🔗
• Monocular Depth Estimation via Self-Supervised Self-Distillation, Sensors 2024, 🔗
• MAL: Motion-Aware Loss with Temporal and Distillation Hints for Self-Supervised Depth Estimation, ICRA 2024, 🔗
• Stereo-Matching Knowledge Distilled Monocular Depth Estimation Filtered by Multiple Disparity Consistency, ICASSP 2024, 🔗

Medical Image Analysis

• Categorical Relation-Preserving Contrastive Knowledge Distillation for Medical Image Classification, MICCAI 2021, 🔗
• Efficient Medical Image Segmentation Based on Knowledge Distillation, IEEE Transactions on Medical Imaging (TMI) 2021, 🔗
• DeSD: Self-Supervised Learning with Deep Self-Distillation for 3D Medical Image Segmentation, MICCAI 2022, 🔗
• Efficient Biomedical Instance Segmentation via Knowledge Distillation, MICCAI 2022, 🔗
• Flat-aware Cross-stage Distilled Framework for Imbalanced Medical Image Classification, MICCAI 2022, 🔗
• Free Lunch for Surgical Video Understanding by Distilling Self-Supervisions, MICCAI 2022, 🔗
• Simcvd: Simple contrastive voxel-wise representation distillation for semi-supervised medical image segmentation, IEEE Trans. Med. Imaging 2022, 🔗
• Deep Mutual Distillation for Semi-Supervised Medical Image Segmentation, MICCAI 2023, 🔗
• Distilling BlackBox to Interpretable models for Efficient Transfer Learning, MICCAI 2023, 🔗
• Learnable Cross-modal Knowledge Distillation for Multi-modal Learning with Missing Modality, MICCAI 2023, 🔗
• Self-distillation for surgical action recognition, MICCAI 2023, 🔗
• Semi-supervised Pathological Image Segmentation via Cross Distillation of Multiple Attentions, MICCAI 2023, 🔗
• Bootstrapping Chest CT Image Understanding by Distilling Knowledge from X-ray Expert Models, CVPR 2024, 🔗
• Reverse Knowledge Distillation: Training a Large Model Using a Small One for Retinal Image Matching on Limited Data, WACV 2024, 🔗
• Learning Robust Shape Regularization for Generalizable Medical Image Segmentation, IEEE Trans. Med. Imaging 2024, 🔗
• Enhancing Medical Imaging with GANs Synthesizing Realistic Images from Limited Data, ICETCI 2024, 🔗
• Exploring Generalizable Distillation for Efficient Medical Image Segmentation, IEEE Journal of Biomedical and Health Informatics 2024, 🔗
• Confidence Matters: Enhancing Medical Image Classification Through Uncertainty-Driven Contrastive Self-Distillation, MICCAI 2024, 🔗
• DES-SAM: Distillation-Enhanced Semantic SAM for Cervical Nuclear Segmentation with Box Annotation, MICCAI 2024, 🔗
• Hallucinated Style Distillation for Single Domain Generalization in Medical Image Segmentation, MICCAI 2024, 🔗
• Progressively Correcting Soft Labels via Teacher Team for Knowledge Distillation in Medical Image Segmentation, MICCAI 2024, 🔗
• Reprogramming Distillation for Medical Foundation Models, MICCAI 2024, 🔗
• HDKD: Hybrid Data-Efficient Knowledge Distillation Network for Medical Image Classification, Engineering Applications of Artificial Intelligence 2024, 🔗
• Shape-Intensity Knowledge Distillation For Robust Medical Image Segmentation, Frontiers of Computer Science 2024, 🔗

Object Tracking

• There is More than Meets the Eye: Self-Supervised Multi-Object Detection and Tracking with Sound by Distilling Multimodal Knowledge, CVPR 2021, 🔗
• Ensemble learning with siamese networks for visual tracking, Neurocomputing 2021, 🔗
• Distilled Siamese Networks for Visual Tracking, TPAMI 2022, 🔗
• SDSTrack: Self-Distillation Symmetric Adapter Learning for Multi-Modal Visual Object Tracking, CVPR 2024, 🔗
• Event Stream-based Visual Object Tracking: A High-Resolution Benchmark Dataset and A Novel Baseline, CVPR 2024, 🔗
• Object Knowledge Distillation for Joint Detection and Tracking in Satellite Videos, IEEE Transactions on Geoscience and Remote Sensing 2024, 🔗

Face Recognition

• Teacher Supervises Students How to Learn From Partially Labeled Images for Facial Landmark Detection, ICCV 2019, 🔗
• Fair Feature Distillation for Visual Recognition, CVPR 2021, 🔗
• Rectifying the Data Bias in Knowledge Distillation, ICCV 2021, 🔗
• Teaching Where to Look: Attention Similarity Knowledge Distillation for Low Resolution Face Recognition, ECCV 2022, 🔗
• Evaluation-oriented knowledge distillation for deep face recognition, CVPR 2022, 🔗
• Rethinking Feature-Based Knowledge Distillation for Face Recognition, CVPR 2023, 🔗
• Probabilistic Knowledge Distillation of Face Ensembles, CVPR 2023, 🔗
• SynthDistill: Face Recognition with Knowledge Distillation from Synthetic Data, IJCB 2023, 🔗
• AdaDistill: Adaptive Knowledge Distillation for Deep Face Recognition, ECCV 2024, 🔗
• How Knowledge Distillation Mitigates the Synthetic Gap in Fair Face Recognition, ECCV Workshop 2024, 🔗
• MST-KD: Multiple Specialized Teachers Knowledge Distillation for Fair Face Recognition, ECCV Workshop 2024, 🔗
• Enhanced Face Recognition using Intra-class Incoherence Constraint, ICLR 2024, 🔗
• ProS: Facial Omni-Representation Learning via Prototype-Based Self-Distillation, WACV 2024, 🔗
• AI-KD: Towards Alignment Invariant Face Image Quality Assessment Using Knowledge Distillation, IWBF 2024, 🔗

Action Recognition

• Structural Knowledge Distillation for Efficient Skeleton-Based Action Recognition, IEEE Trans. Image Processing 2021, 🔗
• Video Pose Distillation for Few-Shot, Fine-Grained Sports Action Recognition, ICCV 2021, 🔗
• Learning an Augmented RGB Representation with Cross-modal Knowledge Distillation for Action Detection, ICCV 2021, 🔗
• Privileged Knowledge Distillation for Online Action Detection, Pattern recognition 2022, 🔗
• Multimodal Distillation for Egocentric Action Recognition, ICCV 2023, 🔗
• FSAR: Federated Skeleton-based Action Recognition with Adaptive Topology Structure and Knowledge Distillation, ICCV 2023, 🔗
• Decomposed Cross-Modal Distillation for RGB-Based Temporal Action Detection, CVPR 2023, 🔗
• Generative Model-Based Feature Knowledge Distillation for Action Recognition, AAAI 2024, 🔗
• FROSTER: Frozen CLIP is A Strong Teacher for Open-Vocabulary Action Recognition, ICLR 2024, 🔗

Pose Estimation

• Dynamic Kernel Distillation for Efficient Pose Estimation in Videos, ICCV 2019, 🔗
• Fast Human Pose Estimation, CVPR 2019, 🔗
• Online Knowledge Distillation for Efficient Pose Estimation, ICCV 2021, 🔗
• When Human Pose Estimation Meets Robustness: Adversarial Algorithms and Benchmarks, CVPR 2021, 🔗
• From Synthetic to Real: Unsupervised Domain Adaptation for Animal Pose Estimation, CVPR 2021, 🔗
• Combining Weight Pruning and Knowledge Distillation for CNN Compression, CVPR Workshop 2021, 🔗
• AlphaPose: Whole-Body Regional Multi-Person Pose Estimation and Tracking in Real-Time, IEEE TPAMI 2022, 🔗
• ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation, NeurIPS 2022, 🔗
• DistilPose: Tokenized Pose Regression With Heatmap Distillation, CVPR 2023, 🔗
• Knowledge Distillation for 6D Pose Estimation by Aligning Distributions of Local Predictions, CVPR 2023, 🔗
• Effective Whole-Body Pose Estimation with Two-Stages Distillation, ICCV 2023, 🔗
• HRPose: Real-Time High-Resolution 6D Pose Estimation Network Using Knowledge Distillation, Chinese Journal of Electronics 2023, 🔗
• SDPose: Tokenized Pose Estimation via Circulation-Guide Self-Distillation, CVPR 2024, 🔗

Image Retrieval

• Uncertainty-aware multi-shot knowledge distillation for image-based object re-identification, AAAI 2020, 🔗
• Robust re-identification by multiple views knowledge distillation, ECCV 2020, 🔗
• Relationship-Preserving Knowledge Distillation for Zero-Shot Sketch Based Image Retrieval, ACMM 2021, 🔗
• Asymmetric metric learning for knowledge transfer, CVPR 2021, 🔗
• Contextual Similarity Distillation for Asymmetric Image Retrieval, CVPR 2022, 🔗
• Large-to-small image resolution asymmetry in deep metric learning, WACV 2023, 🔗
• Towards a Smaller Student: Capacity Dynamic Distillation for Efficient Image Retrieval, CVPR 2023, 🔗
• D3still: Decoupled Differential Distillation for Asymmetric Image Retrieval, CVPR 2024, 🔗
• LSTKC: Long Short-Term Knowledge Consolidation for Lifelong Person Re-identification, AAAI 2024, 🔗
• Pairwise difference relational distillation for object re-identification, Pattern Recognition 2024, 🔗