Skip to content

Latest commit

 

History

History
542 lines (440 loc) · 77.8 KB

README.md

File metadata and controls

542 lines (440 loc) · 77.8 KB

Distillation Applications

Large Language Models

  • TinyBERT: Distilling BERT for Natural Language Understanding, arXiv 2019, 🔗 :octocat:
  • Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, arXiv 2019, 🔗
  • Patient Knowledge Distillation for BERT Model Compression, arXiv 2019, 🔗 :octocat:
  • Well-read students learn better: On the importance of pre-training compact models, arXiv 2019, 🔗 :octocat:
  • Xtremedistil: Multi-stage distillation for massive multilingual models, arXiv 2020, 🔗 :octocat:
  • MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices, ACL 2020, 🔗 :octocat:
  • MINILM: deep self-attention distillation for task-agnostic compression of pre-trained transformers, NeurIPS 2020, 🔗
  • xtremedistiltransformers: Task transfer for task-agnostic distillation, arXiv 2021, 🔗 :octocat:
  • Explanations from Large Language Models Make Small Reasoners Better, arXiv 2022, 🔗
  • BERT Learns to Teach: Knowledge Distillation with Meta Learning, ACL 2022, 🔗 :octocat:
  • Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes, arXiv 2023, 🔗 :octocat:
  • Large language models are reasoning teachers, ACL 2023, 🔗 :octocat:
  • Less is More: Task-aware Layer-wise Distillation for Language Model Compression, ICML 2023, 🔗 :octocat:
  • MCC-KD: Multi-CoT Consistent Knowledge Distillation, EMNLP 2023, 🔗 :octocat:
  • Teaching Small Language Models to Reason, ACL 2023, 🔗
  • Symbolic Chain-of-Thought Distillation: Small Models Can Also “Think” Step-by-Step, ACL 2023, 🔗
  • Gkd: Generalized knowledge distillation for auto-regressive sequence models, arXiv 2023, 🔗
  • SCOTT: Self-consistent chain-of-thought distillation, NeurIPS 2023, 🔗 :octocat:
  • Dialogue Chain-of-Thought Distillation for Commonsense-aware Conversational Agents, EMNLP 2023, 🔗 :octocat:
  • MiniLLM: Knowledge Distillation of Large Language Models, ICLR 2024, 🔗 :octocat:
  • On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes, ICLR 2024, 🔗
  • PaD: Program-aided Distillation Specializes Large Models in Reasoning, ACL 2024, 🔗
  • Turning dust into gold: Distilling complex reasoning capabilities from llms by leveraging negative data, AAAI 2024, 🔗 :octocat:
  • PC-LoRA: Low-Rank Adaptation for Progressive Model Compression with Knowledge Distillation, arXiv 2024, 🔗
  • Reverse Thinking Makes LLMs Stronger Reasoners, arXiv 2024, 🔗
  • UniCoTT: A Unified Framework for Structural Chain-of-Thought Distillation, ICLR 2025, 🔗
  • MiniPLM: Knowledge Distillation for Pre-Training Language Models, ICLR 2025, 🔗 :octocat:

Foundation Models

Prominent Foundation Models

  • CLIP : Learning Transferable Visual Models From Natural Language Supervision, PMLR 2021, 🔗 :octocat:
  • GPT
    • GPT-1: Improving Language Understanding by Generative Pre-Training, OpenAI 2018, 🔗
    • Gpt-3: Language Models are Few-Shot Learners, NeurIPS 2020, 🔗
    • Gpt-4: A Review on Advancements and Opportunities in Natural Language Processing, arXiv 2023, 🔗
  • DALL-E: Zero-Shot Text-to-Image Generation, ICML 2021, 🔗 :octocat:
  • SAM: Segment Anything, ICCV 2023, 🔗 :octocat:
  • ViT: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, arXiv 2020, 🔗 :octocat:
  • DeiT: Training data-efficient image transformers & distillation through attention, ICML 2021, 🔗 :octocat:
  • DINO: Emerging Properties in Self-Supervised Vision Transformers, ICCV 2021, 🔗 :octocat:
  • DETR: End-to-End Object Detection with Transformers, ECCV 2020, 🔗 :octocat:
  • Swin Transformer: Hierarchical Vision Transformer using Shifted Windows, ICCV 2021, 🔗 :octocat:

Vision-Language Models (VLMs)

  • Simple Open-Vocabulary Object Detection with Vision Transformers, ECCV 2022, 🔗 :octocat:

  • Learning to Detect and Segment for Open Vocabulary Object Detection, CVPR 2023, 🔗

  • PracticalDG: Perturbation Distillation on Vision-Language Models for Hybrid Domain Generalization, CVPR 2024, 🔗 :octocat:

  • CLIP

    • Open-Vocabulary Object Detection

      • Open-vocabulary Object Detection via Vision and Language Knowledge Distillation, ICLR 2021, 🔗 :octocat:
      • Detecting Twenty-thousand Classes using Image-level Supervision, ECCV 2022, 🔗 :octocat:
      • Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model, CVPR 2022, 🔗 :octocat:
      • Open-Vocabulary DETR with Conditional Matching, ECCV 2022, 🔗 :octocat:
      • F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models, arXiv 2022, 🔗
      • Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detection, NeurIPS 2022, 🔗 :octocat:
      • PromptDet: Towards Open-vocabulary Detection using Uncurated Images, ECCV 2022, 🔗 :octocat:
      • DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detection, NeurIPS 2022, 🔗
      • Exploiting Unlabeled Data with Vision and Language Models for Object Detection, ECCV 2022, 🔗 :octocat:
      • Learning Object-Language Alignments for Open-Vocabulary Object Detection, arXiv 2022, 🔗 :octocat:
      • Open-Vocabulary One-Stage Detection with Hierarchical Visual-Language Knowledge Distillation, CVPR 2022, 🔗 :octocat:
      • Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labeling, CVPR 2022, 🔗 :octocat:
      • Open Vocabulary Object Detection with Pseudo Bounding-Box Labels, ECCV 2022, 🔗 :octocat:
      • Fine-grained Visual-Text Prompt-Driven Self-Training for Open-Vocabulary Object Detection, TNNLS 2022, 🔗
      • Zero-shot Object Detection Through Vision-Language Embedding Alignment, ICDM 2022, 🔗 :octocat:
      • Aligning Bag of Regions for Open-Vocabulary Object Detection, CVPR 2023, 🔗 :octocat:
      • Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformers, CVPR 2023, 🔗
      • Object-Aware Distillation Pyramid for Open-Vocabulary Object Detection, CVPR 2023, 🔗 :octocat:

    • Open-Vocabulary Semantic Segmentation

      • Language-driven Semantic Segmentation, ICLR 2021, 🔗 :octocat:
      • Semantic Segmentation In-the-Wild Without Seeing Any Segmentation Examples, arXiv 2021, 🔗
      • Extract Free Dense Labels from CLIP, ECCV 2022, 🔗 :octocat:
      • Image Segmentation Using Text and Image Prompts, CVPR 2022, 🔗 :octocat:
      • Scaling Open-Vocabulary Image Segmentation with Image-Level Labels, ECCV 2022, 🔗 :octocat:
      • Decoupling Zero-Shot Semantic Segmentation, CVPR 2022, 🔗 :octocat:
      • A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model, ECCV 2022, 🔗 :octocat:
      • ReCo: Retrieve and Co-segment for Zero-shot Transfer, NeurIPS 2022, 🔗 :octocat:
      • Open-vocabulary Semantic Segmentation with Frozen Vision-Language Models, arXiv 2022, 🔗 :octocat:
      • Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP, CVPR 2023, 🔗 :octocat:
      • ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation, CVPR 2023, 🔗 :octocat:
      • FreeSeg: Unified, Universal and Open-Vocabulary Image Segmentation, CVPR 2023, 🔗 :octocat:
      • Exploring Open-Vocabulary Semantic Segmentation from CLIP Vision Encoder Distillation Only, ICCV 2023, 🔗 :octocat:
      • CLIP-DIY: CLIP Dense Inference Yields Open-Vocabulary Semantic Segmentation For-Free, WACV 2024, 🔗

    • Open-Vocabulary Part Segmentation

      • Understanding Multi-Granularity for Open-Vocabulary Part Segmentation, NeurIPS 2025, 🔗 :octocat:

    • Open-Vocabulary Customization

      • Open-Vocabulary Customization from CLIP via Data-Free Knowledge Distillation, ICLR 2025, 🔗

    • Weakly Supervised Semantic Segmentation

      • Cross Language Image Matching for Weakly Supervised Semantic Segmentation, CVPR 2022, 🔗 :octocat:
      • CLIP is Also an Efficient Segmenter: A Text-Driven Approach for Weakly Supervised Semantic Segmentation, CVPR 2023, 🔗 :octocat:
      • Learning Multi-Modal Class-Specific Tokens for Weakly Supervised Dense Object Localization, CVPR 2023, 🔗 :octocat:
      • Question-Answer Cross Language Image Matching for Weakly Supervised Semantic Segmentation, ACM MM 2023, 🔗 :octocat:
      • Prompting classes: Exploring the Power of Prompt Class Learning in Weakly Supervised Semantic Segmentation, WACV 2024, 🔗 :octocat:
      • Frozen CLIP: A Strong Backbone for Weakly Supervised Semantic Segmentation, CVPR 2024, 🔗 :octocat:
      • DIAL: Dense Image-text ALignment for Weakly Supervised Semantic Segmentation, ECCV 2024, 🔗
      • WeakCLIP: Adapting CLIP for Weakly-Supervised Semantic Segmentation, IJCV 2025, 🔗 :octocat:

    • Prompt Learning

      • Learning to Prompt for Vision-Language Models, IJCV 2022, 🔗 :octocat:
      • Conditional Prompt Learning for Vision-Language Models, CVPR 2022, 🔗 :octocat:
      • DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting, CVPR 2022, 🔗 :octocat:
      • Domain Adaptation via Prompt Learning, T-NNLS 2023, 🔗
      • PromptKD: Unsupervised Prompt Distillation for Vision-Language Models, CVPR 2024, 🔗 :octocat:

    • Generation and Editing

      • StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery, ICCV 2021, 🔗 :octocat:
      • StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generators, ACM SIGGRAPH 2022, 🔗 :octocat:
      • Zero-Shot Text-Guided Object Generation with Dream Fields, CVPR 2022, 🔗 :octocat:
      • CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields, CVPR 2022, 🔗 :octocat:
      • Text2Mesh: Text-Driven Neural Stylization for Meshes, CVPR 2022, 🔗 :octocat:
      • Decomposing NeRF for Editing via Feature Field Distillation, NeurIPS 2022, 🔗 :octocat:
      • AvatarCLIP: Zero-Shot Text-Driven Generation and Animation of 3D Avatars, ACM Trans. Graph 2022, 🔗
      • CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation, CVPR 2022, 🔗 :octocat:
      • CLIP2StyleGAN: Unsupervised Extraction of StyleGAN Edit Directions, ACM SIGGRAPH 2022, 🔗
      • Text and Image Guided 3D Avatar Generation and Manipulation, WACV 2023, 🔗
      • Local 3D Editing via 3D Distillation of CLIP Knowledge, CVPR 2023, 🔗

    • Person Re-Identification

      • Distilling CLIP with Dual Guidance for Learning Discriminative Human Body Shape Representation, CVPR 2024, 🔗

    • Zero-Shot Human-Object Interaction (HOI) Detection

      • Exploiting CLIP for Zero-shot HOI Detection Requires Knowledge Distillation at Multiple Levels, WACV 2024, 🔗 :octocat:

    • Open-Vocabulary Out-of-Distribution Classification

      • Distilling Large Vision-Language Model with Out-of-Distribution Generalizability, ICCV 2023, 🔗 :octocat:

    • Domain Generalization

      • A Sentence Speaks a Thousand Images: Domain Generalization through Distilling CLIP with Language Guidance, ICCV 2023, 🔗

    • Video-Language Retrieval

      • CLIPPING: Distilling CLIP-Based Models with a Student Base for Video-Language Retrieval, CVPR 2023, 🔗

    • Affordance Detection

      • What does CLIP know about peeling a banana?, CVPR 2024, 🔗

    • Video Highlight Detection

      • Unleash the Potential of CLIP for Video Highlight Detection, CVPR 2024, 🔗

    • Monocular Depth Estimation

      • CaBins: CLIP-based Adaptive Bins for Monocular Depth Estimation, CVPR 2024, 🔗

    • Multiple Tasks (Image Classification, Object Detection, Semantic Segmentation, Instance Segmentation, Image-Text Retrieval)

      • MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretraining, CVPR 2023, 🔗 :octocat:
      • TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritance, ICCV 2023, 🔗 :octocat:
      • CLIP-KD: An Empirical Study of CLIP Model Distillation, CVPR 2024, 🔗 :octocat:
      • CLIP-Embed-KD: Computationally Efficient Knowledge Distillation Using Embeddings as Teachers, HPEC 2024, 🔗 :octocat:

Segment Anything (SAM)

  • Road Network Graph Extraction

    • Segment Anything Model for Road Network Graph Extraction, CVPR 2024, 🔗

  • Polyp Segmentation

    • PP-SAM: Perturbed Prompts for Robust Adaptation of Segment Anything Model for Polyp Segmentation, CVPR 2024, 🔗 :octocat:

  • Image Restoration

    • Distilling Semantic Priors from SAM to Efficient Image Restoration Models, CVPR 2024, 🔗

  • Weakly Supervised Semantic Segmentation

    • Segment Anything Model (SAM) Enhanced Pseudo Labels for Weakly Supervised Semantic Segmentation, arXiv 2023, 🔗 :octocat:
    • An Alternative to WSSS? An Empirical Study of the Segment Anything Model (SAM) on Weakly-Supervised Semantic Segmentation Problems, arXiv 2023, 🔗 :octocat:
    • Segment Anything is A Good Pseudo-label Generator for Weakly Supervised Semantic Segmentation, arXiv 2023, 🔗
    • Weakly-Supervised Semantic Segmentation with Image-Level Labels: from Traditional Models to Foundation Models, arXiv 2023, 🔗
    • From SAM to CAMs: Exploring Segment Anything Model for Weakly Supervised Semantic Segmentation, CVPR 2024, 🔗 :octocat:

Multi-model Applications

  • SAM + CLIP

    • Foundation Model Assisted Weakly Supervised Semantic Segmentation, WACV 2023, 🔗 :octocat:
    • SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understanding, CVPR 2024, 🔗
    • Test-Time Adaptation with SaLIP: A Cascade of SAM and CLIP for Zero shot Medical Image Segmentation, CVPR 2024, 🔗 :octocat:

  • DINO + CLIP

    • LERF: Language Embedded Radiance Fields, ICCV 2023, 🔗 :octocat:
    • CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentation, ECCV 2025, 🔗 :octocat:

  • DETR + CLIP

    • T-Rex2: Towards Generic Object Detection via Text-Visual Prompt Synergy, ECCV 2024, 🔗 :octocat:

  • SAM + GDINO

    • Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks, arXiv 2024, 🔗 :octocat:
    • Prompting Foundational Models for Omni-supervised Instance Segmentation, CVPR 2024, 🔗

  • Others

    • FeatureNeRF: Learning Generalizable NeRFs by Distilling Foundation Models, ICCV 2023, 🔗 :octocat:
    • DIME-FM: DIstilling Multimodal and Efficient Foundation Models, ICCV 2023, 🔗 :octocat:
    • AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into One, CVPR 2024, 🔗 :octocat:
    • Theia: Distilling Diverse Vision Foundation Models for Robot Learning, arXiv 2024, 🔗 :octocat:
    • Strategies to Leverage Foundational Model Knowledge in Object Affordance Grounding, CVPR 2024, 🔗
    • Enrich Distill and Fuse: Generalized Few-Shot Semantic Segmentation in Remote Sensing Leveraging Foundation Model's Assistance, CVPR 2024, 🔗
    • Exploring the Benefits of Vision Foundation Models for Unsupervised Domain Adaptation, CVPR 2024, 🔗 :octocat:
    • Distilling Knowledge from Multiple Foundation Models for Zero-shot Image classification, PlosOne 2024, 🔗 :octocat:

Vision Transformers

  • Vision Transformer (ViT)

    • Supervised Masked Knowledge Distillation for Few-Shot Transformers, CVPR 2023, 🔗 :octocat:
    • Masked Autoencoders Enable Efficient Knowledge Distillers, CVPR 2023, 🔗 :octocat:
    • Distilling Self-Supervised Vision Transformers for Weakly-Supervised Few-Shot Classification & Segmentation, CVPR 2023, 🔗 :octocat:
    • TinyMIM: An Empirical Study of Distilling MIM Pre-trained Models, CVPR 2023, 🔗 :octocat:
    • Incrementer: Transformer for Class-Incremental Semantic Segmentation With Knowledge Distillation Focusing on Old Class, CVPR 2023, 🔗
    • Cumulative Spatial Knowledge Distillation for Vision Transformers, ICCV 2023, 🔗 :octocat:
    • Feature-based Knowledge Distillation for Vision Transformers, CVPR 2024, 🔗 :octocat:

  • Distillation with No Labels (DINO)

    • Unsupervised Object Discovery

      • Localizing Objects with Self-Supervised Transformers and no Labels, BMVC 2021, 🔗 :octocat:
      • Self-Supervised Transformers for Unsupervised Object Discovery using Normalized Cut, CVPR 2022, 🔗 :octocat:
      • Unsupervised Object Localization: Observing the Background to Discover Objects, CVPR 2023, 🔗 :octocat:
      • Cut and Learn for Unsupervised Object Detection and Instance Segmentation, CVPR 2023, 🔗 :octocat:
      • Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection, ECCV 2024, 🔗 :octocat:
      • DIOD: Self-Distillation Meets Object Discovery, CVPR 2024, 🔗 :octocat:

    • Open-world Counting

      • CountGD: Multi-Modal Open-World Counting, NeurIPS 2025, 🔗 :octocat:

    • Zero-shot Semantic Segmentation

      • Diffusion Models for Open-Vocabulary Segmentation, arXiv 2023, 🔗

  • DEtection TRansformer (DETR)

    • Knowledge Distillation for Detection Transformer with Consistent Distillation Points Sampling, CVPR 2024, 🔗
    • DETRDistill: A Universal Knowledge Distillation Framework for DETR-families, ICCV 2023, 🔗 :octocat:

  • Swin Transformer

    • Per-Pixel Classification is Not All You Need for Semantic Segmentation, NeurIPS 2021, 🔗 :octocat:
    • Dynamic Feature Regularized Loss for Weakly Supervised Semantic Segmentation, arXiv 2021, 🔗
    • SwinNet: Swin Transformer Drives Edge-Aware RGB-D and RGB-T Salient Object Detection, TCSVT 2021, 🔗 :octocat:
    • SwinTrack: A Simple and Strong Baseline for Transformer Tracking, NeurIPS 2022, 🔗 :octocat:
    • SwinF: Swin Transformer with feature fusion in target detection, JPCS 2022, 🔗
    • Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis, CVPR 2022, 🔗 :octocat:
    • SSformer: A Lightweight Transformer for Semantic Segmentation, MMSP 2022, 🔗 :octocat:
    • Weakly Supervised Intracranial Hemorrhage Segmentation using Head-Wise Gradient-Infused Self-Attention Maps from a Swin Transformer in Categorical Learning, arXiv 2023, 🔗 :octocat:
    • HiFormer: Hierarchical Multi-scale Representations Using Transformers for Medical Image Segmentation, WACV 2023, 🔗 :octocat:
    • Swin-Fusion: Swin-Transformer with Feature Fusion for Human Action Recognition, NPL 2023, 🔗
    • Pyramid Swin Transformer: Different-Size Windows Swin Transformer for Image Classification and Object Detection, VISIGRAPP 2023, 🔗
    • SCUNet++: Swin-UNet and CNN Bottleneck Hybrid Architecture with Multi-Fusion Dense Skip Connection for Pulmonary Embolism CT Image Segmentation, WACV 2024, 🔗 :octocat:
    • Leveraging Swin Transformer for Local-to-Global Weakly Supervised Semantic Segmentation, MVIP 2024, 🔗 :octocat:

Self-Supervised Learning

  • A simple framework for contrastive learning of visual representations, NeurIPS 2020, 🔗 :octocat:
  • Momentum contrast for unsupervised visual representation learning, CVPR 2020, 🔗 :octocat:
  • Improved Baselines with Momentum Contrastive Learning, arXiv 2020, 🔗 :octocat:
  • Bootstrap your own latent-a new approach to self-supervised learning, NeurIPS 2020, 🔗 :octocat:
  • Unsupervised learning of visual features by contrasting cluster assignments, NeurIPS 2020, 🔗 :octocat:
  • Self-Supervised Learning of Pretext-Invariant Representations, CVPR 2020, 🔗 :octocat:
  • CompRess: Self-Supervised Learning by Compressing Representations, NeurIPS 2020, 🔗 :octocat:
  • SimReg: Regression as a Simple Yet Effective Tool for Self-supervised Knowledge Distillation, BMVC 2021, 🔗 :octocat:
  • ISD: Self-Supervised Learning by Iterative Similarity Distillation, ICCV 2021, 🔗 :octocat:
  • SEED: Self-supervised Distillation For Visual Representation, ICLR 2021, 🔗 :octocat:
  • Simple Distillation Baselines for Improving Small Self-supervised Models, ICCV 2021, 🔗 :octocat:
  • Emerging Properties in Self-Supervised Vision Transformers, ICCV 2021, 🔗 :octocat:
  • Bag of Instances Aggregation Boosts Self-supervised Distillation, ICLR 2022, 🔗
  • DisCo: Remedying Self-supervised Learning on Lightweight Models with Distilled Contrastive Learning, ECCV 2022, 🔗 :octocat:
  • Auxiliary Learning for Self-Supervised Video Representation via Similarity-Based Knowledge Distillation, CVPR 2022, 🔗 :octocat:
  • Masked Video Distillation: Rethinking Masked Feature Modeling for Self-Supervised Video Representation Learning, CVPR 2023, 🔗 :octocat:
  • Multi-Mode Online Knowledge Distillation for Self-Supervised Visual Representation Learning, CVPR 2023, 🔗 :octocat:
  • DINOv2: Learning Robust Visual Features without Supervision, TMLR 2024, 🔗 :octocat:

Diffusion Models

  • Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed, arXiv 2021, 🔗 :octocat:
  • Progressive Distillation for Fast Sampling of Diffusion Models, ICLR 2022, 🔗 :octocat:
  • Accelerating diffusion sampling with classifier-based feature distillation, ICME 2023, 🔗 :octocat:
  • Consistency Models, PMLR 2023, 🔗 :octocat:
  • Fast Sampling of Diffusion Models via Operator Learning, PMLR 2023, 🔗
  • Diffusion-GAN: Training GANs with Diffusion, ICLR 2023, 🔗 :octocat:
  • ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation, NeurIPS 2023, 🔗 :octocat:
  • Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Models, NeurIPS 2023, 🔗 :octocat:
  • Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference, arXiv 2023, 🔗 :octocat:
  • Adversarial Diffusion Distillation, ECCV 2024, 🔗 :octocat:
  • Improved Techniques for Training Consistency Models, ICLR 2024, 🔗
  • Relational diffusion distillation for efficient image generation, ACM MM 2024, 🔗 :octocat:
  • Multistep Distillation of Diffusion Models via Moment Matching, NeurIPS 2024, 🔗
  • EM Distillation for One-step Diffusion Models, NeurIPS 2024, 🔗
  • Simple and Fast Distillation of Diffusion Models, NeurIPS 2024, 🔗 :octocat:
  • One-step Diffusion with Distribution Matching Distillation, CVPR 2024, 🔗 :octocat:
  • Simplifying, Stabilizing and Scaling Continuous-Time Consistency Models, arXiv 2024, 🔗 :octocat:
  • Consistency Models Made Easy, arXiv 2024, 🔗 :octocat:
  • TLCM: Training-efficient Latent Consistency Model for Image Generation with 2-8 Steps, arXiv 2024, 🔗 :octocat:

Knowledge Distillation in Visual Recognition

Object Detection

  • Learning Efficient Object Detection Models with Knowledge Distillation, NIPS 2017, 🔗
  • Distilling Object Detectors with Fine-grained Feature Imitation, CVPR 2019, 🔗 :octocat:
  • An end-to-end architecture for class-incremental object detection with knowledge distillation, ICME 2019, 🔗 :octocat:
  • LabelEnc: A New Intermediate Supervision Method for Object Detection, ECCV 2020, 🔗 :octocat:
  • MimicDet: Bridging the Gap Between One-Stage and Two-Stage Object Detection, ECCV 2020, 🔗
  • Improve object detection with feature-based knowledge distillation: Towards accurate and efficient detectors, ICLR 2021, 🔗 :octocat:
  • Distilling Object Detectors via Decoupled Features, CVPR 2021, 🔗 :octocat:
  • General Instance Distillation for Object Detection, CVPR 2021, 🔗
  • Instance-Conditional Knowledge Distillation for Object Detection, NIPS 2021, 🔗 :octocat:
  • Improving Object Detection by Label Assignment Distillation, WACV 2022, 🔗 :octocat:
  • Distilling image classifiers in object detectors, NIPS 2021, 🔗 :octocat:
  • G-DetKD: Towards General Distillation Framework for Object Detectors via Contrastive and Semantic-guided Feature Imitation, ICCV 2021, 🔗
  • Distilling Object Detectors with Feature Richness, NIPS 2021, 🔗 :octocat:
  • Prediction-Guided Distillation for Dense Object Detection, ECCV 2022, 🔗 :octocat:
  • Focal and Global Knowledge Distillation for Detectors, CVPR 2022, 🔗 :octocat:
  • GLAMD: Global and Local Attention Mask Distillation for Object Detectors, ECCV 2022, 🔗
  • Masked Generative Distillation, ECCV 2022, 🔗 :octocat:
  • LGD: Label-guided Self-distillation for Object Detection, AAAI 2022, 🔗 :octocat:
  • Knowledge Distillation for Object Detection via Rank Mimicking and Prediction-Guided Feature Imitation, AAAI 2022, 🔗
  • Task-balanced distillation for object detection, Pattern Recognition 2023, 🔗
  • Structured Knowledge Distillation for Accurate and Efficient Object Detection, TPAMI 2023, 🔗
  • Spatial Self-Distillation for Object Detection with Inaccurate Bounding Boxes, ICCV 2023, 🔗 :octocat:
  • Dual Relation Knowledge Distillation for Object Detection, IJCAI 2023, 🔗
  • ScaleKD: Distilling Scale-Aware Knowledge in Small Object Detector, CVPR 2023, 🔗
  • Multi-level Logit Distillation, CVPR 2023, 🔗 :octocat:
  • Bridging Cross-task Protocol Inconsistency for Distillation in Dense Object Detection, ICCV 2023, 🔗 :octocat:
  • Distilling DETR with Visual-Linguistic Knowledge for Open-Vocabulary Object Detection, ICCV 2023, 🔗
  • UniKD: Universal Knowledge Distillation for Mimicking Homogeneous or Heterogeneous Object Detectors, ICCV 2023, 🔗
  • Texture-Guided Saliency Distilling for Unsupervised Salient Object Detection, CVPR 2023, 🔗 :octocat:
  • Object-Aware Distillation Pyramid for Open-Vocabulary Object Detection, CVPR 2023, 🔗 :octocat:
  • Localization distillation for object detection, IEEE TPAMI 2023, 🔗 :octocat:
  • CrossKD: Cross-Head Knowledge Distillation for Dense Object Detection, CVPR 2024, 🔗 :octocat:
  • MSSD: multi-scale self-distillation for object detection, Visual Intelligence 2024, 🔗
  • CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Prediction, ICLR 2024, 🔗 :octocat:
  • Gradient-Guided Knowledge Distillation for Object Detectors, WACV 2024, 🔗
  • Efficient Feature Distillation for Zero-Shot Annotation Object Detection, WACV 2024, 🔗 :octocat:

Super Resolution

  • Image Super-Resolution Using Knowledge Distillation, ACCV 2018, 🔗
  • Feature-Affinity Based Knowledge Distillation for Efficient Image Super-Resolution, ICIP 2020, 🔗 :octocat:
  • Learning with Privileged Information for Efficient Image Super-Resolution, ECCV 2020, 🔗 :octocat:
  • Data-Free Knowledge Distillation for Image Super-Resolution, CVPR 2021, 🔗
  • Towards Compact Single Image Super-Resolution via Contrastive Self-distillation, IJCAI 2021, 🔗 :octocat:
  • DIPNet: Efficiency Distillation and Iterative Pruning for Image Super-Resolution, CVPRW 2023, 🔗 :octocat:
  • Generative Adversarial Super-Resolution at the edge with knowledge distillation, Engineering Applications of Artificial Intelligence 2023, 🔗 :octocat:
  • Hybrid knowledge distillation from intermediate layers for efficient Single Image Super-Resolution, Neurocomputing 2023, 🔗
  • Dual cross knowledge distillation for image super-resolution, Journal of Visual Communication and Image Representation 2023, 🔗
  • Pairwise Distance Distillation for Unsupervised Real-World Image Super-Resolution, ECCV 2024, 🔗 :octocat:
  • Knowledge Distillation for Single Image Super-Resolution via Contrastive Learning, ICMR 2024, 🔗
  • MTKD: Multi-Teacher Knowledge Distillation for Image Super-Resolution, ECCV 2024, 🔗 :octocat:
  • You Only Need One Step: Fast Super-Resolution with Stable Diffusion via Scale Distillation, ECCV 2024, 🔗
  • Attention Guidance Distillation Network for Efficient Image Super-Resolution, CVPRW 2024, 🔗 :octocat:
  • SinSR: Diffusion-Based Image Super-Resolution in a Single Step, CVPR 2024, 🔗 :octocat:
  • OSFFNet: Omni-Stage Feature Fusion Network for Lightweight Image Super-Resolution, AAAI 2024, 🔗
  • Semantic Super-Resolution via Self-Distillation and Adversarial Learning, IEEE Access 2024, 🔗

Image Segmentation

  • Structured knowledge distillation for semantic segmentation, CVPR 2019, 🔗 :octocat:
  • Knowledge Adaptation for Efficient Semantic Segmentation, CVPR 2019, 🔗
  • Intra-class Feature Variation Distillation for Semantic Segmentation, ECCV 2020, 🔗 :octocat:
  • Domain Adaptive Knowledge Distillation for Driving Scene Semantic Segmentation, WACVW 2020, 🔗 :octocat:
  • Channel-wise Knowledge Distillation for Dense Prediction, ICCV 2021, 🔗 :octocat:
  • Double Similarity Distillation for Semantic Image Segmentation, IEEE Transactions on Image Processing (TIP) 2021, 🔗
  • Robust Semantic Segmentation With Multi-Teacher Knowledge Distillation, IEEE Access 2021, 🔗
  • Cross-Image Relational Knowledge Distillation for Semantic Segmentation, CVPR 2022, 🔗 :octocat:
  • Structural and Statistical Texture Knowledge Distillation for Semantic Segmentation, CVPR 2022, 🔗
  • Adaptive Perspective Distillation for Semantic Segmentation, TPAMI 2022, 🔗 :octocat:
  • Masked Generative Distillation, ECCV 2022, 🔗 :octocat:
  • Class Similarity Weighted Knowledge Distillation for Continual Semantic Segmentation, CVPR 2022, 🔗
  • Cross-Domain Correlation Distillation for Unsupervised Domain Adaptation in Nighttime Semantic Segmentation, CVPR 2022, 🔗
  • DiGA: Distil to Generalize and then Adapt for Domain Adaptive Semantic Segmentation, CVPR 2023, 🔗 :octocat:
  • A Good Student is Cooperative and Reliable: CNN-Transformer Collaborative Learning for Semantic Segmentation, ICCV 2023, 🔗
  • Endpoints Weight Fusion for Class Incremental Semantic Segmentation, CVPR 2023, 🔗 :octocat:
  • Multi-Task Learning with Knowledge Distillation for Dense Prediction, ICCV 2023, 🔗
  • Hierarchical Dense Correlation Distillation for Few-Shot Segmentation, CVPR 2023, 🔗
  • Distilling Self-Supervised Vision Transformers for Weakly-Supervised Few-Shot Classification & Segmentation, CVPR 2023, 🔗 :octocat:
  • Incrementer: Transformer for Class-Incremental Semantic Segmentation With Knowledge Distillation Focusing on Old Class, CVPR 2023, 🔗
  • FAKD: Feature Augmented Knowledge Distillation for Semantic Segmentation, WACV 2024, 🔗
  • Knowledge Distillation for Efficient Instance Semantic Segmentation with Transformers, CVPRW 2024, 🔗
  • CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Prediction, ICLR 2024, 🔗 :octocat:
  • P2Seg: Pointly-supervised Segmentation via Mutual Distillation, ICLR 2024, 🔗 :octocat:
  • Rethinking Knowledge Distillation With Raw Features for Semantic Segmentation, WACV 2024, 🔗
  • BPKD: Boundary Privileged Knowledge Distillation for Semantic Segmentation, WACV 2024, 🔗 :octocat:
  • Guided Distillation for Semi-Supervised Instance Segmentation, WACV 2024, 🔗 :octocat:
  • Distilling efficient Vision Transformers from CNNs for semantic segmentation, Pattern Recognition 2025, 🔗
  • AICSD: Adaptive Inter-Class Similarity Distillation for Semantic Segmentation, Multimedia Tools and Applications 2025, 🔗 :octocat:

Depth Estimation

  • Knowledge Distillation for Fast and Accurate Monocular Depth Estimation on Mobile Devices, CVPRW 2021, 🔗
  • Realtime single image depth perception in the wild with handheld devices, Sensors 2021, 🔗
  • Excavating the Potential Capacity of Self-Supervised Monocular Depth Estimation, ICCV 2021, 🔗 :octocat:
  • Self-distilled Feature Aggregation for Self-supervised Monocular Depth Estimation, ECCV 2022, 🔗
  • Attention-based depth distillation with 3d-aware positional encoding for monocular 3d object detection, AAAI 2022, 🔗
  • Boosting Light-Weight Depth Estimation Via Knowledge Distillation, KSEM 2023, 🔗 :octocat:
  • Urcdc-depth: Uncertainty rectified cross-distillation with cutflip for monocular depth estimation, IEEE Transactions on Multimedia 2023, 🔗 :octocat:
  • Multi-task learning with knowledge distillation for dense prediction, CVPR 2023, 🔗
  • Attention-Based Knowledge Distillation in Scene Recognition: The Impact of a DCT-Driven Loss, IEEE Transactions on Circuits and Systems for Video Technology 2023, 🔗
  • Monocular Depth Estimation from a Fisheye Camera Based on Knowledge Distillation, Sensors 2023, 🔗
  • 3D Distillation: Improving Self-Supervised Monocular Depth Estimation on Reflective Surfaces, ICCV 2023, 🔗
  • Multi-Task Learning with Knowledge Distillation for Dense Prediction, ICCV 2023, 🔗
  • GasMono: Geometry-Aided Self-Supervised Monocular Depth Estimation for Indoor Scenes, ICCV 2023, 🔗 :octocat:
  • Depth Anywhere: Enhancing 360 Monocular Depth Estimation via Perspective Distillation and Unlabeled Data Augmentation, NIPS 2024, 🔗 :octocat:
  • Structure-Centric Robust Monocular Depth Estimation via Knowledge Distillation, ACCV 2024, 🔗
  • MonoProb: Self-Supervised Monocular Depth Estimation With Interpretable Uncertainty, WACV 2024, 🔗 :octocat:
  • Monocular Depth Estimation via Self-Supervised Self-Distillation, Sensors 2024, 🔗
  • MAL: Motion-Aware Loss with Temporal and Distillation Hints for Self-Supervised Depth Estimation, ICRA 2024, 🔗 :octocat:
  • Stereo-Matching Knowledge Distilled Monocular Depth Estimation Filtered by Multiple Disparity Consistency, ICASSP 2024, 🔗

Medical Image Analysis

  • Categorical Relation-Preserving Contrastive Knowledge Distillation for Medical Image Classification, MICCAI 2021, 🔗 :octocat:
  • Efficient Medical Image Segmentation Based on Knowledge Distillation, IEEE Transactions on Medical Imaging (TMI) 2021, 🔗 :octocat:
  • DeSD: Self-Supervised Learning with Deep Self-Distillation for 3D Medical Image Segmentation, MICCAI 2022, 🔗 :octocat:
  • Efficient Biomedical Instance Segmentation via Knowledge Distillation, MICCAI 2022, 🔗
  • Flat-aware Cross-stage Distilled Framework for Imbalanced Medical Image Classification, MICCAI 2022, 🔗
  • Free Lunch for Surgical Video Understanding by Distilling Self-Supervisions, MICCAI 2022, 🔗 :octocat:
  • Simcvd: Simple contrastive voxel-wise representation distillation for semi-supervised medical image segmentation, IEEE Trans. Med. Imaging 2022, 🔗
  • Deep Mutual Distillation for Semi-Supervised Medical Image Segmentation, MICCAI 2023, 🔗 :octocat:
  • Distilling BlackBox to Interpretable models for Efficient Transfer Learning, MICCAI 2023, 🔗 :octocat:
  • Learnable Cross-modal Knowledge Distillation for Multi-modal Learning with Missing Modality, MICCAI 2023, 🔗
  • Self-distillation for surgical action recognition, MICCAI 2023, 🔗 :octocat:
  • Semi-supervised Pathological Image Segmentation via Cross Distillation of Multiple Attentions, MICCAI 2023, 🔗 :octocat:
  • Bootstrapping Chest CT Image Understanding by Distilling Knowledge from X-ray Expert Models, CVPR 2024, 🔗
  • Reverse Knowledge Distillation: Training a Large Model Using a Small One for Retinal Image Matching on Limited Data, WACV 2024, 🔗
  • Learning Robust Shape Regularization for Generalizable Medical Image Segmentation, IEEE Trans. Med. Imaging 2024, 🔗 :octocat:
  • Enhancing Medical Imaging with GANs Synthesizing Realistic Images from Limited Data, ICETCI 2024, 🔗
  • Exploring Generalizable Distillation for Efficient Medical Image Segmentation, IEEE Journal of Biomedical and Health Informatics 2024, 🔗 :octocat:
  • Confidence Matters: Enhancing Medical Image Classification Through Uncertainty-Driven Contrastive Self-Distillation, MICCAI 2024, 🔗 :octocat:
  • DES-SAM: Distillation-Enhanced Semantic SAM for Cervical Nuclear Segmentation with Box Annotation, MICCAI 2024, 🔗 :octocat:
  • Hallucinated Style Distillation for Single Domain Generalization in Medical Image Segmentation, MICCAI 2024, 🔗
  • Progressively Correcting Soft Labels via Teacher Team for Knowledge Distillation in Medical Image Segmentation, MICCAI 2024, 🔗
  • Reprogramming Distillation for Medical Foundation Models, MICCAI 2024, 🔗
  • HDKD: Hybrid Data-Efficient Knowledge Distillation Network for Medical Image Classification, Engineering Applications of Artificial Intelligence 2024, 🔗 :octocat:
  • Shape-Intensity Knowledge Distillation For Robust Medical Image Segmentation, Frontiers of Computer Science 2024, 🔗 :octocat:

Object Tracking

  • There is More than Meets the Eye: Self-Supervised Multi-Object Detection and Tracking with Sound by Distilling Multimodal Knowledge, CVPR 2021, 🔗 :octocat:
  • Ensemble learning with siamese networks for visual tracking, Neurocomputing 2021, 🔗
  • Distilled Siamese Networks for Visual Tracking, TPAMI 2022, 🔗
  • SDSTrack: Self-Distillation Symmetric Adapter Learning for Multi-Modal Visual Object Tracking, CVPR 2024, 🔗 :octocat:
  • Event Stream-based Visual Object Tracking: A High-Resolution Benchmark Dataset and A Novel Baseline, CVPR 2024, 🔗 :octocat:
  • Object Knowledge Distillation for Joint Detection and Tracking in Satellite Videos, IEEE Transactions on Geoscience and Remote Sensing 2024, 🔗

Face Recognition

  • Teacher Supervises Students How to Learn From Partially Labeled Images for Facial Landmark Detection, ICCV 2019, 🔗 :octocat:
  • Fair Feature Distillation for Visual Recognition, CVPR 2021, 🔗
  • Rectifying the Data Bias in Knowledge Distillation, ICCV 2021, 🔗
  • Teaching Where to Look: Attention Similarity Knowledge Distillation for Low Resolution Face Recognition, ECCV 2022, 🔗 :octocat:
  • Evaluation-oriented knowledge distillation for deep face recognition, CVPR 2022, 🔗
  • Rethinking Feature-Based Knowledge Distillation for Face Recognition, CVPR 2023, 🔗
  • Probabilistic Knowledge Distillation of Face Ensembles, CVPR 2023, 🔗
  • SynthDistill: Face Recognition with Knowledge Distillation from Synthetic Data, IJCB 2023, 🔗 :octocat:
  • AdaDistill: Adaptive Knowledge Distillation for Deep Face Recognition, ECCV 2024, 🔗 :octocat:
  • How Knowledge Distillation Mitigates the Synthetic Gap in Fair Face Recognition, ECCV Workshop 2024, 🔗 :octocat:
  • MST-KD: Multiple Specialized Teachers Knowledge Distillation for Fair Face Recognition, ECCV Workshop 2024, 🔗 :octocat:
  • Enhanced Face Recognition using Intra-class Incoherence Constraint, ICLR 2024, 🔗
  • ProS: Facial Omni-Representation Learning via Prototype-Based Self-Distillation, WACV 2024, 🔗
  • AI-KD: Towards Alignment Invariant Face Image Quality Assessment Using Knowledge Distillation, IWBF 2024, 🔗 :octocat:

Action Recognition

  • Structural Knowledge Distillation for Efficient Skeleton-Based Action Recognition, IEEE Trans. Image Processing 2021, 🔗 :octocat:
  • Video Pose Distillation for Few-Shot, Fine-Grained Sports Action Recognition, ICCV 2021, 🔗 :octocat:
  • Learning an Augmented RGB Representation with Cross-modal Knowledge Distillation for Action Detection, ICCV 2021, 🔗
  • Privileged Knowledge Distillation for Online Action Detection, Pattern recognition 2022, 🔗
  • Multimodal Distillation for Egocentric Action Recognition, ICCV 2023, 🔗 :octocat:
  • FSAR: Federated Skeleton-based Action Recognition with Adaptive Topology Structure and Knowledge Distillation, ICCV 2023, 🔗
  • Decomposed Cross-Modal Distillation for RGB-Based Temporal Action Detection, CVPR 2023, 🔗
  • Generative Model-Based Feature Knowledge Distillation for Action Recognition, AAAI 2024, 🔗 :octocat:
  • FROSTER: Frozen CLIP is A Strong Teacher for Open-Vocabulary Action Recognition, ICLR 2024, 🔗 :octocat:

Pose Estimation

  • Dynamic Kernel Distillation for Efficient Pose Estimation in Videos, ICCV 2019, 🔗
  • Fast Human Pose Estimation, CVPR 2019, 🔗 :octocat:
  • Online Knowledge Distillation for Efficient Pose Estimation, ICCV 2021, 🔗 :octocat:
  • When Human Pose Estimation Meets Robustness: Adversarial Algorithms and Benchmarks, CVPR 2021, 🔗 :octocat:
  • From Synthetic to Real: Unsupervised Domain Adaptation for Animal Pose Estimation, CVPR 2021, 🔗 :octocat:
  • Combining Weight Pruning and Knowledge Distillation for CNN Compression, CVPR Workshop 2021, 🔗
  • AlphaPose: Whole-Body Regional Multi-Person Pose Estimation and Tracking in Real-Time, IEEE TPAMI 2022, 🔗 :octocat:
  • ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation, NeurIPS 2022, 🔗 :octocat:
  • DistilPose: Tokenized Pose Regression With Heatmap Distillation, CVPR 2023, 🔗 :octocat:
  • Knowledge Distillation for 6D Pose Estimation by Aligning Distributions of Local Predictions, CVPR 2023, 🔗 :octocat:
  • Effective Whole-Body Pose Estimation with Two-Stages Distillation, ICCV 2023, 🔗 :octocat:
  • HRPose: Real-Time High-Resolution 6D Pose Estimation Network Using Knowledge Distillation, Chinese Journal of Electronics 2023, 🔗
  • SDPose: Tokenized Pose Estimation via Circulation-Guide Self-Distillation, CVPR 2024, 🔗 :octocat:

Image Retrieval

  • Uncertainty-aware multi-shot knowledge distillation for image-based object re-identification, AAAI 2020, 🔗
  • Robust re-identification by multiple views knowledge distillation, ECCV 2020, 🔗 :octocat:
  • Relationship-Preserving Knowledge Distillation for Zero-Shot Sketch Based Image Retrieval, ACMM 2021, 🔗
  • Asymmetric metric learning for knowledge transfer, CVPR 2021, 🔗
  • Contextual Similarity Distillation for Asymmetric Image Retrieval, CVPR 2022, 🔗
  • Large-to-small image resolution asymmetry in deep metric learning, WACV 2023, 🔗
  • Towards a Smaller Student: Capacity Dynamic Distillation for Efficient Image Retrieval, CVPR 2023, 🔗
  • D3still: Decoupled Differential Distillation for Asymmetric Image Retrieval, CVPR 2024, 🔗
  • LSTKC: Long Short-Term Knowledge Consolidation for Lifelong Person Re-identification, AAAI 2024, 🔗 :octocat:
  • Pairwise difference relational distillation for object re-identification, Pattern Recognition 2024, 🔗