Match me if you can: Semi-Supervised Semantic Correspondence Learning with Unpaired Images

Jiwon Kim^1,2*, Byeongho Heo¹, Sangdoo Yun¹, Seungryong Kim³, Dongyoon Han¹†,
^{* Work done during an internship at NAVER AI Lab, currently at LG AI Research
† Corresponding author
1}NAVER AI LAB, ²LG AI Research, ³ KAIST

Abstract

Semantic correspondence methods have advanced to obtaining high-quality correspondences employing complicated networks, aiming to maximize the model capacity. However, despite the performance improvements, they may remain constrained by the scarcity of training keypoint pairs, a consequence of the limited training images and the sparsity of keypoints. This paper builds on the hypothesis that there is an inherent data-hungry matter in learning semantic correspondences and uncovers the models can be more trained by employing densified training pairs. We demonstrate a simple machine annotator reliably enriches paired key points via machine supervision, requiring neither extra labeled key points nor trainable modules from unlabeled images. Consequently, our models surpass current state-of-the-art models on semantic correspondence learning benchmarks like SPair-71k, PF-PASCAL, and PF-WILLOW and enjoy further robustness on corruption benchmarks.

Our Motivation:

Current semantic correspondence learning suffers from data hunger during training:

• (a) Labeled images in SPair-71k contain sparse manually annotated keypoint pairs

• (b) Unlabeled images could be hidden supplementary sources to increase the pairs' density.

• (c) Newly expanded image pairs can provide abundant densified pairs to alleviate the data-hungry matter.

  

New Benchmark

Visualization of corrupted images in SPair-C

Corrupted images with different severities

• Download Spair-C

Updates

• (2024/10/11): Code is under internal review.
• (2024/09/20): Our paper has been accepted at ACCV 2024🎉🎉🎉