This repo is the official implementation of "Confidence-Aware Active Feedback for Interactive Instance Search" by Yue Zhang, Chao Liang and Longxiang Jiang.
Online relevance feedback (RF) is widely utilized in instance search (INS) tasks to further refine imperfect ranking results, but it often has low interaction efficiency. The active learning (AL) technique addresses this problem by selecting valuable feedback candidates. However, mainstream AL methods require an initial labeled set for a cold start and are often computationally complex to solve. Therefore, they cannot fully satisfy the requirements for online RF in interactive INS tasks. To address this issue, we propose a confidence-aware active feedback method (CAAF) that is specifically designed for online RF in interactive INS tasks. Inspired by the explicit difficulty modeling scheme in self-paced learning, CAAF utilizes a pairwise manifold ranking loss to evaluate the ranking confidence of each unlabeled sample. The ranking confidence improves not only the interaction efficiency by indicating valuable feedback candidates but also the ranking quality by modulating the diffusion weights in manifold ranking. In addition, we design two acceleration strategies, an approximate optimization scheme and a top-K search scheme, to reduce the computational complexity of CAAF. Extensive experiments on both image INS tasks and video INS tasks searching for buildings, landscapes, persons, and human behaviors demonstrate the effectiveness of the proposed method. Notably, in the real-world, large-scale video INS task of NIST TRECVID 2021, CAAF uses 25% fewer feedback samples to achieve a performance that is nearly equivalent to the champion solution. Moreover, with the same number of feedback samples, CAAF's mAP is 51.9%, significantly surpassing the champion solution by 5.9%.
We provide the preprocessed data of Oxford5k, Holidays and CUHK03 on GoogleDrive and Baiduyun, please put them (*.mat) into the ./data/ folder.
You can test CAAF on Oxford5k with default settings using the following code:
pip install -r requirements.txt --extra-index-url https://download.pytorch.org/whl/cu113
cd code
python main.py
You can also test on Holidays and CUHK03 with other parameters by modifying the following lines in main.py:
dataset = 'oxford5k' #['oxford5k','holidays','cuhk03']
params = init_params(dataset=dataset,t=5,q=5,k=300,alpha=1e-2,method="CAAF")
| T | Holidays | Oxford5k | CUHK03 |
|---|---|---|---|
| 0 | 67.14 | 43.19 | 53.90 |
| 1 | 79.83 | 50.75 | 72.17 |
| 2 | 84.02 | 55.78 | 85.55 |
| 3 | 86.21 | 59.46 | 89.97 |
| 4 | 87.46 | 61.51 | 91.33 |
If you find this repository useful, please consider giving ⭐ or citing:
@article{zhang2022confidence,
title={Confidence-Aware Active Feedback for Interactive Instance Search},
author={Zhang, Yue and Liang, Chao and Jiang, Longxiang},
journal={IEEE Transactions on Multimedia},
year={2022},
publisher={IEEE}
}