PCL: Proposal Cluster Learning for Weakly Supervised Object Detection

By Peng Tang, Xinggang Wang, Song Bai, Wei Shen, Xiang Bai, Wenyu Liu, and Alan Yuille.

The code to train and eval our original OICR using PyTorch as backend is available here. Thanks Vadim!

We provide a small trick that can improve the result of a single VGG16 model to 47.2% mAP on PASCAL VOC 2007!

We provide an implementation that supports multi-gpu training at here.

I have written a PyTorch implementation of our PCL. The codes are available here. This version codes obtain 49.2% mAP and 65.0% CorLoc 54.1% mAP and 69.5% on PASCAL VOC 2007 dataset!

Introduction

Proposal Cluster Learning (PCL) is a framework for weakly supervised object detection with deep ConvNets.

• It achieves state-of-the-art performance on weakly supervised object detection (Pascal VOC 2007 and 2012, ImageNet DET).
• Our code is written by C++ and Python, based on Caffe, fast r-cnn, and faster r-cnn.

The original paper has been accepted by CVPR 2017. This is an extened version. For more details, please refer to here and here.

Comparison with other methods

(a) Conventional MIL method; (b) Our original OICR method with newly proposed proposal cluster generation method; (c) Our PCL method.

Architecture

Results

Method	VOC2007 test mAP	VOC2007 trainval CorLoc	VOC2012 test mAP	VOC2012 trainval CorLoc	ImageNet mAP
PCL-VGG_M	40.8	59.6	37.6	62.9	14.4
PCL-VGG16	43.5	62.7	40.6	63.2	18.4
PCL-Ens.	45.8	63.0	41.6	65.0	18.8
PCL-Ens.+FRCNN	48.8	66.6	44.2	68.0	19.6

Visualizations

Some PCL visualization results.

Some visualization comparisons among WSDDN, WSDDN+context, and PCL.

License

PCL is released under the MIT License (refer to the LICENSE file for details).

Citing PCL

If you find PCL useful in your research, please consider citing:

@article{tang2018pcl,
    author = {Tang, Peng and Wang, Xinggang and Bai, Song and Shen, Wei and Bai, Xiang and Liu, Wenyu and Yuille, Alan},
    title = {{PCL}: Proposal Cluster Learning for Weakly Supervised Object Detection},
    journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
    volume = {},
    number = {},
    pages = {1--1},
    year = {2018}
}

@inproceedings{tang2017multiple,
    author = {Tang, Peng and Wang, Xinggang and Bai, Xiang and Liu, Wenyu},
    title = {Multiple Instance Detection Network with Online Instance Classifier Refinement},
    booktitle = {IEEE Conference on Computer Vision and Pattern Recognition},
    pages = {3059--3067},
    year = {2017}
}

Contents

1. Requirements: software
2. Requirements: hardware
3. Basic installation
4. Installation for training and testing
5. Extra Downloads (selective search)
6. Extra Downloads (ImageNet models)
7. Extra Downloads (Models trained on PASCAL VOC)
8. Usage
9. A small trick

Requirements: software

1. Requirements for Caffe and pycaffe (see: Caffe installation instructions)

Note: Caffe must be built with support for Python layers!

# In your Makefile.config, make sure to have this line uncommented
WITH_PYTHON_LAYER := 1

2. Python packages you might not have: cython, python-opencv, easydict, sklearn
3. MATLAB

Requirements: hardware

1. NVIDIA GTX TITANX (~12G of memory)

Installation

1. Clone the PCL repository

git clone https://github.com/ppengtang/oicr.git & cd oicr
git checkout pcl
git clone https://github.com/ppengtang/caffe.git

2. Build the Cython modules

cd $PCL_ROOT/lib
make

3. Build Caffe and pycaffe

cd $PCL_ROOT/caffe
# Now follow the Caffe installation instructions here:
#   http://caffe.berkeleyvision.org/installation.html

# If you're experienced with Caffe and have all of the requirements installed
# and your Makefile.config in place, then simply do:
make all -j 8
make pycaffe

Installation for training and testing

1. Download the training, validation, test data and VOCdevkit

wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtrainval_06-Nov-2007.tar
wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtest_06-Nov-2007.tar
wget http://host.robots.ox.ac.uk/pascal/VOC/voc2012/VOCdevkit_18-May-2011.tar

2. Extract all of these tars into one directory named VOCdevkit

tar xvf VOCtrainval_06-Nov-2007.tar
tar xvf VOCtest_06-Nov-2007.tar
tar xvf VOCdevkit_18-May-2011.tar

3. It should have this basic structure

$VOCdevkit/                           # development kit
$VOCdevkit/VOCcode/                   # VOC utility code
$VOCdevkit/VOC2007                    # image sets, annotations, etc.
# ... and several other directories ...

4. Create symlinks for the PASCAL VOC dataset

cd $PCL_ROOT/data
ln -s $VOCdevkit VOCdevkit2007

Using symlinks is a good idea because you will likely want to share the same PASCAL dataset installation between multiple projects.

5. [Optional] follow similar steps to get PASCAL VOC 2012.

6. You should put the generated proposal data under the folder $PCL_ROOT/data/selective_search_data, with the name "voc_2007_trainval.mat", "voc_2007_test.mat", just as the form of fast-rcnn.

7. The pre-trained models are all available in the Caffe Model Zoo. You should put it under the folder $PCL_ROOT/data/imagenet_models, just as the form of fast-rcnn.

Download pre-computed Selective Search object proposals

Pre-computed selective search boxes can also be downloaded for VOC2007 and VOC2012.

cd $PCL_ROOT
./data/scripts/fetch_selective_search_data.sh

This will populate the $PCL_ROOT/data folder with selective_selective_data. (The script is copied from the fast-rcnn).

Download pre-trained ImageNet models

Pre-trained ImageNet models can be downloaded.

cd $PCL_ROOT
./data/scripts/fetch_imagenet_models.sh

These models are all available in the Caffe Model Zoo, but are provided here for your convenience. (The script is copied from the fast-rcnn).

Download models trained on PASCAL VOC

Models trained on PASCAL VOC can be downloaded here.

Usage

Train a PCL network. For example, train a VGG16 network on VOC 2007 trainval

./tools/train_net.py --gpu 1 --solver models/VGG16/solver_voc2007.prototxt \
  --weights data/imagenet_models/$VGG16_model_name --iters 50000

Test a PCL network. For example, test the VGG 16 network on VOC 2007 test:

On trainval

./tools/test_net.py --gpu 1 --def models/VGG16/test.prototxt \
  --net output/default/voc_2007_trainval/vgg16_pcl_iter_50000.caffemodel \
  --imdb voc_2007_trainval

On test

./tools/test_net.py --gpu 1 --def models/VGG16/test.prototxt \
  --net output/default/voc_2007_trainval/vgg16_pcl_iter_50000.caffemodel \
  --imdb voc_2007_test

Test output is written underneath $PCL_ROOT/output.

Evaluation

For mAP, run the python code tools/reval.py

./tools/reval.py $output_dir --imdb voc_2007_test --matlab

For CorLoc, run the python code tools/reval_discovery.py

./tools/reval_discovery.py $output_dir --imdb voc_2007_trainval

A small trick

Uncomment these two lines to achieve 47.2% mAP on PASCAL VOC 2007 using a single VGG16 model! See here for details.

The codes for training fast rcnn by pseudo ground truths are available on here.