Skip to content

Latest commit

 

History

History
194 lines (138 loc) · 10.1 KB

README.md

File metadata and controls

194 lines (138 loc) · 10.1 KB
Raw

Un-Mix: Rethinking Image Mixtures for Unsupervised Visual Representation Learning

Update (06/30/2022)

New: We add more implementation code and models in this repo:

(Insights of Un-Mix on clustering-based methods like SwAV can be found here.)

(Un-Mix w/ MoCo on CIFAR-10 and CIFAR-100 has already been included in this repo.)

Notes: As Un-Mix implementation on CIFAR with SwAV was done in 2020, while the SwAV authors simplified their cluster assignment implementation in April, 2021. Thus, our CIFAR code follows their old implementation before simplification, but the ImageNet code is based on their updated version. Nevertheless, the performance of them will not be affected.

Update (01/07/2022)

  • Un-Mix has been accepted in AAAI 2022!! Please check out our camera-ready paper on arXiv.

  • We provide a demo of Un-Mix on CIFAR-10 and 100 adapted from Colab notebook of MoCo.

    To train an Un-Mix model with MoCo and symmetric loss, run unmix_c10.py or unmix_c100.py:

     # CIFAR-10
 CUDA_VISIBLE_DEVICES=0 python unmix_c10.py 
 
 # CIFAR-100
 CUDA_VISIBLE_DEVICES=1 python unmix_c100.py

    Results on CIFAR-10:

    Model epochs acc. (Top-1) weights (last) logs args
    MoCo w/ asymmetric 1000 89.11% link link link
    +Un-Mix w/ asymmetric 1000 91.34% link link link
    MoCo w/ symmetric 1000 90.49% link link link
    +Un-Mix w/ symmetric 1000 92.25% link link link

    Results on CIFAR-100:

    Model epochs acc. (Top-1) weights (last) logs args
    MoCo w/ asymmetric 1000 63.64% link link link
    +Un-Mix w/ asymmetric 1000 67.33% link link link
    MoCo w/ symmetric 1000 65.49% link link link
    +Un-Mix w/ symmetric 1000 68.83% link link link

  • Our pre-trained ResNet-50 model on ImageNet (MoCo V2 based) is available at: Download. The training code is available at: code.

Update (02/15/2021)

We update our manuscript with a more comprehensive study using image mixtures method on unsupervised learning. The core codes of our method can be summarized as follows:

# P: probability of global or local level mixtures 
# beta: hyperparameter for Beta distribution
# lam: mixture ratio in global-level mixture or bounding box location in region-level mixture
args.beta = 1.0
for x in loader: # load a minibatch x with N samples
    # Probability of choosing global or local level mixtures
    prob = np.random.rand(1)
    lam = np.random.beta(args.beta, args.beta) 
    images_reverse = torch.flip(x[0], (0,))
    if prob < args.P:
	# global-level mixtures
	mixed_images = lam*x[0]+(1-lam)* images_reverse
	mixed_images_flip = torch.flip(mixed_images, (0,))
    else:
	# region-level mixtures
	mixed_images = x[0].clone()
	bbx1, bby1, bbx2, bby2 = utils.rand_bbox(x[0].size(), lam)
	mixed_images[:, :, bbx1:bbx2, bby1:bby2] = images_reverse[:, :, bbx1:bbx2, bby1:bby2] 
	mixed_images_flip = torch.flip(mixed_images,(0,))
	lam = 1 - ((bbx2 - bbx1) * (bby2 - bby1) / (x[0].size()[-1] * x[0].size()[-2]))
    # original loss term
    loss_ori = model(x)
    # In following two losses, we found using ''x[0]'' may perform better on some particular datasets
    # loss for the normal order of mixtures
    loss_m1 = model([x[1], mixed_images])
    # loss for the reverse order of mixtures
    loss_m2 = model([x[1], mixed_images_flip])
    # final loss function (our core code)
    loss = loss_ori + lam * loss_m1 + (1-lam) * loss_m2
    # update gradients
    optimizer.zero_grad() 
    loss.backward() 
    optimizer.step()
    ...

This repo contains the implementation for Un-Mix: Rethinking Image Mixtures for Unsupervised Visual Representation Learning, which perturbs input image space to soften the output prediction space indirectly, meanwhile, assigning new label values in the unsupervised frameworks accordingly. So that the proposed method can smooth decision boundaries and prevent the learner from becoming over-confident.

Our Motivation: Soften Input/label Spaces

Figure 1: Illustration of our motivation on contrastive-based unsupervised learning approaches. Contrastive learning measures the similarity of sample pairs in the latent representation space. With flattened prediction, the model is encouraged to treat each incorrect instance as equally probable, which will smooth decision boundaries and prevent the learner from becoming over-confident.

Results

We run our method with SimCLR, BYOL, MoCo and MoCo V2, the results are as follows:

Figure 2: The curves of training loss and testing accuracy of SimCLR, BYOL and MoCo on CIFAR-10/100 datasets.
Figure 3: Linear classification accuracy of Top-1 (left) and Top-5 (right) with MoCo V2 and ours on ImageNet dataset.
Table 1: Comparison of linear classification on standard ImageNet. † denotes our result using multi-scale training.
Table 2: Object detection results fine-tuned on PASCAL VOC (a) and COCO (b).

Visualizations

Figure 4: Visualization of feature embedding with MoCo on CIFAR-10.
Figure 5: Illustration of weight distributions at 1, 10, 20, 30, 40 and 50-th convolutional layers in a ResNet-50 on ImageNet.

Citation

If you find this repo useful for your research, please consider citing the paper

@inproceedings{shen2022unmix,
  title={Un-Mix: Rethinking Image Mixtures for Unsupervised Visual Representation Learning},
  author={Shen, Zhiqiang and Liu, Zechun and Liu, Zhuang and Savvides, Marios and Darrell, Trevor and Xing, Eric},
  journal={Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)},
  year={2022}
}

Contact

For any questions and comments, please contact Zhiqiang Shen (zhiqiangshen0214 at gmail.com).

Acknowledgements

MoCo V1&V2 (https://github.com/facebookresearch/moco)

Whitening (https://github.com/htdt/self-supervised)

PyTorch Image Classification (https://github.com/hysts/pytorch_image_classification)