Compact Convolution Transformers

This repo contain pytorch implementation of Compact Convolution Transformers as explained in the Escaping the Big Data Paradigm with Compact Transformers paper, for official implementation of this paper visit here

Usage:

import torch
import numpy as np
from cct import CompactTransformer

img = torch.ones([1, 3, 224, 224])

cvt = CompactTransformer(224, 16, 1000) # For CVT

parameters = filter(lambda p: p.requires_grad, cvt.parameters())
parameters = sum([np.prod(p.size()) for p in parameters]) / 1_000_000
print('Trainable Parameters in CVT: %.3fM' % parameters)

out = cvt(img)

print("Shape of out :", out.shape)  # [B, num_classes]

cct = CompactTransformer(224, 16, 1000, conv_embed=True) # For CCT

parameters = filter(lambda p: p.requires_grad, cct.parameters())
parameters = sum([np.prod(p.size()) for p in parameters]) / 1_000_000
print('Trainable Parameters in CCT: %.3fM' % parameters)

out = cct(img)

print("Shape of out :", out.shape)  # [B, num_classes]

Citation

@misc{hassani2021escaping,
      title={Escaping the Big Data Paradigm with Compact Transformers}, 
      author={Ali Hassani and Steven Walton and Nikhil Shah and Abulikemu Abuduweili and Jiachen Li and Humphrey Shi},
      year={2021},
      eprint={2104.05704},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}