Is it compatible with transformers library?

[snaik2016]

❓ Questions

This is a great implementation of quantized noise. Has anyone tested it with HF transformer library?

Thanks

[loretoparisi]

Also adding, given a pretrained HF transformer model (let's say a torch RoBERTa model) how the quantization step will work in model's fine tuning process?

[adefossez]

@snaik2016 the implementation is agnostic to the architecture it works on, as long as it is defined as PyTorch modules. Some modules can do very tricky things (for instance RNNs, LSTMs etc for which we had to add explicit support) that would break with DiffQ, so I'm not sure about the HF library. I would say it should work out of the box, but we haven't tested :)

One thing we noticed is if you use layer drop with transformer, things tend not to work great, at least for large values of layer drops (>20%). If you do use layer drop, then you should call quantizer.check_unsused() before the optimizer step.

@loretoparisi we did a bit of fine tuning but mostly experimented with training from scratch, which usually gives the best performance. If you fine tune, you should still fine tune for a relatively large number of epochs (as opposed to having a large model size penalty for very few epochs), for instance you need to do maybe 50 epochs, rather than just 4, as this will give a better chance to the model to adapt gradually.