bitsandbytes compatibility (#163

[gsarti]

Description

With bitsandbytes v0.37.0 Int8 matmul backward is officially supported. For Inseq, this means that all gradient-based methods become viable for 8-bit models (addressing points raised in #141 and bitsandbytes-foundation/bitsandbytes#67).

Example usage (tested from this branch and transformers==4.27.0.dev0) with T5-11b in a Google Colab session with 15Gb GPU RAM:

!pip install --quiet bitsandbytes
!pip install git+https://github.com/huggingface/transformers.git
!pip install --quiet --upgrade accelerate
!pip install --quiet sentencepiece
!pip install --quiet git+https://github.com/inseq-team/inseq@8bit-compatibility

import inseq
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model_id=f"ybelkada/t5-11b-sharded"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model_8bit = AutoModelForSeq2SeqLM.from_pretrained(model_id, device_map="auto", load_in_8bit=True)

t5_xxl = inseq.load_model(model_8bit, "saliency", tokenizer=model_id)
out = t5_xxl.attribute(
    "translate English to German: Inseq now supports saliency attribution with 8-bit models!"
)

The main fix to enable interoperability is to prevent casting the model to default device if it is loaded in int8.