Does it work for gemma3 4b

[Oxi84]

System Info

Latest everything, just installed today, and ubuntu 22.04 and another ubuntu, i tried on 2 sistems and is the same. No output!

Reproduction

I tried to use 4bit and 8bit of gemma3 4b and the output is always empty. Fp16 and 32 work properly.

Expected behavior

I tried to use 4bit and 8bit of gemma3 4b and the output is always empty. Fp16 and 32 work properly.

[matthewdouglas]

Hi! For Gemma3 models, it is best to use torch_dtype=torch.bfloat16 or torch_dtype=torch.float32 when loading with transformers. Using float16 precision can result in this behavior.

[Oxi84]

I thought that the model is maybe not supported by BNB.

I load like this, and did not use bf16 at all not sure what is the default:

`model_id = "google/gemma-3-4b-it"

quantization_config = BitsAndBytesConfig(load_in_8bit=True)

model = Gemma3ForConditionalGeneration.from_pretrained(
model_id, quantization_config=quantization_config
).eval()`

[matthewdouglas]

The default for int8 would be float16; this may also be the default for 4bit in the transformers integration. It is best to specify torch_dtype with transformers.