Potentially slow when running quantized versions on Desktop CPU

[mergennachin]

I tried the following four versions on my Macbook Pro M1.

(1) - Really slow

python3 torchchat.py generate llama3 --prompt "Hello, my name is" --quantize '{"embedding": {"bitwidth": 4, "groupsize":32}, "linear:int8": {"groupsize" : 64}}' 

Average tokens/sec: 0.56

(2) - Acceptable?

python3 torchchat.py generate llama3 --prompt "Hello, my name is" --quantize '{"embedding" : {"bitwidth": 8, "groupsize": 0}}'

Average tokens/sec: 4.26

(3) - Slow

python3 torchchat.py generate llama3 --prompt "Hello, my name is" --quantize '{"embedding" : {"bitwidth": 4, "groupsize": 32}}'

Average tokens/sec: 2.31

(4) - Slow

python3 torchchat.py generate llama3 --prompt "Hello, my name is" --quantize '{"linear:int4": {"groupsize" : 256}}'

Average tokens/sec: 2.94

Setup:

git commit: 695a581
python version: 3.10.0
macbook pro M1

Internal Task: T187752023

[mergennachin]

cc @malfet

what is the best way to trigger optimized int4 kernels? shall I use --compile?

[malfet]

{"linear:int4": {"groupsize" : 256}} should be a way to do that, need to debug why it's slower that float16
And linear:int8": {"groupsize" : 64}} does not have a dedicated op right now, and linear:int8": {"groupsize" : 0}} has a bug, tracked in #602