feat: add int4 and int8 weight-only quantisation

[vatsalaggarwal]

resolves #83

• int4 tok/s is almost 2x fp16/bf16...
• INT8 is slower than bf/16fp16 for some reason, so have added a warning...
• impact on memory usage is minimal.

A10G

bf16: 125tok/s, 8170GB
int8: 65tok/s, 7208GB
Int4: 225tok/s, 6660GB

4090

bf16: 229 tok/s, 8.75GB mem
Int8: 193tok/s, 7.68GB
int4: 431tok/s, 7.51GB

[arthurwolf]

If this fits in 8.75GB does that mean a 3060 should be able to run this? it didn't fit last time I tried. would the int4 version have a better chance ? (amazing work btw)

[vatsalaggarwal]

If this fits in 8.75GB does that mean a 3060 should be able to run this? it didn't fit last time I tried. would the int4 version have a better chance ? (amazing work btw)

I think so... worth trying again!

[arthurwolf]

when can we expect to have documentation/instructions on how to run it in a beginner-friendly way? I'll try as soon as I have that, for sure.

…

On Thu, Mar 14, 2024 at 7:11 PM Vatsal Aggarwal ***@***.***> wrote: If this fits in 8.75GB does that mean a 3060 should be able to run this? it didn't fit last time I tried. would the int4 version have a better chance ? (amazing work btw) I think so... worth trying again! — Reply to this email directly, view it on GitHub <#95 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAA2SFJHJANP5MPJQCOYXADYYHR4JAVCNFSM6AAAAABEWSIEJWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJYGA2DSNRUGQ> . You are receiving this because you commented.Message ID: ***@***.***>

-- 勇気とユーモア

[vatsalaggarwal]

@arthurwolf have you tried the instructions in the README? What were the problems you ran into and how can we make it more friendly?

[lucapericlp]

LGTM