SGLang int8 kernels #2196
vadimkantorov
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
I wonder if these Triton kernels are any relevant for wider torchao / pytorch usage (and if this Triton impl is also any portable for CPU):
and if not - I wonder why sglang does not use the quant triton kernels/bindings from ao?
(Also similar question on liger / unsloth kernels - including the notorious rmsnorm kernels - any plans to upstream their main components like linear + chunked cross entropy some place upstream?)
Beta Was this translation helpful? Give feedback.
All reactions