Throughputs of Long Sequences #12608 #1985
Unanswered
simmonssong
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Hi, I am testing throughputs of input sequences with different lengths. I found that throughput increases with length on several different models and quantization, is this caused by build-in infrastructure optimization of Llama.cpp?
Beta Was this translation helpful? Give feedback.
All reactions