Throughputs of Long Sequences #12608

[simmonssong]

Hi, I am testing throughputs of input sequences with different lengths. I found that throughput increases with length on several different models and quantization, is this caused by build-in infrastructure optimization of Llama.cpp?