Bench token generation at long context sizes

[Mushoz]

Is there a way to benchmark models at long context sizes? I am trying to use llama-bench, but I am not able to figure out if it's even possible. I have tried:

• Setting a big -p value. This only seems to bench how long it takes to prompt process these tokens. The token generation seems to be with a short context, as it's just as fast as having a small -p value
• Setting 2 -pg values. While this does seem to do what I want (process a long prompt and then generate tokens), it only lists the combined speed in tokens/sec.

What I want to do, is compare how fast my model is able to generate say 128 tokens with a short 128 token prompt, versus how long it takes to generate those same 128 tokens, with 15k tokens in the context. But I am not able to figure out how to do that. Does anyone know if this is even possible? Thanks!

[ggerganov]

You can get these numbers with the llama-batched-bench tool, although it does not compute uncertainties like the llama-bench tool.

[Mushoz]

This is perfect, thank you!