Bug: Strange dips in TG performance

[saood06]

What happened?

As mentioned in #273 I've seen this behavior occur with llama-server (sorry, I never really noted the configurations or models it occurs with), and I can usually mitigate it by canceling and then restarting generation until TG performance goes back to the expected value, the chart below shows this behavior captured in a benchmark.

Also I'm fairly certain I've never encountered this bug in batched-bench only in server and sweep-bench both of which manipulate the KV more than batched-bench.

Name and Version

Graph capturing this behavior was on 3d6e25c

What operating system are you seeing the problem on?

Linux

Relevant log output

[saood06]

Closing via #282

PP performance for those options:

For my primary use case MLA-3 on is the best with nice PP and TG, it seems like though for tasks with very small PP and TG keeping context under 8K MLA-1 off is useful.

Thank you for the quick find and fix.