perplexity : faster Winogrande via batching

[ggerganov]

Similar to #5017 but for the Winogrande evaluation. Here the performance gain is bigger since the tasks are smaller and we can fit more in a batch.

I'm not 100% sure about the correctness of this implementation - the final value is different from master:

# master
Final Winogrande score(1267 tasks): 67.6401 +/- 1.3149

# PR
Final Winogrande score(1267 tasks): 69.1397 +/- 1.2982

It's possible that it fixes some existing issue, but it is also possible that it introduces a bug. The softmax optimization is not applied yet - it can be applied in a follow-up PR

[ikawrakow]

The llama_kv_cache_seq_rm call is no longer needed here?

[ggerganov]

It's not needed because we clear the entire KV cache before each batch:

llama.cpp/examples/perplexity/perplexity.cpp

Lines 942 to 944 in 9e4ad80

	llama_kv_cache_clear(ctx);

In the old implementation, it was reusing tokens from a previous batch, so the llama_kv_cache_seq_rm was used to evict the unused ones (i.e. the second sentence).

[ggerganov]

The score with Mistral 7B has changed in the "positive" direction as well, closer to the HF leaderboard results:

# master
Final Winogrande score(1267 tasks): 73.6385 +/- 1.2383

# PR
Final Winogrande score(1267 tasks): 74.5856 +/- 1.2236

[ikawrakow]

The score with Mistral 7B has changed in the "positive" direction as well, closer to the HF leaderboard results:

Yes, I saw this change, but could not understand the reason.