Fix gsm8k task to enhance accuracy

[zhangxinyuehfad]

When using the original GSM8K configuration, I found that the output accuracy was low. In issue #2707 (#2707), this was fixed by optimizing the output and evaluation format, which significantly improved the accuracy.

The results are as follows:
before:

after:

[CLAassistant]

All committers have signed the CLA.

[leo-pony]

@baberabb
Could you please help review this PR? I also found the low accuracy problem caused by the difference between output format and judgment format in the above gsm8k test.

[leo-pony]

@baberabb
Could you please help review this PR? I also found the low accuracy problem caused by the difference between output format and judgment format in the above gsm8k test.

@StellaAthena Could you please help review this PR?

[baberabb]

Hi @leo-pony. Thanks for the PR. Could you reference where you sourced this prompt from? Our library aims to provide standardized metrics for direct model-to-model comparisons, which is why we're careful about prompt modifications. Some details can be found in the readme.
Have you tried evaluating on gsm8k_llama, which is based on the llama evals? It's very similar to your modification, so I think results might be similar.

[leo-pony]

Hi @leo-pony. Thanks for the PR. Could you reference where you sourced this prompt from? Our library aims to provide standardized metrics for direct model-to-model comparisons, which is why we're careful about prompt modifications. Some details can be found in the readme. Have you tried evaluating on gsm8k_llama, which is based on the llama evals? It's very similar to your modification, so I think results might be similar.

@baberabb Thanks for your review!
We applied the answer result formatting prompts in this PR to the llama3.1-8B-Instruct model and found that this prompt had no effect on llama3.1-8B-Instruct. It seems that qwen2.5 did not understand the current result formatting prompts of the lm-eval community well, but understood the result formatting prompts in this PR better.
The detailed test results of llama3.1-7B-instruct using this PR prompt are as follows:

The detailed test results of llama3.1-7B-instruct using community prompt are as follows: