Questions for mode == "single_inference"

[KZF-kzf]

Thank you very much for your work! Here are some of my questions, which I hope you can answer:

I want to know if this mode is suitable for evaluating the performance of editing models on a fixed test set. The scores output by this EditReward are usually between -3 and 2; I don’t know if this is normal (the comparison tables in the paper seem to be all percentages). The code shows that editreward will be evaluated from two dimensions, so why is the final value taken as reward[0][0].item? Does reward[0][1] have any practical meaning?

[KemingWu]

Thanks for the questions! Here is a brief explanation:

Benchmark Results: The percentages in the paper represent accuracy—they measure how often the model's predicted preference aligns with the ground truth human labels on the test set. This is why they differ from the raw output scores.

Raw Scores: The raw values (e.g., -3 to 2) are expected because we use a probabilistic modeling approach to calculate scores. We aggregate the two evaluation dimensions into a single final value using specific aggregation strategies.

Reward Values: In the code, reward[0][0] represents the mean (the final score), and reward[0][1] represents the variance.

For more technical details on the probabilistic formulation and aggregation, please refer to the corresponding sections in our paper.

[KZF-kzf]

Thank you very much for your reply! It was very helpful. I will use editreward to evaluate my editing model 😊