Question about the metric MAP and nDCG

[DaoD]

Thanks for your great work!

I'm currently reimplementing the results, and I found a problem.

In the repository, there is a qrels_og and qrels_changed in the directory. Which one should be used to compute the metrics MAP and nDCG reported in the paper?

Thanks!

[orionw]

Yes, good question - the original ones (og).

[DaoD]

Thanks! After carefully checking the results, I found that the results provded in this repo (https://github.com/orionw/FollowIR/blob/master/results.zip) seems to use the qrels_changed as the labels to evaluate the results retrived by the original queries + instructions. Could you please explain why we should use this setiing?

[orionw]

Which version of MTEB are you using? MTEB went through a lot of development, I would recommend using the commit at the time of FollowIR (spring 2024) or otherwise using the v2 branch. There was rapid change between these and it broke for a while.

The v2 branch has all the fixes from the last 6+ months and will become the new main, but has been taking a while to be merged in.

EDIT: perhaps I am misunderstanding, you're saying the results previously are from changed? Let me pull up the MTEB version and check. I don't have time in the next couple days but will loop back later this week (feel free to ping if I forget).

[DaoD]

Thanks! I find that if I use qrel_og for evaluation, the results will be significantly higher. If I want to reproduce the results, I have to use the labels in qrel_changed with the instruction in og, which is very weird. Please help me to check this result once you have time.

[orionw]

And what version of MTEB are you using?

Yes with qrels_changed the task is easier so all models do better. Thats why we didn't use it (at least from what I remember).

[DaoD]

I use the newest version. I will carefully checked the evaluation process.

The current problem is:

• If I use qrels_og as the label, and use the query + instruction_og for retrieving, the results are significantly higher than that reported in the paper.
• If I use qrels_changed as the label, and use the query + instruction_changed for retrieving, the results are worse than that reported in the paper.
• If I use qrels_changed as the label, but use the query + instruction_og for retrieving, the results are close to that reported in the paper.

This is what I find weird.

[orionw]

Thanks for the update. If you are using the latest main of MTEB it will be wrong. That branch doesn't have the fixes that were added last October when it broke. You have to use the v2 branch or the main branch at the time of FollowIR (spring 2024). I think switching will alleviate your issue as there is a known problem with the main branch.

[DaoD]

Hi, we're sorry to bother your time.

We have clearly checked the code in MTEB and found that the evaluation code is incorrect.

The code link is: https://github.com/embeddings-benchmark/mteb/blob/main/mteb/abstasks/AbsTaskInstructionRetrieval.py#L117

self.changed_qrels.map(qrels_dict_init)
self.og_qrels.map(qrels_changed_dict_init)

As can be seen, these two lines of code mistakenly changed the order of the qrels_og and qrels_changed, making the final results incorrect.

Please help us to check this problem if you have time.

Many thanks!

Regards

[orionw]

Hi @DaoD, as I've mentioned before, sadly MTEB main is broken. I fixed it in Oct/Nov of last year, but they told me not to merge it into main and only into v2. I just advocated recently to push the v2 into main ASAP, as I agree this is quite confusing. Also, I think the above error only affects the non p-MRR scores, correct?

If you switch to the v2 branch it will be correct evaluation for both p-MRR and non p-MRR.

I will set aside some time this upcoming week to check the initial FollowIR mteb branch from spring 2024 as I haven't looked at that version for quite some time. If the results.zip file matches the incorrect evaluation exactly than that does seem suspicious.

[orionw]

But to be especially clear - please do not use the mteb main branch for instruction-following evaluation!!

There was a lot of rapid development in MTEB last summer and many things broke, including this. What you mention is true and is a known and fixed error in v2. We used the v2 version for example in the MMTEB paper which added instruction following to the benchmarks. We hope to merge it soon into main so that we can resolve this and other issues (known issues with rerankers, etc.).

[DaoD]

Not really. I guess the results of MAP and nDCG are all incorrect (because the qrels file are incorrectly loadded).

In v2, I do not find the full evaluation process (only a function named evaluate_p_mrr_change, but no file loading process). Is there any script that I can run directly?

Thanks!

[orionw]

The task-specific metrics have been moved into each task. It also loads the qrel diff directly from the dataset. There's not one single script that evaluates, other than the main mteb function:

model = mteb.get_model(model_name) # if the model is not implemented in MTEB it will be eq. to SentenceTransformer(model_name)
tasks = mteb.get_tasks(tasks=["NameOfTask"])
evaluation = mteb.MTEB(tasks=tasks)
results = evaluation.run(model, output_folder=f"results/{model_name}")

This way metrics are designed such that evaluation is run with the normal qrels and so MAP/nDCG are unaffected by any other metrics.