Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[v2] reupload reranking datasets in old format #2097

Open
wants to merge 11 commits into
base: v2.0.0
Choose a base branch
from

Conversation

Samoed
Copy link
Collaborator

@Samoed Samoed commented Feb 18, 2025

Closes #1878

  • "MindSmallReranking",
  • "SciDocsRR",
  • "StackOverflowDupQuestions",
  • "WebLINXCandidatesReranking",
  • "AlloprofReranking",
  • "SyntecReranking",
  • "VoyageMMarcoReranking",
  • "ESCIReranking",
  • "MIRACLReranking",
  • "WikipediaRerankingMultilingual",
  • "RuBQReranking",
  • "T2Reranking",
  • "MMarcoReranking",
  • "CMedQAv1-reranking",
  • "CMedQAv2-reranking",
  • "NamaaMrTydiReranking",

Code Quality

  • Code Formatted: Format the code using make lint to maintain consistent style.

Documentation

  • Updated Documentation: Add or update documentation to reflect the changes introduced in this PR.

Testing

  • New Tests Added: Write tests to cover new functionality. Validate with make test-with-coverage.
  • Tests Passed: Run tests locally using make test or make test-with-coverage to ensure no existing functionality is broken.

@orionw Can you review, please?

@Samoed Samoed added the v2 Issues and PRs related to `v2` branch label Feb 18, 2025
@Samoed Samoed marked this pull request as ready for review February 20, 2025 15:22
@Samoed Samoed requested a review from orionw February 20, 2025 15:22
@Samoed Samoed mentioned this pull request Feb 21, 2025
4 tasks
Copy link
Contributor

@orionw orionw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great @Samoed, thanks for finalizing this.

My only question is does this affect the listing of tasks? e.g. what distinguishes reranking vs retrieval now, when you load with the mteb.load_tasks for reranking only? I would assume we need some mapping now to maintain that they are reranking tasks.

EDIT: perhaps that's in the task meta? If so then I have no concerns.

@Samoed
Copy link
Collaborator Author

Samoed commented Feb 21, 2025

These tasks will still have type="Reranking" in their TaskMetadata, so this won’t break anything. However, for now, I think I’ll roll back the RerankingTask deletion since there might be more reranking tasks from main that need to be uploaded. I’ll wait for #2116 to be merged, as it includes a new reranking task.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
v2 Issues and PRs related to `v2` branch
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants