Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bugfix AbsTaskRetrieval for DREPS evaluation #80

Merged
merged 3 commits into from
Aug 25, 2023

Conversation

ashokrajab
Copy link
Contributor

@ashokrajab ashokrajab commented Aug 20, 2023

Issues fixed:

  1. When debugging the evalution_model.py in multi_gpu case, there occurs a debugger port collision between parent and child processes. In order to avoid this: nested the evaluation_model.py within name == 'main'.
  2. The expected behaviour of sentence_transformer_encode_multi_process_worker() function is to encode the corpus, evaluate the score against all the queries and store the metric. This was not properly handled by this function. Hence made use of the default SentenceTransformer._encode_multi_process_worker().
  3. Appended instruction in sentences list in encode_corpus_parallel(). This change is made with reference taken from encode_corpus() function.
  4. USE_BEIR_DEVELOPMENT is removed from evaluation/MTEB/mteb/abstasks/BeIRTask.py. This boolean just skips cqadupstack dataset download. I do not find any valid reason to do so. Hence removed.
  5. Added explicit requirement pyarrow==8.0.0. Pip auto dependency resolver installs the latest version of pyarrow and that does not play well with the evaluate package.
  6. Added necessary changes in train.py in order to feed in validation dataset.

@hongjin-su hongjin-su merged commit 554e944 into xlang-ai:main Aug 25, 2023
@ashokrajab ashokrajab deleted the bugfix_abs_retrieval_multi_gpu branch September 24, 2023 08:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants