Add distinct_1/2 metric #108
Conversation
| import datasets | ||
| from nltk import ngrams | ||
| from rageval.metrics import Metric, add_attribute | ||
|
|
There was a problem hiding this comment.
import/from is not standardized here
rageval/metrics/_answer_precision.py
Outdated
There was a problem hiding this comment.
This metric is lacked of a unit test file.
_answer_precision is not a suitable file name. It is recommended to change it to _answer_perplexity
rageval/metrics/_answer_precision.py
Outdated
| from rageval.metrics import Metric, add_attribute | ||
|
|
||
|
|
||
| _CITATION = """\ |
There was a problem hiding this comment.
This citation doesn't seem to be correct.
rageval/metrics/_answer_precision.py
Outdated
| longer than the max input length of the model, then it is truncated to the | ||
| max length for the perplexity computation. | ||
|
|
||
| Examples: |
There was a problem hiding this comment.
This makes it difficult to pass CI tests in an environment without a GPU.
rageval/metrics/_answer_precision.py
Outdated
| import evaluate | ||
| from evaluate import logging | ||
| from rageval.metrics import Metric, add_attribute | ||
|
|
There was a problem hiding this comment.
import/from is not standardized here
…te-community/rageval into feature/distinct_12_metrics
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #108 +/- ##
==========================================
+ Coverage 81.66% 82.17% +0.50%
==========================================
Files 33 34 +1
Lines 1129 1161 +32
==========================================
+ Hits 922 954 +32
Misses 207 207
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
I add a new evaluation metric, Distinct 1/2, for the generate task evaluation. I have uploaded the new scripts "_answer_distinct12.py" and "test_answer_distinct12.py", and the modified version of "init.py".