[Feature] Add ROUGE to mmeval #72

go-with-me000 · 2023-01-03T03:24:34Z

Motivation

The task of this PR is to realize ROUGE metric. I have completed the implementation of rouge and provided test files for the test function. And the BLEU metric is modified.

Modification

In order to implement the route metric, we also made amendments to the previous situation that bleu could not evaluate Chinese data.

Checklist

Add rouge metric
Provide test file for rouge
Modified bleu metric

mmeval/metrics/bleu.py

mmeval/metrics/rouge.py

mmeval/metrics/bleu.py

zhouzaida · 2023-01-12T06:03:35Z

mmeval/metrics/bleu.py

+        if isinstance(predictions, str):
+            predictions = [predictions]
+
+        if isinstance(references, str):


If references is sequence[str], should it be wrapped with []?

In lines 105-113 of the code, the situation of sequence [str] is analyzed

mmeval/metrics/rouge.py

zhouzaida · 2023-01-12T06:17:09Z

mmeval/metrics/rouge.py

+                f'`tokenizer` supports Callable, str or None, but not `{type(tokenizer)}`'  # noqa: E501
+        self.accumulate = accumulate
+
+    def add(self, predictions: Union[str, Sequence[str]], references: Union[str, Sequence[str], Sequence[Sequence[str]]]) -> None:  # type: ignore # yapf: disable # noqa: E501


Is it necessary to support the string input?

Considering that there may be only one statement for reference and prediction in the simplest case, you can use the simplest method of passing in two strings instead of adding many layer brackets, which may be more convenient for users.

zhouzaida · 2023-01-14T08:20:21Z

mmeval/metrics/rouge.py

+    recall = matches / reference_len
+    if precision == recall == 0.0:
+        return dict(
+            precision=float(0.0), recall=float(0.0), fmeasure=float(0.0))


Suggested change

precision=float(0.0), recall=float(0.0), fmeasure=float(0.0))

precision=0., recall=0., fmeasure=0.)

zhouzaida · 2023-01-14T08:22:32Z

mmeval/metrics/rouge.py

+        Dict[str, float]: Calculate the score of rougeL.
+    """
+    pred_len, reference_len = len(pred), len(reference)
+    if 0 in (pred_len, reference_len):


Suggested change

if 0 in (pred_len, reference_len):

if pred_len == 0 or reference_len == 0:

zhouzaida · 2023-01-14T08:23:00Z

mmeval/metrics/rouge.py

+    pred_len, reference_len = len(pred), len(reference)
+    if 0 in (pred_len, reference_len):
+        return dict(
+            precision=float(0.0), recall=float(0.0), fmeasure=float(0.0))


Suggested change

precision=float(0.0), recall=float(0.0), fmeasure=float(0.0))

precision=0., recall=0., fmeasure=0.)

mmeval/metrics/rouge.py

zhouzaida · 2023-01-14T08:48:54Z

mmeval/metrics/rouge.py

+    reference_ngarms = _create_ngrams(reference, n_gram)
+    pred_len = sum(pred_ngarms.values())
+    reference_len = sum(reference_ngarms.values())
+    if 0 in (pred_len, reference_len):


Suggested change

if 0 in (pred_len, reference_len):

if pred_len == 0 or reference_len == 0:

zhouzaida · 2023-01-14T08:49:31Z

mmeval/metrics/rouge.py

+    reference_len = sum(reference_ngarms.values())
+    if 0 in (pred_len, reference_len):
+        return dict(
+            precision=float(0.0), recall=float(0.0), fmeasure=float(0.0))


Suggested change

precision=float(0.0), recall=float(0.0), fmeasure=float(0.0))

precision=0., recall=0.0, fmeasure=0.)

mmeval/metrics/utils/ngram_process.py

zhouzaida · 2023-01-14T08:56:57Z

mmeval/metrics/bleu.py

+        tokenizer_fn (Union[Callable, str, None]): A user's own tokenizer function.
+            Defaults to None.


Suggested change

tokenizer_fn (Union[Callable, str, None]): A user's own tokenizer function.

Defaults to None.

tokenizer_fn (Callable or str, optional): A user's own tokenizer function.

Defaults to None.

New in version 0.3.0.

zhouzaida · 2023-01-14T09:07:47Z

mmeval/metrics/utils/ngram_process.py

+    Args:
+        token (Sequence[str]): A series of tokens about sentences.
+        n_gram (int): The maximum number of words contained in a phrase
+            when calculating word fragments. Defaults to 4.


Suggested change

when calculating word fragments. Defaults to 4.

when calculating word fragments.

mmeval/metrics/rouge.py

zhouzaida · 2023-01-14T09:24:34Z

mmeval/metrics/rouge.py

+        Args:
+             predictions (Sequence[str]): An iterable of predicted sentences.
+             references (Sequence[Sequence[str]): An iterable of
+                referenced sentences.


Suggested change

referenced sentences.

referenced sentences.

mmeval/metrics/rouge.py

zhouzaida · 2023-01-14T09:35:25Z

test_get_n_gram in tests/test_metrics/test_bleu.py should be moved to tests/test_metrics/test_utils/test_ngram_process.py

tests/test_metrics/test_bleu.py

mmeval/metrics/rouge.py

mmeval/metrics/utils/ngram_process.py

mmeval/metrics/rouge.py

mmeval/metrics/bleu.py

mmeval/metrics/rouge.py

ice-tong

LGTM!

ice-tong · 2023-01-18T04:55:23Z

Suggest rename mmeval/metrics/utils/ngram_process.py -> mmeval/metrics/utils/grammar.py

mmeval/metrics/bleu.py

mmeval/metrics/rouge.py

mmeval/metrics/utils/grammar.py

mmeval/metrics/rouge.py

zhouzaida · 2023-01-25T04:33:59Z

This PR can be merged after resolving the above comments.

mmeval/metrics/rouge.py

go-with-me000 marked this pull request as draft January 3, 2023 03:25

go-with-me000 closed this Jan 3, 2023

go-with-me000 force-pushed the cky/rouge_metric branch from 961270f to 7f60d59 Compare January 3, 2023 03:52

go-with-me000 reopened this Jan 3, 2023

zhouzaida requested a review from ice-tong January 3, 2023 06:06

go-with-me000 marked this pull request as ready for review January 4, 2023 09:08

go-with-me000 force-pushed the cky/rouge_metric branch from 98ffb2e to 653c45d Compare January 5, 2023 11:11

go-with-me000 changed the title ~~Cky/rouge metric~~ Add ROUGE to mmeval Jan 9, 2023

go-with-me000 changed the title ~~Add ROUGE to mmeval~~ [Feature] Add ROUGE to mmeval Jan 9, 2023

ice-tong reviewed Jan 11, 2023

View reviewed changes

zhouzaida reviewed Jan 12, 2023

View reviewed changes

mmeval/metrics/bleu.py Outdated Show resolved Hide resolved

zhouzaida reviewed Jan 12, 2023

View reviewed changes

mmeval/metrics/rouge.py Show resolved Hide resolved

zhouzaida reviewed Jan 12, 2023

View reviewed changes

mmeval/metrics/rouge.py Show resolved Hide resolved

zhouzaida reviewed Jan 12, 2023

View reviewed changes

mmeval/metrics/rouge.py Show resolved Hide resolved

zhouzaida reviewed Jan 12, 2023

View reviewed changes

mmeval/metrics/rouge.py Outdated Show resolved Hide resolved

zhouzaida reviewed Jan 12, 2023

View reviewed changes

mmeval/metrics/rouge.py Outdated Show resolved Hide resolved

zhouzaida reviewed Jan 12, 2023

View reviewed changes

zhouzaida reviewed Jan 14, 2023

View reviewed changes

mmeval/metrics/rouge.py Show resolved Hide resolved

zhouzaida reviewed Jan 14, 2023

View reviewed changes

mmeval/metrics/utils/ngram_process.py Outdated Show resolved Hide resolved

zhouzaida reviewed Jan 14, 2023

View reviewed changes

mmeval/metrics/rouge.py Outdated Show resolved Hide resolved

zhouzaida reviewed Jan 14, 2023

View reviewed changes

mmeval/metrics/rouge.py Outdated Show resolved Hide resolved

zhouzaida reviewed Jan 14, 2023

View reviewed changes

mmeval/metrics/rouge.py Outdated Show resolved Hide resolved

zhouzaida reviewed Jan 14, 2023

View reviewed changes

mmeval/metrics/rouge.py Outdated Show resolved Hide resolved

zhouzaida reviewed Jan 14, 2023

View reviewed changes

mmeval/metrics/rouge.py Outdated Show resolved Hide resolved

zhouzaida reviewed Jan 14, 2023

View reviewed changes

mmeval/metrics/rouge.py Show resolved Hide resolved

zhouzaida requested a review from yhcc January 14, 2023 09:36

zhouzaida reviewed Jan 14, 2023

View reviewed changes

tests/test_metrics/test_bleu.py Outdated Show resolved Hide resolved

go-with-me000 closed this Jan 16, 2023

go-with-me000 force-pushed the cky/rouge_metric branch from 107cb9f to e635ba6 Compare January 16, 2023 08:19

go-with-me000 reopened this Jan 16, 2023

ice-tong reviewed Jan 17, 2023

View reviewed changes

ice-tong reviewed Jan 18, 2023

View reviewed changes

ice-tong approved these changes Jan 18, 2023

View reviewed changes

zhouzaida reviewed Jan 25, 2023

View reviewed changes

mmeval/metrics/bleu.py Outdated Show resolved Hide resolved

zhouzaida reviewed Jan 25, 2023

View reviewed changes

mmeval/metrics/rouge.py Outdated Show resolved Hide resolved

zhouzaida reviewed Jan 25, 2023

View reviewed changes

mmeval/metrics/utils/grammar.py Outdated Show resolved Hide resolved

zhouzaida reviewed Jan 25, 2023

View reviewed changes

mmeval/metrics/rouge.py Outdated Show resolved Hide resolved

zhouzaida reviewed Jan 26, 2023

View reviewed changes

mmeval/metrics/rouge.py Outdated Show resolved Hide resolved

zhouzaida approved these changes Jan 26, 2023

View reviewed changes

Add lowercase super parameter

07e20c6

go-with-me000 force-pushed the cky/rouge_metric branch from 0e4dc73 to 07e20c6 Compare January 29, 2023 01:40

go-with-me000 added 2 commits January 29, 2023 09:47

Add lowercase super parameter

c2f1941

Add lowercase super parameter

921b1dc

zhouzaida reviewed Jan 29, 2023

View reviewed changes

mmeval/metrics/rouge.py Outdated Show resolved Hide resolved

Add lowercase super parameter

f6898a0

zhouzaida approved these changes Jan 30, 2023

View reviewed changes

zhouzaida merged commit 5bd89db into open-mmlab:main Jan 30, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Add ROUGE to mmeval #72

[Feature] Add ROUGE to mmeval #72

go-with-me000 commented Jan 3, 2023

zhouzaida Jan 12, 2023 •

edited

Loading

go-with-me000 Jan 12, 2023

zhouzaida Jan 12, 2023

go-with-me000 Jan 12, 2023

zhouzaida Jan 14, 2023

zhouzaida Jan 14, 2023

zhouzaida Jan 14, 2023

zhouzaida Jan 14, 2023

zhouzaida Jan 14, 2023

zhouzaida Jan 14, 2023 •

edited

Loading

zhouzaida Jan 14, 2023

zhouzaida Jan 14, 2023

zhouzaida commented Jan 14, 2023 •

edited

Loading

ice-tong left a comment

ice-tong commented Jan 18, 2023

zhouzaida commented Jan 25, 2023

	precision=float(0.0), recall=float(0.0), fmeasure=float(0.0))
	precision=0., recall=0., fmeasure=0.)

	if 0 in (pred_len, reference_len):
	if pred_len == 0 or reference_len == 0:

	precision=float(0.0), recall=float(0.0), fmeasure=float(0.0))
	precision=0., recall=0.0, fmeasure=0.)

		tokenizer_fn (Union[Callable, str, None]): A user's own tokenizer function.
		Defaults to None.

	when calculating word fragments. Defaults to 4.
	when calculating word fragments.

[Feature] Add ROUGE to mmeval #72

[Feature] Add ROUGE to mmeval #72

Conversation

go-with-me000 commented Jan 3, 2023

Motivation

Modification

Checklist

zhouzaida Jan 12, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhouzaida Jan 14, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhouzaida commented Jan 14, 2023 • edited Loading

ice-tong left a comment

Choose a reason for hiding this comment

ice-tong commented Jan 18, 2023

zhouzaida commented Jan 25, 2023

zhouzaida Jan 12, 2023 •

edited

Loading

zhouzaida Jan 14, 2023 •

edited

Loading

zhouzaida commented Jan 14, 2023 •

edited

Loading