Skip to content

the result of sumeval is different from another popular rouge package #28

@imempty

Description

@imempty

below is my code using sumeval:

from sumeval.metrics.rouge import RougeCalculator

rouge = RougeCalculator(stopwords=True, lang="zh")

reference = "刚刚过去的这个月,美股总市值暴跌了将近6万亿美元(折合人民币超过40万亿),这背后的原因可能不仅仅是加息这么简单。最近瑞士信贷知名分析师Zoltan Polzsar撰写了一篇极其重要的文章,详细分析了现有世界秩序的崩坏本质以及美国和西方将要采取的应对策略。在该文中,Zoltan Polzsar直指美国通胀的本质和其长期性。同期,A股市场亦出现了大幅杀跌的情况。"
candidate = "###刚刚发声,A股这种情况十分罕见!大聪明逆市抄底330亿,一篇研报引爆全球,市场逻辑生变?"

rouge_1 = rouge.rouge_n(
summary=candidate,
references=reference,
n=1)

print("ROUGE-1: {}".format(rouge_1).replace(", ", "\n"))

result:ROUGE-1: 0.14925373134328357

====================================================================

rouge版本

from rouge import Rouge
import jieba

def preprocess_text(text):
words = jieba.lcut(text)
processed_text = ' '.join(words)
return processed_text

reference = "刚刚过去的这个月,美股总市值暴跌了将近6万亿美元(折合人民币超过40万亿),这背后的原因可能不仅仅是加息这么简单。最近瑞士信贷知名分析师Zoltan Polzsar撰写了一篇极其重要的文章,详细分析了现有世界秩序的崩坏本质以及美国和西方将要采取的应对策略。在该文中,Zoltan Polzsar直指美国通胀的本质和其长期性。同期,A股市场亦出现了大幅杀跌的情况。"
candidate = "###刚刚发声,A股这种情况十分罕见!大聪明逆市抄底330亿,一篇研报引爆全球,市场逻辑生变?"

processed_reference = preprocess_text(reference)
processed_candidate = preprocess_text(candidate)

rouge = Rouge()

scores = rouge.get_scores(processed_candidate, processed_reference, avg=True)
print(scores)

result:
{'rouge-1': {'r': 0.08571428571428572, 'p': 0.25, 'f': 0.12765957066545958}, 'rouge-2': {'r': 0.011235955056179775, 'p': 0.04, 'f': 0.017543856224992972}, 'rouge-l': {'r': 0.05714285714285714, 'p': 0.16666666666666666, 'f': 0.08510637917609795}}

so f1 is 0.12765957066545958

why they are different?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions