测试集上复现的分数(TEDS)极低

[vivavilavida]

大佬们好👋我在复现TEDS评分的时候得到的分数和仓库首页上面的分数相差极大，如下图所示：
我的结果，TEDS(structure_only): 0.34447

仓库页面结果TEDS(structure_only): 0.84580

纯CPU运行，平台为google colab, Python 3.11.12
表格OCR代码来自repo首页 https://github.com/RapidAI/TableStructureRec
评测代码来自 https://github.com/SWHL/TableRecognitionMetric
代码和结果分别如下：

from pathlib import Path
from wired_table_rec.utils.utils import VisTable
from table_cls import TableCls
from wired_table_rec.main import WiredTableInput, WiredTableRecognition
from lineless_table_rec.main import LinelessTableInput, LinelessTableRecognition
from rapidocr import RapidOCR

# table OCR init
ocr_engine = RapidOCR()
wired_input = WiredTableInput()
lineless_input = LinelessTableInput()
wired_engine = WiredTableRecognition(wired_input)
lineless_engine = LinelessTableRecognition(lineless_input)
viser = VisTable()
table_cls = TableCls()

def table_OCR(img_path):
    cls, elasp = table_cls(img_path)
    if cls == "wired":
        table_engine = wired_engine
    else:
        table_engine = lineless_engine
    # 使用RapidOCR输入
    rapid_ocr_output = ocr_engine(img_path, return_word_box=True)
    ocr_result = list(zip(rapid_ocr_output.boxes, rapid_ocr_output.txts, rapid_ocr_output.scores))
    table_results = table_engine(img_path, ocr_result=ocr_result, enhance_box_line=True)
    return table_results.pred_html

################     Run on Dataset      #################
from datasets import load_dataset
from rapid_table import RapidTable
from tqdm import tqdm
from table_recognition_metric import TEDS

dataset = load_dataset("SWHL/table_rec_test_dataset")
test_data = dataset["test"]

teds = TEDS(structure_only=True)

content = []
i = 0
for one_data in tqdm(test_data):
    i+=1
    print(f"\n========now the turn is {i}=========")
    img = one_data.get("image")
    gt = one_data.get("html")

    pred_str = table_OCR(np.array(raw))

    scores = teds(gt, pred_str)
    content.append(scores)

    avg = sum(content) / len(content)
    print(f"TEDS: {avg:.5f}")

  0%|          | 0/93 [00:00<?, ?it/s]
========now the turn is 1=========
  1%|          | 1/93 [00:13<20:48, 13.57s/it]
TEDS: 0.30120
========now the turn is 2=========
  2%|▏         | 2/93 [00:27<20:38, 13.61s/it]
TEDS: 0.59337
========now the turn is 3=========
  3%|▎         | 3/93 [00:40<20:31, 13.68s/it]
TEDS: 0.65663

......

========now the turn is 90=========
 97%|█████████▋| 90/93 [20:00<00:39, 13.23s/it]
TEDS: 0.34732
========now the turn is 91=========
 98%|█████████▊| 91/93 [20:13<00:26, 13.26s/it]
TEDS: 0.34535
========now the turn is 92=========
 99%|█████████▉| 92/93 [20:27<00:13, 13.29s/it]
TEDS: 0.34291
========now the turn is 93=========
100%|██████████| 93/93 [20:41<00:00, 13.35s/it]
TEDS: 0.34447

[Joker1212]

数据集中有些图片是有扭曲和透视的，没有前置处理的话，指标会很差

[Joker1212]

emmm,这个测试结果是基于上一次改变测得的，本次改动应该不会影响，具体原因我需要排查一下，肯定是不正常的哈

[vivavilavida]

好的谢谢作者

[Joker1212]

我最新的结果是这样，但是我是本地有数据集，直接传的img_path,我也用你的方法测试了一下，发现确实有差别，但是也在可控范围内

from pathlib import Path
from wired_table_rec.utils.utils import VisTable
from table_cls import TableCls
from wired_table_rec.main import WiredTableInput, WiredTableRecognition
from lineless_table_rec.main import LinelessTableInput, LinelessTableRecognition
from rapidocr import RapidOCR

table OCR init

ocr_engine = RapidOCR()
wired_input = WiredTableInput()
lineless_input = LinelessTableInput()
wired_engine = WiredTableRecognition(wired_input)
lineless_engine = LinelessTableRecognition(lineless_input)
viser = VisTable()
table_cls = TableCls()

def table_OCR(img_path):
cls, elasp = table_cls(img_path)
if cls == "wired":
table_engine = wired_engine
else:
table_engine = lineless_engine
# 使用RapidOCR输入
rapid_ocr_output = ocr_engine(img_path, return_word_box=True)
ocr_result = list(zip(rapid_ocr_output.boxes, rapid_ocr_output.txts, rapid_ocr_output.scores))
table_results = table_engine(img_path, ocr_result=ocr_result, enhance_box_line=True)
return table_results.pred_html

################ Run on Dataset #################
from datasets import load_dataset
from tqdm import tqdm
from table_recognition_metric import TEDS

dataset = load_dataset("SWHL/table_rec_test_dataset")
test_data = dataset["test"]

teds = TEDS(structure_only=True)

content = []
i = 0
for one_data in tqdm(test_data):
i+=1
print(f"\n========now the turn is {i}=========")
img = one_data.get("image")
gt = one_data.get("html")
img = np.array(img)
pred_str = table_OCR(img)

scores = teds(gt, pred_str)
content.append(scores)

avg = sum(content) / len(content)
print(f"TEDS: {avg:.5f}")