🍐 PEAR: Pairwise Evaluation for Automatic Relative Scoring in Machine Translation

This repository provides the inference toolkit for 🍐PEAR: Pairwise Evaluation for Automatic Relative Scoring in Machine Translation.

PEAR reframes reference-free machine translation evaluation as a graded pairwise comparison: given a source segment and two candidate translations, it predicts both the direction and magnitude of their quality difference. The toolkit also supports reference-anchored PEAR scoring and PEAR-based minimum Bayes risk (MBR) decoding.

• 📄 Paper: ACL Anthology
• 🤗 Models and resources: PEAR Hugging Face collection

Installation

PEAR supports Python 3.11, 3.12, and 3.13. Install the published package from PyPI:

python -m pip install pear-mt

The PyPI distribution is named pear-mt; the Python import package is named pear, and the primary command-line program is also pear:

import pear

To install from a local clone instead:

git clone https://github.com/prosho-97/pear.git
cd pear
python -m pip install .

For local development, install the development extra too:

python -m pip install -e ".[dev]"

Loading a model

import pear

metric = pear.load_metric("pear")       # resolves to Prosho/pear on Hugging Face
metric_xl = pear.load_metric("pear-xl") # resolves to Prosho/pear-xl on Hugging Face

You can also pass a Hugging Face repository ID or a local PEAR model checkpoint path:

metric = pear.load_metric("Prosho/pear")
metric = pear.load_metric("/path/to/checkpoints/model.ckpt")

Reproducible model revisions

The official pear and pear-xl aliases use checkpoint and encoder revisions pinned by this package. This keeps their results stable even when files on the Hugging Face Hub change.

The aliases and their full Prosho/pear and Prosho/pear-xl repository IDs load those immutable official snapshots as trusted Lightning checkpoints. PEAR therefore suppresses PyTorch's full-checkpoint deserialization warning for those exact revisions. Local checkpoints, third-party repositories, and revision overrides retain PyTorch's default security behavior.

For another Hugging Face repository, PEAR follows that repository's default branch unless you provide revisions explicitly. revision pins the PEAR checkpoint repository; encoder_revision pins the underlying Transformers encoder. A full commit hash gives the strongest reproducibility:

metric = pear.load_metric(
    "organization/custom-pear",
    revision="<checkpoint-commit>",
    encoder_revision="<encoder-commit>",
)

The same controls are available from the CLI:

pear score \
    --hf-model organization/custom-pear \
    --revision <checkpoint-commit> \
    --encoder-revision <encoder-commit> \
    --input pairs.tsv \
    --output scored.tsv

Passing either option overrides the corresponding default. Local checkpoint paths do not use a Hub checkpoint revision, although encoder_revision can still pin an encoder that the checkpoint configuration loads from the Hub.

Pairwise QE scoring

scores = pear.score_pairwise(
    metric,
    sources=["The cat is on the mat."],
    translations_a=["El gato está en la alfombra."],
    translations_b=["El gato está en el mapa."],
)

Positive scores mean translations_a is preferred over translations_b.

To score both candidate orders:

scores = pear.score_pairwise(
    metric,
    sources=["The cat is on the mat."],
    translations_a=["El gato está en la alfombra."],
    translations_b=["El gato está en el mapa."],
    mode="both",
)
# {"forward": [...], "reverse": [...]}

Reference-anchored scoring

Reference-anchored PEAR uses the human reference (or any other list of candidate translations) as the second candidate/anchor.

scores = pear.score_reference_anchored(
    metric,
    sources=["The cat is on the mat."],
    translations=["El gato está en la alfombra."],
    references=["El gato está sobre la alfombra."],
)

As with pairwise QE scoring, set mode="both" to score both candidate orders for reference-anchored inputs.

PEAR for MBR decoding

from pear.mbr import pear_utility_matrix, select_mbr_hypothesis

hyps = ["This is a good translation.", "This is a very good translation.", "This is a bad translation."]
utility = pear_utility_matrix(metric, "Questa è una traduzione molto buona.", hyps, mode="half")
index, expected_utility = select_mbr_hypothesis(utility)
best = hyps[index]
# best -> "This is a very good translation."

Use mode="full" to score all off-diagonal ordered pairs (N² - N comparisons). Use mode="half" to score one triangular half and fill the opposite direction by PEAR antisymmetry.

Batching, devices, and progress bars

Python and CLI inference are batched. Use batch_size / --batch-size to control the number of pairwise examples per PEAR forward batch.

By default, PEAR uses:

gpus="auto"

This selects one available CUDA/MPS accelerator when present. Pass gpus=0 or --gpus 0 to force CPU. Progress bars are shown by default; in Python calls, control them with the progress_bar argument (progress_bar=True by default, or progress_bar=False to disable them).

CLI

Score a TSV file

Pairwise input TSV must contain src, mt_0, and mt_1 columns:

pear score --model pear --input pairs.tsv --output scored.tsv --batch-size 16
pear score --hf-model Prosho/pear --input pairs.tsv --output scored.tsv --batch-size 16
pear score --checkpoint /path/to/model.ckpt --input pairs.tsv --output scored.tsv --batch-size 16

Reference-anchored input TSV must contain src, mt, and ref columns:

pear score --model pear --mode reference --input refs.tsv --output scored.tsv --batch-size 16

Add --both to score both candidate orders.

Run MBR

Input JSONL rows must contain src and hypotheses:

{"src": "source text", "hypotheses": ["hyp A", "hyp B"]}

Run half- or full-matrix PEAR MBR:

pear mbr --model pear --input nbest.jsonl --output selected.jsonl --utility half --batch-size 16
pear mbr --hf-model Prosho/pear-xl --input nbest.jsonl --output selected.jsonl --utility full --batch-size 16
pear mbr --checkpoint /path/to/model.ckpt --input nbest.jsonl --output selected.jsonl --batch-size 16

Development

Install the development extra before running formatter checks:

python -m pip install -e ".[dev]"

Then run:

python -m black --check pear tests
python -m pytest
python -m compileall pear

Network tokenizer regressions use immutable encoder snapshots and run in CI against both Transformers 4.40.2 and the latest supported 5.x release. The multi-gigabyte checkpoint regression remains opt-in. Before creating a release tag, record its baseline under Transformers 4.40.2:

python -m pytest \
    tests/test_regressions.py::test_released_checkpoints_match_440_scores_and_key_mismatches \
    --run-checkpoint-integration \
    --write-checkpoint-baseline=/tmp/pear-checkpoints-4.40.2.json

Then, on the same hardware and dtype under the latest Transformers 5.x, compare against that file:

python -m pytest \
    tests/test_regressions.py::test_released_checkpoints_match_440_scores_and_key_mismatches \
    --run-checkpoint-integration \
    --checkpoint-baseline=/tmp/pear-checkpoints-4.40.2.json

The comparison requires the same checkpoint-key mismatch sets, finite scores, preference directions and rankings, and scores within rtol=1e-4 and atol=1e-4.

Citation

PEAR has been published at ACL 2026 (Main Conference). If you use it, please consider citing our paper as follows:

@inproceedings{proietti-etal-2026-pear,
    title = "{PEAR}: Pairwise Evaluation for Automatic Relative Scoring in Machine Translation",
    author = "Proietti, Lorenzo  and
      Grundkiewicz, Roman  and
      Post, Matt",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.acl-long.1953/",
    doi = "10.18653/v1/2026.acl-long.1953",
    pages = "42189--42207",
    ISBN = "979-8-89176-390-6"
}

License

This project is released under the Apache License 2.0. See LICENSE for the full license text.