approx representation of details failed when using a ApproxSequenceLike which is not list or tuple

[thorink]

Description

When using a custom sequence like type then the comparison of pytest.approx works as expected if the assert is True. But in case the assertion fails, the detailed output of what went wrong is broken.

Given the following example:

import pytest

class MyVec3:
    def __init__(self, x, y, z):
        self._x, self._y, self._z = x, y, z

    def __len__(self):
        return 3

    def __getitem__(self, key):
        if key == 0:
            return self._x
        if key == 1:
            return self._y
        if key == 2:
            return self._z
        raise IndexError


def test_vec_approx():
    assert MyVec3(0, 1, 2) == pytest.approx(MyVec3(1, 2, 3), abs=2)
    assert MyVec3(0, 1, 2) == pytest.approx(MyVec3(1, 2, 3), abs=0.5)

The first assert statement is fine, as expected.

In the second assert fails, as expected.

But I get this output:

============================================ test session starts ============================================
platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.3.0
rootdir: /home/user
collected 1 item                                                                                            

test_approx.py F                                                                                      [100%]

================================================= FAILURES ==================================================
______________________________________________ test_vec_approx ______________________________________________

    def test_vec_approx():
        assert MyVec3(0, 1, 2) == pytest.approx(MyVec3(1, 2, 3), abs=2)
>       assert MyVec3(0, 1, 2) == pytest.approx(MyVec3(1, 2, 3), abs=0.5)
E       AssertionError: assert <test_approx....x7f657f208880> == approx([1 ± 5... 3 ± 5.0e-01])
E         (pytest_assertion plugin: representation of details failed: /home/user/venv/lib/python3.10/site-packages/_pytest/python_api.py:342: TypeError: object of type 'ApproxScalar' has no len().
E          Probably an object has a faulty __repr__.)

test_approx.py:22: AssertionError
========================================== short test summary info ==========================================
FAILED test_approx.py::test_vec_approx - AssertionError: assert <test_approx....x7f657f208880> == approx([1 ± 5... 3 ± 5.0e-01])
============================================= 1 failed in 0.01s =============================================

Setup

Package        Version
-------------- -------
exceptiongroup 1.2.0
iniconfig      2.0.0
packaging      23.2
pip            22.0.2
pluggy         1.3.0
pytest         7.4.4
setuptools     59.6.0
tomli          2.0.1

Also happens with pytest version 7.1.2 and 7.1.3

OS is Ubuntu 22.04

[thorink]

I digged into it a bit. pytest.approx(MyVec3(1, 2, 3), abs=0.5) results in an object of type ApproxSequenceLike which sound right to me. This is the check that is responsible for that:

    elif (
        hasattr(expected, "__getitem__")
        and isinstance(expected, Sized)
        # Type ignored because the error is wrong -- not unreachable.
        and not isinstance(expected, STRING_TYPES)  # type: ignore[unreachable]
    ):
        cls = ApproxSequenceLike

When generating the detailed output the MyVec3 is wrapped in a ApproxScalar. When doing the same with a list [1,2,3], then it is transformed into [ApproxScalar(1), ApproxScalar(2), ApproxScalar(3)]. This happens in python_api.py::_recursive_sequence_map:

    if isinstance(x, (list, tuple)):
        seq_type = type(x)
        return seq_type(_recursive_sequence_map(f, xi) for xi in x)
    else:
        return f(x)

This isinstance(x, (list, tuple) is much more narrow as hasattr(expected, "__getitem__") and isinstance(expected, Sized) and not isinstance(expected, STRING_TYPES) from above.

[thorink]

Replacing _recursive_sequence_map with

def _recursive_sequence_map(f, x):
    """Recursively map a function over a sequence of arbitrary depth"""
    if (
        hasattr(x, "__getitem__")
        and isinstance(x, Sized)
        # Type ignored because the error is wrong -- not unreachable.
        and not isinstance(x, STRING_TYPES)  # type: ignore[unreachable]
    ):
        if isinstance(x, (list, tuple)):
            seq_type = type(x)
        else:
            seq_type = list
        return seq_type(_recursive_sequence_map(f, xi) for xi in x)
    else:
        return f(x)

would result in

    def test_vec_approx():
        assert MyVec3(0, 1, 2) == pytest.approx(MyVec3(1, 2, 3), abs=2)
>       assert MyVec3(0, 1, 2) == pytest.approx(MyVec3(1, 2, 3), abs=0.5)
E       assert <test_approx....x7f9c09e6ca60> == approx([1 ± 5... 3 ± 5.0e-01])
E         comparison failed. Mismatched elements: 3 / 3:
E         Max absolute difference: 1
E         Max relative difference: inf
E         Index | Obtained | Expected   
E         0     | 0        | 1 ± 5.0e-01
E         1     | 1        | 2 ± 5.0e-01
E         2     | 2        | 3 ± 5.0e-01

[RonnyPfannschmidt]

Approx is not general,the support for lists as is, is already a stretch

Custom collection objects are not easily supportable

[thorink]

Hi @RonnyPfannschmidt, I see your point and I'm of course totally fine with not changing code for a stretch use case! We would then just implement our own MyVec approx type.

On the other hand, approx accepts as input (hasattr(x, "__getitem__") and isinstance(x, Sized)) and also works as expected in terms of the comparison it performs. My idea would be to just make sure the __repr__ of ApproxSequenceLike is on the same level.

If applying such a change would be considered, I would be happy to work on a proper MR.

[RonnyPfannschmidt]

I need to do a little digging today,but it seems like a Good idea, I'll try to get it together soon

[RonnyPfannschmidt]

i'm slightly hesitant to transform any custom collection into a list
however since its only happen on the internal part of the compare we might be fine

that being said, the rep traceback shouldn't happen as well, i'll try to figure the exact exception

[RonnyPfannschmidt]

self = approx([1 ± 5.0e-01, 2 ± 5.0e-01, 3 ± 5.0e-01])
other_side = <approx.TestApprox.test_strange_sequence.<locals>.MyVec3 object at 0x7f8237a32d50>

    def _repr_compare(self, other_side: Sequence[float]) -> List[str]:
        import math
    
        if len(self.expected) != len(other_side):
            return [
                "Impossible to compare lists with different sizes.",
                f"Lengths: {len(self.expected)} and {len(other_side)}",
            ]
    
        approx_side_as_map = _recursive_sequence_map(self._approx_scalar, self.expected)
    
>       number_of_elements = len(approx_side_as_map)
E       TypeError: object of type 'ApproxScalar' has no len()

../../src/_pytest/python_api.py:332: TypeError

i have a error indicating reproducer i'll extract a is_sequencelike and use that