Defining iterator in a separate class no longer works in 3.13

[antonio-rojas]

Bug report

Bug description:

Defining an interator for a class in a separate class no longer works properly in 3.13. With the following test_iter.py:

class list2(list):
    def __iter__(self):
        return list2iterator(self)

class list2iterator:
    def __init__(self, X):
        self._X = X
        self._pointer = -1

    def __next__(self):
        self._pointer += 1
        if self._pointer == len(self._X):
            self._pointer = -1
            raise StopIteration
        return self._X[self._pointer]

With Python 3.13.1 one gets:

>>> from test_iter import list2
>>> X=list2([1,2,3])
>>> [x for x in X]
Traceback (most recent call last):
  File "<python-input-2>", line 1, in <module>
    [x for x in X]
                ^
TypeError: 'list2iterator' object is not iterable

With Python 3.12.7 it works:

>>> from test_iter import list2
>>> X=list2([1,2,3])
>>> [x for x in X]
[1, 2, 3]

Bisected to bcc7227e

CPython versions tested on:

3.12, 3.13

Operating systems tested on:

Linux

Linked PRs

• [3.14] GH-128161: Remove redundant GET_ITER from list comprehension code #134778
• [3.13] gh-128161: Remove redundant GET_ITER from list comprehension code (GH-134778) #134788
• [3.13] GH-128161: Fix refleak introduced in GH-134788 #134799

[Eclips4]

cc @efimov-mikhail @markshannon @JelleZijlstra

[ronaldoussoren]

https://docs.python.org/3/library/stdtypes.html#iterator-types says that iterators must also implement __iter__ (returning self). That's a long standing requirement on iterators, and not one introduced in 3.13.

[antonio-rojas]

Alright, thanks

[Eclips4]

@ronaldoussoren You're right, and this code is definitely doing something weird, but it works in 3.12:
example.py:

class list2(list):
    def __iter__(self):
        return list2iterator(self)

class list2iterator:
    def __init__(self, X):
        self._X = X
        self._pointer = -1

    def __next__(self):
        self._pointer += 1
        if self._pointer == len(self._X):
            self._pointer = -1
            raise StopIteration
        return self._X[self._pointer]


target = list2([1, 2, 3])
for i in target:
    print(i)


[print(i) for i in target]

3.12.8:

eclips4@suffering ~/tmp> python3.12 example.py
1
2
3
1
2
3

3.13.1:

1                                                                                  
2
3
Traceback (most recent call last):
  File "/Users/eclips4/tmp/example.py", line 23, in <module>
    [print(i) for i in target]
                       ^^^^^^
TypeError: 'list2iterator' object is not iterable

Though it probably shouldn't work, it is definitely a regression.

Bytecode for [x for x in y] 3.12.8:

  0           0 RESUME                   0

  1           2 LOAD_NAME                0 (y)
              4 GET_ITER
              6 LOAD_FAST_AND_CLEAR      0 (x)
              8 SWAP                     2
             10 BUILD_LIST               0
             12 SWAP                     2
        >>   14 FOR_ITER                 4 (to 26)
             18 STORE_FAST               0 (x)
             20 LOAD_FAST                0 (x)
             22 LIST_APPEND              2
             24 JUMP_BACKWARD            6 (to 14)
        >>   26 END_FOR
             28 SWAP                     2
             30 STORE_FAST               0 (x)
             32 RETURN_VALUE
        >>   34 SWAP                     2
             36 POP_TOP
             38 SWAP                     2
             40 STORE_FAST               0 (x)
             42 RERAISE                  0
ExceptionTable:
  10 to 26 -> 34 [2]

3.13.1:

   0           RESUME                   0

   1           LOAD_NAME                0 (y)
               GET_ITER
               LOAD_FAST_AND_CLEAR      0 (x)
               SWAP                     2
       L1:     BUILD_LIST               0
               SWAP                     2
               GET_ITER  - problematic instruction
       L2:     FOR_ITER                 4 (to L3)
               STORE_FAST_LOAD_FAST     0 (x, x)
               LIST_APPEND              2
               JUMP_BACKWARD            6 (to L2)
       L3:     END_FOR
               POP_TOP
       L4:     SWAP                     2
               STORE_FAST               0 (x)
               RETURN_VALUE

  --   L5:     SWAP                     2
               POP_TOP

   1           SWAP                     2
               STORE_FAST               0 (x)
               RERAISE                  0
ExceptionTable:
  L1 to L4 -> L5 [2]

[ronaldoussoren]

@ronaldoussoren You're right, and this code is definitely doing something weird, but it works in 3.12: example.py:

Right, the code from the OP is broken according to the data model, but we still did a change that's not backward compatible. I'm not deep enough into this to fully understand the ramifications.

[efimov-mikhail]

This behavior change is a direct consequence of change in bytecode generation in my PR. I don't think this is a bug.

[terryjreedy]

The exact example worked in the CPython implementation of 3.12, but that may be an accident, as it should not have by the language definition. It might have already failed in 3.12 in other implementations. In any case, adding the following exposes the defect in list2iterable even in 3.12.

target2 = iter(target)  # A list2iterator object
target3 = iter(target2)  # Fails in 3.12.8 
# TypeError: 'list2iterator' object is not iterable

If there is still a question about changing anything, Guido might remember the original design discussions.

[pochmann3]

Relevant bit from glossary: iterator:

CPython implementation detail: CPython does not consistently apply the requirement that an iterator define __iter__().