Idea: something like PreservedIterator

[sobolevn]

Sometimes I have several layers for Iterator calls, which look like this:

def first() -> Iterator[int]:
    yield from range(5)

def second() -> Iterator[int]:
    yield from first()

def third() -> Iterator[int]:
    yield from second()

Sometimes, people can break things unintentionally. For example, imagine that second has this line now:

def second() -> Iterator[int]:
    gen = first()
    print(next(gen))
    yield from gen

Now, second will break the chain and produce one value less than first. Sometimes we want to explicitly disallow that.
Is there a way to express this using phantom-types? If no, let's consider adding a support for it:

def second() -> PreservedIterator[int]:
    gen = first()
    print(next(gen))  # error
    yield from gen

[antonagestam]

Hi @sobolevn 👋

I agree that it would be immensely valuable to have something like this. But, off the top of my head, I can't come up with a way to express this using phantom types. I'll elaborate a little bit.

Phantom types aren't really compatible with mutable data structures, as they can be thought of as recording a current state in the type system. If the state of the observed structure can change, the type system will not have an accurate depiction of the program state any more.

A construct like this would need to understand that applying a certain mutating operation to an instance changes its type, so that after applying next() to the value it becomes Iterator instead of PreservedIterator. The only thing that works similar to that in mypy are type guards. Perhaps it could be achieved when type refinement in function calls are shipped in mypy? python/mypy#5206

Do you have more ideas about how this could be implemented?

[antonagestam]

A related note is that, given that we were able to make next() change the type of the iterator variable, I think PreservedIterator would need to be a super-type to Iterator? Given that type guards really only should narrow types and not broaden them. Perhaps that would be achievable using ABC.register()?

Edit: not sure about this :)

Edit: Seems like PEP 647 will not strictly require narrowing: https://www.python.org/dev/peps/pep-0647/#enforcing-strict-narrowing