Check file type annotations against open mode

[flother]

I have a function argument whose type I want to limit to a file object that's open for binary reading — i.e. I want only an instance of io.BufferedReader as returned by open() when mode="rb".

from typing import IO


def parser(file_obj: IO[bytes]) -> None:
    pass

As I would expect, mypy runs without complaint on this code:

parser(open("example.bin", "rb"))

But it doesn't output an error for this:

parser(open("example.txt", "rt"))

In that case, I would want an error along the lines of Argument 1 to "parser" has incompatible type IO[str]; expected "IO[bytes]".

[gvanrossum]

This would probably have to wait for some kind of "dependent types", unless we decide that `open()` is so important we write custom code for it in mypy. (Which IMO isn't totally unreasonable given that it's a builtin.)

[JukkaL]

I think that special casing open and a few other stdlib functions would be totally reasonable. We already special case the string % operator so there is even a precedent.

[illume]

Hello,

I was trying to type check if the file was open in binary mode too, and noticed a few things (and a workaround right at the end of all this)...

mypy -2 doesn't seem to know that io.BufferedReader is there.

io.BufferedReader
afile = io.open('/etc/passwd', 'rb')

def read_data(f):
    # type: (io.BufferedReader) -> bytes
    return f.read()

bytes_issue.py:2: error: "module" has no attribute "BufferedReader"
bytes_issue.py:5: error: Name 'io.BufferedReader' is not defined

If I run mypy in python3 mode, it passes.

Also if I change it to use _io, it works under mypy -2, but not in python3 mode.

_io.BufferedReader
afile = _io.open('/etc/passwd', 'rb')
def read_data(f):
    # type: (_io.BufferedReader) -> bytes
    return f.read()

With mypy in 3 mode I get:

bytes_issue.py:1: error: Cannot find module named '_io'

In python python2.7 and 3.6, the types for io.open are like so.

>>> type(io.open('/etc/passwd', 'r'))
<type '_io.TextIOWrapper'>
>>> type(io.open('/etc/passwd', 'rb'))
<type '_io.BufferedReader'>

Hacking about some more I make this mess below which does let me check that every file obj passed into the object is a BufferedReader. Of course, I need to manually cast every instance... not at all nice.

import io, sys
from typing import cast
PY3 = sys.version_info > (3,)
if PY3:
    from io import BufferedReader
else:
    from _io import BufferedReader
afile = io.open('/etc/passwd', 'rb')
def read_data(f):
    # type: (BufferedReader) -> bytes
    return f.read()
read_data(cast(BufferedReader, afile))
#The following code will give a type error, as I haven't cast it to a binary file.
#another_file = io.open('/etc/passwd', 'r')
#read_data(afile)

Finally I decide to have a separate open function wrapper for every mode. This way I can use a cast inside it, and the type checker works. Any file not using my custom wrappers also fails - which is great because I know where I have to track things down to check them.

import io, sys
from typing import cast
PY3 = sys.version_info > (3,)

if PY3:
    from io import BufferedReader, TextIOWrapper
else:
    from _io import BufferedReader, TextIOWrapper

def my_io_open_r(fname): # type: (str) -> TextIOWrapper
    return cast(TextIOWrapper, io.open(fname, 'r'))
def my_io_open_rb(fname): # type: (str) -> BufferedReader
    return cast(BufferedReader, io.open(fname, 'rb'))

def read_data(f): # type: (BufferedReader) -> bytes
    return f.read()

read_data(my_io_open_rb('/etc/passwd'))

# below fails, as it is opened with 'r'
# read_data(my_io_open_r('/etc/passwd'))

# below also fails type check with a got IO[Any] error.
# read_data(io.open('/etc/passwd', 'r'))

I think this technique would work anywhere you have a factory pattern where the return types are dependent on logic inside the function, or the parameters. As long as number of combinations are not too big that is! (open_rb, open_r, open_w, open_wb)

cheers,

[JukkaL]

#3299 has fixed this for calls where the mode argument is either a literal or not explicitly provided.

[gvanrossum]

Actually some of @illume's issues (e.g. io.BufferedReader doesn't exist in PY2) are pure typeshed issues. @illume Could you open a new issue in typeshed about this (and the other issues you've found related to missing classes in io.pyi)?