Pylance gives a false positive for fdst argument to shutil.copyfileobj

[willfrey]

Environment data

• Language Server version: 2020.10.0
• OS and version: macOS Catalina Version 10.15.7
• Python version (& distribution if applicable, e.g. Anaconda): Python 3.8.5

Expected behaviour

I don't expect Pylance to report an issue.

Actual behaviour

Consider the following code snippet:

import shutil
from pathlib import Path

source_path = Path("/tmp/source")
source_path.write_text("Hello there!")

destination_path = Path("/tmp/destination")

with source_path.open("rb") as source, destination_path.open("wb") as destination:
    shutil.copyfileobj(source, destination)

assert destination_path.read_text() == "Hello there!"

Pylance will complain about the destination argument to shutil.copyfileobj and raise the following warning:

Argument of type "BufferedWriter" cannot be assigned to parameter "fdst" of type "SupportsWrite[TypeVar('AnyStr')]" in function "copyfileobj"
  TypeVar "_T_contra" is contravariant
    Type "bytes | bytearray | memoryview | array[_T] | mmap" is not compatible with constrained type "AnyStr"

I think this should be fine, though.

Pylance also complains about the builtin open(...) function, which you can see with the following snippet:

import shutil

source_path = "/tmp/source"
with open(source_path, "w") as source:
    source.write("Hello there!")

destination_path = "/tmp/destination"
with open(source_path, "rb") as source, open(destination_path, "wb") as destination:
    shutil.copyfileobj(source, destination)

with open(destination_path) as destination:
    assert destination.read() == "Hello there!"

Logs

I don't think logs will be helpful here but I'm happy to include them if you'd like.

Code Snippet / Additional information

If you pass an integer larger than 1 to the buffering argument to either open(...) or the instance method pathlib.Path.open(...) then the Pylance stubs infer that the returned object is just BinaryIO, which makes Pylance happy. Any of the other return types for the overloads of either function/method make Pylance complain.

I can also make Pylance happy with an explicit cast(BinaryIO, destination) but that seems to defeat the point of type checking catching errors.

[erictraut]

Thanks for filing the issue.

The type checker is doing the correct thing based on the type information in the stubs. I've confirmed that mypy (another type checker) also reports the same issue. The problem is that shutil.copyfileobj requires a destination that supports writing of strings (strs or bytes). Specifically, it requires an argument that satisfies the generic protocol SupportsWrite[AnyStr]. You are passing it a BufferedWriter object, which does not meet those requirements.

These stubs are from the typeshed repo. If you think that the stub for shutil is incorrect, please file a bug or submit a PR there. Pylance picks up new versions of the typeshed stubs regularly.

[willfrey]

Thanks for the response!

I'll check out the typeshed repo because I think this should work based on the examples of shutil.copyfileobj I see in the CPython repository, such as zipapp.py.

[willfrey]

It looks like this is fixed in mypy 0.790.

See python/mypy#8962

[jakebailey]

That closed issue links python/typeshed#4203, which was merged many months ago, and we've definitely had that in our copy of typeshed for a good while. I'm not sure how this would be fixed in mypy but not us unless there is a behavior difference between mypy and pyright. That linked issue also is talking about failing inference, but our message is different and mentions the types not matching, so I'm not certain it's exactly the same.

[jakebailey]

Revisiting this; I think this may only work in mypy because they special case some of the IO oddities; typeshed's IO types predate Protocol and different stubs use different types (leading to problems like this). I'd have to find the typeshed issue tracking the IO changes, but as far as I can tell we're still doing "the right thing" when reading these stubs (and special casing things isn't really something we like to have to do...).

[jakebailey]

This one's been fixed for a while; typeshed converted a whole load of stubs that were using IO to the more permissive SupportsRead and SupportsWrite protocols. All of the code in the original issue now typecheck.