Protocols

docs/source/class_basics.rst

@@ -151,7 +151,236 @@ concrete. As with normal overrides, a dynamically typed method can
 implement a statically typed abstract method defined in an abstract
 base class.
 
+.. _protocol-types:
+
+Protocols and structural subtyping
+**********************************
+
+.. note::
+
+   The support for structural subtyping is still experimental. Some features
+   might be not yet implemented, mypy could pass unsafe code or reject
+   working code.
+
+There are two main type systems with respect to subtyping: nominal subtyping
+and structural subtyping. The *nominal* subtyping is based on class hierarchy,
+so that if class ``D`` inherits from class ``C``, then it is a subtype
+of ``C``. This type system is primarily used in mypy since it allows
+to produce clear and concise error messages, and since Python provides native
+``isinstance()`` checks based on class hierarchy. The *structural* subtyping
+however has its own advantages. In this system class ``D`` is a subtype
+of class ``C`` if the former has all attributes of the latter with
+compatible types. For example:
+
+.. code-block:: python
+
+   from typing import Sized
+
+   def my_len(obj: Sized) -> int:
+       ...
+
+   class MyCollection:
+       ...
+       def __len__(self) -> int:
+           return 42
+
+   my_len(MyCollection())  # OK, since 'MyCollection' is a subtype of 'Sized'
+
+This type system is a static equivalent of duck typing, well known by Python
+programmers. Mypy provides an opt-in support for structural subtyping via
+protocol classes described in this section.
+See `PEP 544 <https://www.python.org/dev/peps/pep-0544/>`_ for
+specification of protocols and structural subtyping in Python.
+
+User defined protocols
+**********************
+
+To define a protocol class, one must inherit the special ``typing.Protocol``
+class:
+
+.. code-block:: python
+
+   from typing import Protocol, Iterable
+
+   class SupportsClose(Protocol):
+       def close(self) -> None:
+          ...
+
+   class Resource:  # Note, this class does not have 'SupportsClose' base.
+       # some methods
+       def close(self) -> None:
+          self.resource.release()
+
+   def close_all(things: Iterable[SupportsClose]) -> None:
+       for thing in things:
+           thing.close()
+
+   close_all([Resource(), open('some/file')])  # This passes type check
+
+Defining subprotocols
+*********************
+
+Subprotocols are also supported. Existing protocols can be extended
+and merged using multiple inheritance. For example:
+
+.. code-block:: python
+
+   # continuing from previous example
+
+   class SizedLabeledResource(SupportsClose, Sized, Protocol):
+       label: str
+
+   class AdvancedResource(Resource):
+       def __init__(self, label: str) -> None:
+           self.label = label
+       def __len__(self) -> int:
+           ...
+
+   resource: SizedLabeledResource
+
+   # some code
+
+   resource = AdvancedResource('handle with care')  # OK
+
+Note that inheriting from existing protocols does not automatically turn
+a subclass into a protocol, it just creates a usual (non-protocol) ABC that
+implements given protocols. The ``typing.Protocol`` base must always be
+explicitly present:
+
+.. code-block:: python
+
+   class NewProtocol(Sized):  # This is NOT a protocol
+       new_attr: int
+
+   class Concrete:
+      new_attr: int
+      def __len__(self) -> int:
+          ...
+
+   x: NewProtocol = Concrete()  # Error, nominal subtyping is used by default
+
+Generic protocols
+*****************
+
+Generic protocols are also supported, generic protocols mostly follow
+the normal rules for generic classes, the main difference is that declaring
+variance is not necessary for protocols, mypy can infer it. Examples:
+
+.. code-block:: python
+
+   from typing import Protocol, TypeVar
+
+   T = TypeVar('T')
+
+   class Box(Protocol[T]):
+       content: T
+
+   def do_stuff(one: Box[str], other: Box[bytes]) -> None:
+       ...
+
+   class StringWrapper:
+       def __init__(self, content: str) -> None:
+           self.content = content
+
+   class BytesWrapper:
+       def __init__(self, content: bytes) -> None:
+           self.content = content
+
+   do_stuff(StringWrapper('one'), BytesWrapper(b'other'))  # OK
+
+   x: Box[float]
+   y: Box[int]
+   x = y  # This is also OK due to inferred covariance of the protocol 'Box'.
+
+See :ref:`generic-classes` for more details on generic classes and variance.
+
+Recursive protocols
+*******************
+
+Protocols (both generic and non-generic) can be recursive and mutually
+recursive. This could be useful for declaring abstract recursive collections
+such as trees and linked lists:
+
+.. code-block:: python
+
+   from typing import Protocol, TypeVar, Optional
+
+   class TreeLike(Protocol):
+      value: int
+      left: Optional['TreeLike']
+      right: Optional['TreeLike']
+
+   class SimpleTree:
+       def __init__(self, value: int) -> None:
+          self.value = value
+          self.left = self.right = None
+
+   root: TreeLike = SimpleTree(0)  # OK
+
+   T = TypeVar('T')
+   class Linked(Protocol[T]):
+       val: T
+       next: 'Linked[T]'
+
+   class L:
+       val: int
+       next: 'L'
+
+   def last(seq: Linked[T]) -> T:
+       ...
+
+   result = last(L())  # The inferred type of 'result' is 'int'
+
+Predefined protocols in ``typing`` module
+*****************************************
+
+Most ABCs in ``typing`` module are protocol classes describing
+common Python protocols such as ``Iterator``, ``Awaitable``, ``Mapping``, etc.
+(see `Python Docs <https://docs.python.org/3/library/typing.html>`_
+for an exhaustive list)
+For example, the following class will be considered a subtype of
+``typing.Sized`` and ``typing.Iterable[int]``:
+
+.. code-block:: python
+
+   from typing import Iterator, Iterable
+
+   class Bucket:
+       ...
+       def __len__(self) -> int:
+           return 22
+       def __iter__(self) -> Iterator[int]:
+           yield 22
+
+   def collect(items: Iterable[int]) -> int: ...
+   result: int = collect(Bucket())  # Passes type check
+
+Using ``isinstance()`` with protocols
+*************************************
+
+To use a protocol class with ``isinstance()``, one needs to decorate it with
+a special ``typing.runtime`` decorator. It will add support for basic runtime
+structural checks:
+
+.. code-block:: python
+
+   from typing import Protocol, runtime
+
+   @runtime
+   class Portable(Protocol):
+       handles: int
+
+   class Mug:
+       def __init__(self) -> None:
+           self.handles = 1
+
+   mug = Mug()
+   if isinstance(mug, Portable):
+      use(mug.handles)  # Works statically and at runtime.
+
 .. note::
+   ``isinstance()`` is with protocols not completely safe at runtime.
+   For example, signatures of methods are not checked. The runtime
+   implementation only checks the presence of all protocol members
+   in object's MRO.
 
-   There are also plans to support more Python-style "duck typing" in
-   the type system. The details are still open.

docs/source/faq.rst

@@ -101,35 +101,39 @@ Is mypy free?
 Yes. Mypy is free software, and it can also be used for commercial and
 proprietary projects. Mypy is available under the MIT license.
 
-Why not use structural subtyping?
-*********************************
+Can I use structural subtyping?
+*******************************
 
-Mypy primarily uses `nominal subtyping
-<https://en.wikipedia.org/wiki/Nominative_type_system>`_ instead of
+Mypy provides support for both `nominal subtyping
+<https://en.wikipedia.org/wiki/Nominative_type_system>`_ and
 `structural subtyping
-<https://en.wikipedia.org/wiki/Structural_type_system>`_. Some argue
-that structural subtyping is better suited for languages with duck
-typing such as Python.
-
-Here are some reasons why mypy uses nominal subtyping:
+<https://en.wikipedia.org/wiki/Structural_type_system>`_.
+Support for structural subtyping is considered experimental.
+Some argue that structural subtyping is better suited for languages with duck
+typing such as Python. Mypy however primarily uses nominal subtyping,
+leaving structural subtyping opt-in. Here are some reasons why:
 
 1. It is easy to generate short and informative error messages when
    using a nominal type system. This is especially important when
    using type inference.
 
-2. Python supports basically nominal isinstance tests and they are
-   widely used in programs. It is not clear how to support isinstance
-   in a purely structural type system while remaining compatible with
-   Python idioms.
+2. Python provides built-in support for nominal ``isinstance()`` tests and
+   they are widely used in programs. Only limited support for structural
+   ``isinstance()`` exists for ABCs in ``collections.abc`` and ``typing``
+   standard library modules.
 
 3. Many programmers are already familiar with nominal subtyping and it
    has been successfully used in languages such as Java, C++ and
    C#. Only few languages use structural subtyping.
 
-However, structural subtyping can also be useful. Structural subtyping
-is a likely feature to be added to mypy in the future, even though we
-expect that most mypy programs will still primarily use nominal
-subtyping.
+However, structural subtyping can also be useful. For example, a "public API"
+will be more flexible and convenient for users if it is typed with protocols.
+Also, using protocol types removes the necessity to explicitly declare
+implementations of ABCs. Finally, protocol types may feel more natural for
+Python programmers. As a rule of thumb, one could prefer protocols for
+function argument types and normal classes for return types. For more details
+about protocol types and structural subtyping see :ref:`protocol-types` and
+`PEP 544 <https://www.python.org/dev/peps/pep-0544/>`_.
 
 I like Python and I have no need for static typing
 **************************************************

docs/source/generics.rst

@@ -1,6 +1,8 @@
 Generics
 ========
 
+.. _generic-classes:
+
 Defining generic classes
 ************************
 

mypy/checker.py

@@ -43,7 +43,8 @@
 from mypy.subtypes import (
     is_subtype, is_equivalent, is_proper_subtype, is_more_precise,
     restrict_subtype_away, is_subtype_ignoring_tvars, is_callable_subtype,
-    unify_generic_callable,
+    unify_generic_callable, get_missing_members, get_conflict_types, get_all_flags,
+    IS_SETTABLE, IS_CLASSVAR, IS_CLASS_OR_STATIC
 )
 from mypy.maptype import map_instance_to_supertype
 from mypy.typevars import fill_typevars, has_no_typevars
@@ -1241,14 +1242,16 @@ def check_assignment(self, lvalue: Lvalue, rvalue: Expression, infer_lvalue_type
                 else:
                     rvalue_type = self.check_simple_assignment(lvalue_type, rvalue, lvalue)
 
-                # Special case: only non-abstract classes can be assigned to variables
-                # with explicit type Type[A].
+                # Special case: only non-abstract non-protocol classes can be assigned to
+                # variables with explicit type Type[A], where A is protocol or abstract.
                 if (isinstance(rvalue_type, CallableType) and rvalue_type.is_type_obj() and
-                        rvalue_type.type_object().is_abstract and
+                        (rvalue_type.type_object().is_abstract or
+                         rvalue_type.type_object().is_protocol) and
                         isinstance(lvalue_type, TypeType) and
                         isinstance(lvalue_type.item, Instance) and
-                        lvalue_type.item.type.is_abstract):
-                    self.fail("Can only assign non-abstract classes"
+                        (lvalue_type.item.type.is_abstract or
+                         lvalue_type.item.type.is_protocol)):
+                    self.fail("Can only assign concrete classes"
                               " to a variable of type '{}'".format(lvalue_type), rvalue)
                     return
                 if rvalue_type and infer_lvalue_type:
@@ -2318,8 +2321,45 @@ def check_subtype(self, subtype: Type, supertype: Type, context: Context,
             if extra_info:
                 msg += ' (' + ', '.join(extra_info) + ')'
             self.fail(msg, context)
+            if (isinstance(supertype, Instance) and supertype.type.is_protocol and
+                    isinstance(subtype, Instance)):
+                self.report_protocol_problems(subtype, supertype, context)
             return False
 
+    def report_protocol_problems(self, subtype: Instance, supertype: Instance,
+                                 context: Context) -> None:
+        """Report possible protocol conflicts between 'subtype' and 'supertype'.
+        This includes missing members, incompatible types, and incompatible
+        attribute flags, such as settable vs read-only or class variable vs
+        instance variable.
+        """
+        missing = get_missing_members(subtype, supertype)
+        if missing:
+            self.note("'{}' missing following '{}' protocol members:"
+                      .format(subtype.type.fullname(), supertype.type.fullname()),
+                      context)
+            self.note(', '.join(missing), context)
+        conflict_types = get_conflict_types(subtype, supertype)
+        if conflict_types:
+            self.note('Following members of {} have '
+                      'conflicts:'.format(subtype), context)
+            for name, got, expected in conflict_types:
+                self.note('{}: expected {}, got {}'.format(name, expected, got),
+                          context)
+        for name, subflags, superflags in get_all_flags(subtype, supertype):
+            if IS_CLASSVAR in subflags and IS_CLASSVAR not in superflags:
+                self.note('Protocol member {}.{}: expected instance variable,'
+                          ' got class variable'.format(supertype, name), context)
+            if IS_CLASSVAR in superflags and IS_CLASSVAR not in subflags:
+                self.note('Protocol member {}.{}: expected class variable,'
+                          ' got instance variable'.format(supertype, name), context)
+            if IS_SETTABLE in superflags and IS_SETTABLE not in subflags:
+                self.note('Protocol member {}.{}: expected settable variable,'
+                          ' got read-only attribute'.format(supertype, name), context)
+            if IS_CLASS_OR_STATIC in superflags and IS_CLASS_OR_STATIC not in subflags:
+                self.note('Protocol member {}.{}: expected class or static method'
+                          .format(supertype, name), context)
+
     def contains_none(self, t: Type) -> bool:
         return (
             isinstance(t, NoneTyp) or