Exhaustive Matching on Enums / Literals #6366

poneill · 2019-02-09T03:05:04Z

Are you reporting a bug, or opening a feature request?

Asking a question. I'd like to know if there is a standard idiom for enforcing exhaustive matching of an Enum / Literal type.

Please insert below the code you are checking with mypy

Suppose there are three types of pets: cats, dogs and iguanas. I want to write a function that makes the appropriate pet sound, depending on the type of pet. I'd also like to make sure that I've caught every possible type of pet. Here was my first attempt, using the NoReturn trick mentioned by @bluetech here. It looks like this works for Unions, but not for Enums / Literals? Apologies if this has been covered before.

def assert_never(x: Any) -> NoReturn:
    assert False, "Unhandled type: {}".format(type(x).__name__)

PetLiteral = Literal['dog', 'cat', 'iguana']

def make_sound_literal(pet: PetLiteral) -> str:
    if pet == 'dog':
        return 'woof'
    elif pet == 'cat':
        return 'meow'
    # elif pet == 'iguana':
    #     return 'confused silence'
    else:
        assert_never(pet)

If I comment out the iguana branch and check this file with:

$ mypy --strict  pet_sounds.py

mypy has no complaints. If I try the same deal with an enum:

class PetEnum(Enum):
    dog: str = 'dog'
    cat: str = 'cat'
    iguana: str = 'iguana'

def make_sound_enum(pet: PetEnum) -> str:
    if pet == PetEnum.dog:
        return 'woof'
    elif pet == PetEnum.cat:
        return 'meow'
#    elif pet == PetEnum.iguana:
#        return 'confused silence'
    else:
        assert_never(pet)

and check again with --strict, I still get no complaints.

What are the versions of mypy and Python you are using?
mypy 0.660
Python 3.6.5
Do you see the same issue after installing mypy from Git master?

Yes, got the same behavior with mypy 0.680+dev.21c8a812c697baf7394eafe360188ededcec6d9c

What are the mypy flags you are using? (For example --strict-optional)
--strict

The text was updated successfully, but these errors were encountered:

ilevkivskyi · 2019-02-09T12:07:41Z

The assert_never() signature proposed in the issue you reference is a bit different:

def assert_never(x: NoReturn) -> NoReturn:
    assert False, "Unhandled type: {}".format(type(x).__name__)

But with enums and literals it will probably always give you a false positive.

The exhaustive checks are not supported yet for enums and literals. But there are plans to add them, see also #5935. I am not sure about the timeline however, @Michael0x2a will you have time to work on this?

This diff adds support for performing reachability and narrowing analysis when doing certain enum checks. For example, given the following enum: class Foo(Enum): A = 1 B = 2 ...this pull request will make mypy do the following: x: Foo if x is Foo.A: reveal_type(x) # type: Literal[Foo.A] elif x is Foo.B: reveal_type(x) # type: Literal[Foo.B] else: reveal_type(x) # No output: branch inferred as unreachable This diff does not attempt to perform this same sort of narrowing for equality checks: I suspect implementing those will be harder due to their overridable nature. (E.g. you can define custom `__eq__` methods within Enum subclasses). This pull request also finally adds support for the enum behavior [described in PEP 484][0] and also sort of partially addresses python#6366 [0]: https://www.python.org/dev/peps/pep-0484/#support-for-singleton-types-in-unions

ilevkivskyi · 2019-07-06T19:00:24Z

Note that #4223 gives an example where this can cause a false positive Missing return statement.

Fixes #1803 This diff adds support for performing reachability and narrowing analysis when doing certain enum checks. For example, given the following enum: class Foo(Enum): A = 1 B = 2 ...this pull request will make mypy do the following: x: Foo if x is Foo.A: reveal_type(x) # type: Literal[Foo.A] elif x is Foo.B: reveal_type(x) # type: Literal[Foo.B] else: reveal_type(x) # No output: branch inferred as unreachable This diff does not attempt to perform this same sort of narrowing for equality checks: I suspect implementing those will be harder due to their overridable nature. (E.g. you can define custom `__eq__` methods within Enum subclasses). This pull request also finally adds support for the enum behavior [described in PEP 484][0] and also sort of partially addresses #6366 [0]: https://www.python.org/dev/peps/pep-0484/#support-for-singleton-types-in-unions

davidshepherd7 · 2019-12-02T12:10:32Z

FWIW a simple Enum example seems to work for me on mypy 0.740:

class MobileOperator(enum.Enum):
    ORANGE = "ORANGE"
    FREE = "FREE"
    EXPRESSO = "EXPRESSO"
    FOO = "FOO"

def _assert_never(x: NoReturn) -> NoReturn:
    assert False, "Unhandled type: {}".format(type(x).__name__)

def _mobile_operator_to_url(operator: MobileOperator) -> str:
    if operator is MobileOperator.ORANGE:
        return "/api/orange/"
    elif operator is MobileOperator.FREE:
        return "/api/free/izi"
    elif operator is MobileOperator.EXPRESSO:
        return "/api/expresso/yakalma"
    else:
        _assert_never(operator)

which gives the type error Argument 1 to "_assert_never" has incompatible type "Literal[MobileOperator.FOO]"; expected "NoReturn". Removing FOO from the enum fixes the error.

ezyang · 2020-08-05T18:35:16Z

And on mypy 0.780 it seems Literal is working too:

def _assert_never(x: NoReturn) -> NoReturn:
    assert False, "Unhandled type: {}".format(type(x).__name__)

def f(x: Literal['a', 'b', 'c']) -> None:
    if x == 'a':
        pass
    elif x == 'b':
        pass
    else:
        _assert_never(x)

correctly errors:

error: Argument 1 to "_assert_never" has incompatible type "Literal['c']"; expected "NoReturn"
Found 1 error in 1 file (checked 1 source file)

ezyang · 2020-08-14T04:27:57Z

Note that you have to use is to test enum equality and not ==

sbdchd · 2020-12-18T02:43:52Z

While the Python docs say that members can be compared by identity, the docs also mentioned that equality operators are defined.

I think as long as the Enum doesn't use a mixin, like class Foo(str, Enum): ... or is an IntEnum, then comparing via equality should work for refinement since only other enum members will be equal.

https://docs.python.org/3.8/library/enum.html#comparisons

sbdchd · 2021-01-29T04:28:37Z

I was looking into this more and it seems that not narrowing based on == is intended behavior to prevent false positives.

from enum import Enum
from typing import Literal

class Foo(Enum):
    a = "a"
    b = "b"
   
x: Foo
A: Literal[Foo.a]

if x == A:
    reveal_type(x) # note: Revealed type is 'Literal[main.Foo.a]'
else:
    reveal_type(x) # note: Revealed type is 'Literal[main.Foo.b]'

this doesn't:

if x == Foo.a:
    reveal_type(x) # note: Revealed type is 'main.Foo'
else:
    reveal_type(x) # note: Revealed type is 'main.Foo'

There are some test cases explaining the decision around narrowing:

mypy/test-data/unit/check-narrowing.test

Lines 783 to 802 in 4d5a1bc

    
           # See comments in testNarrowingEqualityRequiresExplicitStrLiteral and 
        
           # testNarrowingEqualityFlipFlop for more on why we can't narrow here. 
        
           x1: Foo 
        
           if x1 == Foo.A: 
        
               reveal_type(x1)  # N: Revealed type is '__main__.Foo' 
        
           else: 
        
               reveal_type(x1)  # N: Revealed type is '__main__.Foo' 
        
           x2: Foo 
        
           if x2 == A_final: 
        
               reveal_type(x2)  # N: Revealed type is '__main__.Foo' 
        
           else: 
        
               reveal_type(x2)  # N: Revealed type is '__main__.Foo' 
        
           # But we let this narrow since there's an explicit literal in the RHS. 
        
           x3: Foo 
        
           if x3 == A_literal: 
        
               reveal_type(x3)  # N: Revealed type is 'Literal[__main__.Foo.A]' 
        
           else: 
        
               reveal_type(x3)  # N: Revealed type is 'Literal[__main__.Foo.B]'

mypy/test-data/unit/check-narrowing.test

Lines 681 to 710 in 4d5a1bc

    
           def test1(switch: FlipFlopEnum) -> None: 
        
               # Naively, we might assume the 'assert' here would narrow the type to 
        
               # Literal[State.A]. However, doing this ends up breaking a fair number of real-world 
        
               # code (usually test cases) that looks similar to this function: e.g. checks 
        
               # to make sure a field was mutated to some particular value. 
        
               # 
        
               # And since mypy can't really reason about state mutation, we take a conservative 
        
               # approach and avoid narrowing anything here. 
        
               assert switch.state == State.A 
        
               reveal_type(switch.state)          # N: Revealed type is '__main__.State' 
        
               switch.mutate() 
        
               assert switch.state == State.B 
        
               reveal_type(switch.state)          # N: Revealed type is '__main__.State' 
        
           def test2(switch: FlipFlopEnum) -> None: 
        
               # So strictly speaking, we ought to do the same thing with 'is' comparisons 
        
               # for the same reasons as above. But in practice, not too many people seem to 
        
               # know that doing 'some_enum is MyEnum.Value' is idiomatic. So in practice, 
        
               # this is probably good enough for now. 
        
               assert switch.state is State.A 
        
               reveal_type(switch.state)          # N: Revealed type is 'Literal[__main__.State.A]' 
        
               switch.mutate() 
        
               assert switch.state is State.B     # E: Non-overlapping identity check (left operand type: "Literal[State.A]", right operand type: "Literal[State.B]") 
        
               reveal_type(switch.state)          # E: Statement is unreachable

And a related commit: 3dce3fd

Maybe narrowing enums via equality could be put behind another flag to prevent breaking existing code?

Here's a diff to change the narrowing, but it does require changing some tests:

mypy.diff

diff --git a/mypy/checker.py b/mypy/checker.py
index fce7e7d7..cf25bdb2 100644
--- a/mypy/checker.py
+++ b/mypy/checker.py
@@ -4083,7 +4083,7 @@ class TypeChecker(NodeVisitor[None], CheckerPluginInterface):
                         should_narrow_by_identity = True
                     else:
                         def is_exactly_literal_type(t: Type) -> bool:
-                            return isinstance(get_proper_type(t), LiteralType)
+                            return isinstance(coerce_to_literal(get_proper_type(t)), LiteralType)
 
                         def has_no_custom_eq_checks(t: Type) -> bool:
                             return (not custom_special_method(t, '__eq__', check_all=False)
@@ -4451,7 +4451,7 @@ class TypeChecker(NodeVisitor[None], CheckerPluginInterface):
             singleton_index = possible_target_indices[-1]
 
         enum_name = None
-        target = get_proper_type(target)
+        target = coerce_to_literal(get_proper_type(target))
         if isinstance(target, LiteralType) and target.is_enum_literal():
             enum_name = target.fallback.type.fullname
 
diff --git a/test-data/unit/check-enum.test b/test-data/unit/check-enum.test
index 0da3217b..de6c606e 100644
--- a/test-data/unit/check-enum.test
+++ b/test-data/unit/check-enum.test
@@ -855,6 +855,24 @@ else:
     reveal_type(z)  # No output: this branch is unreachable
 [builtins fixtures/bool.pyi]
 
+[case testEnumReachabilityChecksEquality]
+from enum import Enum
+from typing_extensions import Literal
+
+class Foo(Enum):
+    A = 1
+    B = 2
+    C = 3
+
+x: Foo
+
+if x == Foo.A:
+    reveal_type(x)  # N: Revealed type is 'Literal[__main__.Foo.A]'
+else:
+    reveal_type(x)  # N: Revealed type is 'Union[Literal[__main__.Foo.B], Literal[__main__.Foo.C]]'
+
+[builtins fixtures/bool.pyi]
+
 [case testEnumReachabilityNoNarrowingForUnionMessiness]
 from enum import Enum
 from typing_extensions import Literal
diff --git a/test-data/unit/check-narrowing.test b/test-data/unit/check-narrowing.test
index 8eb6f643..5eb356c6 100644
--- a/test-data/unit/check-narrowing.test
+++ b/test-data/unit/check-narrowing.test
@@ -688,12 +688,12 @@ def test1(switch: FlipFlopEnum) -> None:
     # approach and avoid narrowing anything here.
 
     assert switch.state == State.A
-    reveal_type(switch.state)          # N: Revealed type is '__main__.State'
+    reveal_type(switch.state)          # N: Revealed type is 'Literal[__main__.State.A]'
 
-    switch.mutate()
+    switch.mutate()                    
 
-    assert switch.state == State.B
-    reveal_type(switch.state)          # N: Revealed type is '__main__.State'
+    assert switch.state == State.B     # E: Non-overlapping equality check (left operand type: "Literal[State.A]", right operand type: "Literal[State.B]")
+    reveal_type(switch.state)          # E: Statement is unreachable
 
 def test2(switch: FlipFlopEnum) -> None:
     # So strictly speaking, we ought to do the same thing with 'is' comparisons
@@ -713,12 +713,12 @@ def test3(switch: FlipFlopStr) -> None:
     # This is the same thing as 'test1', except we try using str literals.
 
     assert switch.state == "state-1"
-    reveal_type(switch.state)          # N: Revealed type is 'builtins.str'
+    reveal_type(switch.state)          # N: Revealed type is 'Literal['state-1']'
 
     switch.mutate()
 
-    assert switch.state == "state-2"
-    reveal_type(switch.state)          # N: Revealed type is 'builtins.str'
+    assert switch.state == "state-2"   # E: Non-overlapping equality check (left operand type: "Literal['state-1']", right operand type: "Literal['state-2']")
+    reveal_type(switch.state)          # E: Statement is unreachable
 [builtins fixtures/primitives.pyi]
 
 [case testNarrowingEqualityRequiresExplicitStrLiteral]
@@ -733,13 +733,13 @@ A_literal: Literal["A"]
 # why more precise inference here is problematic.
 x_str: str
 if x_str == "A":
-    reveal_type(x_str)  # N: Revealed type is 'builtins.str'
+    reveal_type(x_str)  # N: Revealed type is 'Literal['A']'
 else:
     reveal_type(x_str)  # N: Revealed type is 'builtins.str'
 reveal_type(x_str)      # N: Revealed type is 'builtins.str'
 
 if x_str == A_final:
-    reveal_type(x_str)  # N: Revealed type is 'builtins.str'
+    reveal_type(x_str)  # N: Revealed type is 'Literal['A']'
 else:
     reveal_type(x_str)  # N: Revealed type is 'builtins.str'
 reveal_type(x_str)      # N: Revealed type is 'builtins.str'
@@ -784,15 +784,15 @@ A_literal: Literal[Foo.A]
 # testNarrowingEqualityFlipFlop for more on why we can't narrow here.
 x1: Foo
 if x1 == Foo.A:
-    reveal_type(x1)  # N: Revealed type is '__main__.Foo'
+    reveal_type(x1)  # N: Revealed type is 'Literal[__main__.Foo.A]'
 else:
-    reveal_type(x1)  # N: Revealed type is '__main__.Foo'
+    reveal_type(x1)  # N: Revealed type is 'Literal[__main__.Foo.B]'
 
 x2: Foo
 if x2 == A_final:
-    reveal_type(x2)  # N: Revealed type is '__main__.Foo'
+    reveal_type(x2)  # N: Revealed type is 'Literal[__main__.Foo.A]'
 else:
-    reveal_type(x2)  # N: Revealed type is '__main__.Foo'
+    reveal_type(x2)  # N: Revealed type is 'Literal[__main__.Foo.B]'
 
 # But we let this narrow since there's an explicit literal in the RHS.
 x3: Foo
diff --git a/test-data/unit/check-optional.test b/test-data/unit/check-optional.test
index 74a27093..6771bb38 100644
--- a/test-data/unit/check-optional.test
+++ b/test-data/unit/check-optional.test
@@ -487,7 +487,7 @@ foo([f])  # E: List item 0 has incompatible type "Callable[[], int]"; expected "
 from typing import Optional
 x = ''  # type: Optional[str]
 if x == '<string>':
-    reveal_type(x)  # N: Revealed type is 'builtins.str'
+    reveal_type(x)  # N: Revealed type is 'Literal['<string>']'
 else:
     reveal_type(x)  # N: Revealed type is 'Union[builtins.str, None]'
 if x is '<string>':
@@ -500,7 +500,7 @@ else:
 from typing import Union
 x = ''  # type: Union[str, int, None]
 if x == '<string>':
-    reveal_type(x)  # N: Revealed type is 'Union[builtins.str, builtins.int]'
+    reveal_type(x)  # N: Revealed type is 'Literal['<string>']'
 else:
     reveal_type(x)  # N: Revealed type is 'Union[builtins.str, builtins.int, None]'
 if x is '<string>':
@@ -526,7 +526,7 @@ else:
 from typing import Optional
 x = ''  # type: Optional[str]
 if x == 0:
-    reveal_type(x)  # N: Revealed type is 'Union[builtins.str, None]'
+    reveal_type(x)
 else:
     reveal_type(x)  # N: Revealed type is 'Union[builtins.str, None]'
 if x is 0:
@@ -554,8 +554,8 @@ from typing import Optional
 x: Optional[int]
 y: Optional[int]
 if x == y == 1:
-    reveal_type(x)  # N: Revealed type is 'builtins.int'
-    reveal_type(y)  # N: Revealed type is 'builtins.int'
+    reveal_type(x)  # N: Revealed type is 'Literal[1]'
+    reveal_type(y)  # N: Revealed type is 'Literal[1]'
 else:
     reveal_type(x)  # N: Revealed type is 'Union[builtins.int, None]'
     reveal_type(y)  # N: Revealed type is 'Union[builtins.int, None]'

`NoReturn` is Python's `never` which we can use to help ensure we handle all cases. I tried looking for other places where we could use this, but didn't really find any easy places to add it. Also we need to use `is` when comparing enum variants due to mypy's backwards compatible refinement outlined in: python/mypy#6366

This feature is not-really known from my experience. I had to explain it several times to other devs. But, I think that this technique should be widely recognised! It is awesome! Refs #6366

erictraut · 2021-12-31T20:24:17Z

This appears to work in the latest version of mypy, so I think this could be closed.

bbatliner · 2022-02-17T18:46:58Z

Is the lack of IntEnum narrowing/exhaustiveness in mypy well-defined behavior? I ask because Pyright/VS Code are able to narrow the signal.Handlers IntEnum, for example, based on integer equality comparisons:

i: signal.Handlers = signal.Handlers.SIG_IGN
if i == signal.Handlers.SIG_IGN:
  reveal_type(i)  # Literal[Handlers.SIG_IGN] in pyright, Handlers in mypy
elif i == signal.Handlers.SIG_DFL:
  reveal_type(i)  # Literal[Handlers.SIG_DFL] in pyright, Handlers in mypy
else:
  assert_never(i)  # fails in mypy

bbatliner · 2022-02-17T18:54:48Z

Interestingly I can compare the integers with is (which I think would run because a small range of cpython integers are backed by a single location in memory per integer IIRC?) and get mypy to pass, but generally this should be unsafe at runtime:

if i is signal.Handlers.SIG_IGN:
  reveal_type(i)
elif i is signal.Handlers.SIG_DFL:
  reveal_type(i)
else:
  assert_never(i)

JelleZijlstra · 2022-02-17T18:59:05Z

Comparing enum values with is is safe if the value is actually an enum.

bbatliner · 2022-02-17T19:43:12Z

Cool, thanks Jelle! So this seems working as intended, then.

hauntsaninja · 2023-04-24T06:54:41Z

Everything in this issue works, and in #11521 I changed mypy to make == narrow enums in addition to is

ilevkivskyi added feature priority-1-normal topic-literal-types labels Feb 9, 2019

Michael0x2a mentioned this issue Jun 16, 2019

Generalize reachability checks to support enums #7000

Merged

ilevkivskyi mentioned this issue Jul 6, 2019

MyPy gives error “Missing return statement” even when all cases are tested #4223

Closed

ilevkivskyi added the false-positive mypy gave an error on correct code label Jul 6, 2019

JukkaL mentioned this issue Jul 26, 2019

Mypy can't deal with exclusive options implied by a Union #7265

Closed

bluetech mentioned this issue Feb 15, 2020

Type annotate CallSpec2 pytest-dev/pytest#6724

Closed

thomascobb mentioned this issue Jun 26, 2020

Support exhausted matching on enums microsoft/pyright#767

Closed

ysangkok mentioned this issue Oct 13, 2020

Branching on all cases of an enum should be Mypy-enforced Simplexum/python-bitcointx#49

Closed

hauntsaninja mentioned this issue Nov 23, 2020

Strictly match all the possibilities in if-else blocks #9748

Closed

parsons-kyle-89 mentioned this issue Dec 9, 2020

== Narrows the type of an Enum in an if-elif block when following an is #9786

Closed

phijor mentioned this issue Feb 6, 2021

Experimental quorum queues metricq/metricq-manager#19

Merged

5 tasks

sbdchd mentioned this issue Feb 13, 2021

ref(bot): use exhaustiveness checking via NoReturn type chdsbd/kodiak#614

Merged

hauntsaninja mentioned this issue Jul 5, 2021

Nested tagged unions stopped working #10764

Closed

sobolevn mentioned this issue Jul 22, 2021

Adds docs about exhaustive literal and enum checks #10860

Merged

AlexWaygood added the topic-enum label Mar 24, 2022

johnthagen mentioned this issue Apr 18, 2022

Exhaustiveness Research johnthagen/sealed-typing-pep#4

Closed

hauntsaninja closed this as completed Apr 24, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Exhaustive Matching on Enums / Literals #6366

Exhaustive Matching on Enums / Literals #6366

poneill commented Feb 9, 2019

ilevkivskyi commented Feb 9, 2019

ilevkivskyi commented Jul 6, 2019

davidshepherd7 commented Dec 2, 2019

ezyang commented Aug 5, 2020

ezyang commented Aug 14, 2020

sbdchd commented Dec 18, 2020

sbdchd commented Jan 29, 2021

erictraut commented Dec 31, 2021

bbatliner commented Feb 17, 2022 •

edited

Loading

bbatliner commented Feb 17, 2022

JelleZijlstra commented Feb 17, 2022

bbatliner commented Feb 17, 2022

hauntsaninja commented Apr 24, 2023

Exhaustive Matching on Enums / Literals #6366

Exhaustive Matching on Enums / Literals #6366

Comments

poneill commented Feb 9, 2019

ilevkivskyi commented Feb 9, 2019

ilevkivskyi commented Jul 6, 2019

davidshepherd7 commented Dec 2, 2019

ezyang commented Aug 5, 2020

ezyang commented Aug 14, 2020

sbdchd commented Dec 18, 2020

sbdchd commented Jan 29, 2021

erictraut commented Dec 31, 2021

bbatliner commented Feb 17, 2022 • edited Loading

bbatliner commented Feb 17, 2022

JelleZijlstra commented Feb 17, 2022

bbatliner commented Feb 17, 2022

hauntsaninja commented Apr 24, 2023

bbatliner commented Feb 17, 2022 •

edited

Loading