Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

“Cannot infer type of lambda” using default parameter to recapture loop variable #15459

Open
andersk opened this issue Jun 18, 2023 · 2 comments
Labels
bug mypy got something wrong topic-inference When to infer types or require explicit annotations

Comments

@andersk
Copy link
Contributor

andersk commented Jun 18, 2023

Bug Report

One of the most common gotchas in Python is that a loop reassigns its iteration variable rather than creating a new binding for each iteration, and a lambda closure created in the loop observes this reassigned value rather than the value from each iteration. A typical workaround is to add a “redundant” default parameter to the lambda to recapture the value from each iteration—for example, the i=i from the above link (reproduced below). But mypy rejects such a lambda with error: Cannot infer type of lambda [misc].

To Reproduce

from typing import Callable

def create_multipliers() -> list[Callable[[int], int]]:
    return [lambda x, i=i: i * x for i in range(5)]

(mypy playground)

Expected Behavior

mypy ought to be able to realize that since the lambda is coerced to Callable[[int], int], its second parameter i=i can only ever receive the default value, so its type should be successfully inferred as (x: int, i: int = ...) -> int.

Actual Behavior

main.py:4: error: Cannot infer type of lambda  [misc]
Found 1 error in 1 file (checked 1 source file)

Your Environment

  • Mypy version used: 1.3.0
  • Mypy command-line flags: none
  • Mypy configuration options from mypy.ini (and other config files): none
  • Python version used: 3.11.3
@andersk andersk added the bug mypy got something wrong label Jun 18, 2023
erictraut pushed a commit to microsoft/pyright that referenced this issue Jun 18, 2023
…ludes a default value and the expected type doesn't include that parameter. This improvement was suggested in the [mypy issue tracker](python/mypy#15459).
andersk added a commit to andersk/zulip that referenced this issue Jun 18, 2023
django-stubs 4.2.1 gives transaction.on_commit a more accurate type
annotation, but this exposed that mypy can’t handle the lambda default
parameters that we use to recapture loop variables such as

    for stream_id in public_stream_ids:
        peer_user_ids = …
        event = …

        transaction.on_commit(
            lambda event=event, peer_user_ids=peer_user_ids: send_event(
                realm, event, peer_user_ids
            )
        )

python/mypy#15459

A workaround that mypy accepts is

        transaction.on_commit(
            (
                lambda event, peer_user_ids: lambda: send_event(
                    realm, event, peer_user_ids
                )
            )(event, peer_user_ids)
        )

But that’s kind of ugly and potentially error-prone, so let’s make a
helper function for this very common pattern.

        send_event_on_commit(realm, event, peer_user_ids)

Signed-off-by: Anders Kaseorg <anders@zulip.com>
erictraut added a commit to microsoft/pyright that referenced this issue Jun 18, 2023
…ludes a default value and the expected type doesn't include that parameter. This improvement was suggested in the [mypy issue tracker](python/mypy#15459). (#5337)

Co-authored-by: Eric Traut <erictr@microsoft.com>
andersk added a commit to andersk/zulip that referenced this issue Jun 18, 2023
django-stubs 4.2.1 gives transaction.on_commit a more accurate type
annotation, but this exposed that mypy can’t handle the lambda default
parameters that we use to recapture loop variables such as

    for stream_id in public_stream_ids:
        peer_user_ids = …
        event = …

        transaction.on_commit(
            lambda event=event, peer_user_ids=peer_user_ids: send_event(
                realm, event, peer_user_ids
            )
        )

python/mypy#15459

A workaround that mypy accepts is

        transaction.on_commit(
            (
                lambda event, peer_user_ids: lambda: send_event(
                    realm, event, peer_user_ids
                )
            )(event, peer_user_ids)
        )

But that’s kind of ugly and potentially error-prone, so let’s make a
helper function for this very common pattern.

        send_event_on_commit(realm, event, peer_user_ids)

Signed-off-by: Anders Kaseorg <anders@zulip.com>
timabbott pushed a commit to zulip/zulip that referenced this issue Jun 19, 2023
django-stubs 4.2.1 gives transaction.on_commit a more accurate type
annotation, but this exposed that mypy can’t handle the lambda default
parameters that we use to recapture loop variables such as

    for stream_id in public_stream_ids:
        peer_user_ids = …
        event = …

        transaction.on_commit(
            lambda event=event, peer_user_ids=peer_user_ids: send_event(
                realm, event, peer_user_ids
            )
        )

python/mypy#15459

A workaround that mypy accepts is

        transaction.on_commit(
            (
                lambda event, peer_user_ids: lambda: send_event(
                    realm, event, peer_user_ids
                )
            )(event, peer_user_ids)
        )

But that’s kind of ugly and potentially error-prone, so let’s make a
helper function for this very common pattern.

        send_event_on_commit(realm, event, peer_user_ids)

Signed-off-by: Anders Kaseorg <anders@zulip.com>
jbradaric added a commit to jbradaric/pyright-inlay-hints that referenced this issue Jun 20, 2023
commit 63f5658
Author: Erik De Bonte <erikd@microsoft.com>
Date:   Tue Jun 20 10:50:18 2023 -0700

    Support PEP 712's new attribute assignment conversion (microsoft#5343)

commit 9cbe8c0
Author: Eric Traut <eric@traut.com>
Date:   Tue Jun 20 09:17:40 2023 -0700

    Fixed a bug that caused an incorrect false positive error for an `assert_type` call involving a `Callable[[], X]` type. Pyright was generating a signature with a positional-only separator in this case. This addresses python/typeshed#10325 (comment). (microsoft#5345)

    Co-authored-by: Eric Traut <erictr@microsoft.com>

commit d12ad6e
Author: Eric Traut <erictr@microsoft.com>
Date:   Mon Jun 19 23:59:08 2023 -0700

    Updated typeshed stubs to the latest version.

commit eff69f5
Author: Eric Traut <erictr@microsoft.com>
Date:   Mon Jun 19 22:35:56 2023 -0700

    Fixed a bug that caused incorrect type inference for parameters in unannotated methods within child classes who derive from a generic parent class. This addresses a bug reported on stack overflow: https://stackoverflow.com/questions/76466874/trouble-specifying-type-parameter-when-inheriting-from-generic-class.

commit 43c5fae
Author: Eric Traut <erictr@microsoft.com>
Date:   Mon Jun 19 20:17:00 2023 -0700

    Finished cleanup of test cases.

commit b0d4080
Author: Eric Traut <erictr@microsoft.com>
Date:   Mon Jun 19 19:15:53 2023 -0700

    Next batch of test case cleanup.

commit efea9b9
Author: Eric Traut <erictr@microsoft.com>
Date:   Mon Jun 19 18:26:49 2023 -0700

    Changed behavior of non-ClassVar variables within a protocol definition. Previously, an error was reported when such variables were accessed from the class (as opposed to an instance of the class). Mypy (which was the reference implementation for PEP 544) does not report an error here. This addresses microsoft/pylance-release#4389.

commit 5568b4d
Author: Eric Traut <erictr@microsoft.com>
Date:   Mon Jun 19 18:00:31 2023 -0700

    Enhanced command-line version of pyright to allow file or directory names to be passed via stdin if `-` option is used in the command line. This addresses microsoft#5342.

commit cd257f7
Author: Eric Traut <erictr@microsoft.com>
Date:   Mon Jun 19 15:38:13 2023 -0700

    Next batch of test case cleanup.

commit 0c5c7b6
Author: Eric Traut <erictr@microsoft.com>
Date:   Mon Jun 19 14:23:50 2023 -0700

    Next batch of text case cleanup.

commit 899799d
Author: Eric Traut <erictr@microsoft.com>
Date:   Mon Jun 19 13:33:41 2023 -0700

    Finished cleaning up "genericTypesX.py" test files.

commit 1111f1b
Author: Eric Traut <erictr@microsoft.com>
Date:   Mon Jun 19 11:49:56 2023 -0700

    Started to untangle the hairball related to the "genericTypesX.py" test cases.

commit 67394ad
Author: Eric Traut <erictr@microsoft.com>
Date:   Mon Jun 19 10:21:36 2023 -0700

    Continued cleanup of test cases.

commit 96e03aa
Author: Eric Traut <erictr@microsoft.com>
Date:   Mon Jun 19 09:39:09 2023 -0700

    Did a cleanup pass for the dataclass test cases. Renamed and reordered test cases for maintainability.

commit b7df200
Author: Eric Traut <erictr@microsoft.com>
Date:   Mon Jun 19 08:20:54 2023 -0700

    Fixed bug that resulted in a false positive error when defining a new type alias using the `TypeAliasType` constructor that defines no new type parameters but references an outer-scoped type parameter in the type alias definition. This addresses microsoft#5341.

commit d816d00
Author: Eric Traut <erictr@microsoft.com>
Date:   Mon Jun 19 00:00:07 2023 -0700

    Started to do a pass over the test cases to make them more consistent.

commit 23bcbce
Author: Eric Traut <erictr@microsoft.com>
Date:   Sun Jun 18 17:42:25 2023 -0700

    Fixed bug in code flow engine that led to incorrect type evaluation of a variable in a nested loop. This addresses https://github.com/microsoft/pylance-release/issues/4509.

commit 056415f
Author: Eric Traut <erictr@microsoft.com>
Date:   Sun Jun 18 16:40:00 2023 -0700

    Fixed a bug in the control flow debugging code that prints the control flow graph. It was not correctly handling one of the node types which led to incomplete graphs.

commit 643bb1d
Author: Eric Traut <eric@traut.com>
Date:   Sun Jun 18 13:20:49 2023 -0700

    Improved type inference for lambdas in the case where a parameter includes a default value and the expected type doesn't include that parameter. This improvement was suggested in the [mypy issue tracker](python/mypy#15459). (microsoft#5337)

    Co-authored-by: Eric Traut <erictr@microsoft.com>

commit 8ce23eb
Author: Eric Traut <eric@traut.com>
Date:   Sun Jun 18 12:55:22 2023 -0700

    Improved `reportUnnecessaryCast` so it works with types other than cl… (microsoft#5336)

    Improved `reportUnnecessaryCast` so it works with types other than class instances. This addresses microsoft#5333.

commit 52c8cac
Author: Eric Traut <erictr@microsoft.com>
Date:   Sun Jun 18 12:41:24 2023 -0700

    Changed type printer (the component that renders types into text) to use the lowercase `type[x]` instead of `Type[x]`. It has now been four years since PEP 585 deprecated the use of the upper-case version, so most developers should be getting comfortable with the lowercase version at this point.

commit e3080b1
Author: Eric Traut <eric@traut.com>
Date:   Sun Jun 18 10:15:58 2023 -0700

    Fixed a bug that led to a false positive error under certain circumstances when a literal type argument was used in conjunction with a protocol that used a covariant type parameter and an implementation of that protocol that used an invariant type parameter. This addresses microsoft#5282. (microsoft#5332)

    Co-authored-by: Eric Traut <erictr@microsoft.com>

commit 2b0f8d2
Author: Eric Traut <erictr@microsoft.com>
Date:   Sun Jun 18 09:46:18 2023 -0700

    Improved hover text to display the calculated variance for a PEP 695-style class-scoped type variable when the user hovers over the type parameter in the type param list.

commit 02b7769
Author: Eric Traut <eric@traut.com>
Date:   Sun Jun 18 00:17:53 2023 -0700

    Fixed a false positive error arising from the use of a binary expression for a base class in a class declaration statement. This addresses microsoft#5326. (microsoft#5331)

    Co-authored-by: Eric Traut <erictr@microsoft.com>

commit 2de35e3
Author: Eric Traut <erictr@microsoft.com>
Date:   Sun Jun 18 00:00:06 2023 -0700

    Added documentation about higher-order functions.

commit 42a37f4
Author: Eric Traut <erictr@microsoft.com>
Date:   Sat Jun 17 23:38:50 2023 -0700

    Re-enabled a test case that was previously disabled because it was broken.

commit 7ea11a1
Author: Eric Traut <erictr@microsoft.com>
Date:   Sat Jun 17 23:17:32 2023 -0700

    Minor code cleanup — rename constant for clarity and refactor validation function. No functional change.

commit 53cb3f9
Author: Eric Traut <erictr@microsoft.com>
Date:   Sat Jun 17 23:00:49 2023 -0700

    Improved consistency of parameter ordering internally to type evaluator. No functional change.

commit 9bf8231
Author: Eric Traut <eric@traut.com>
Date:   Sat Jun 17 22:48:18 2023 -0700

    Fixed a bug that led to incorrect type evaluation when passing a generic class (with a constructor that includes class-scoped TypeVars) as an argument for a callable parameter. The class was being specialized prematurely (with type arguments set to `Unknown`) before the constraint solver was able to solve the higher-order function's type variables. This addresses microsoft#5324. (microsoft#5328)

    Co-authored-by: Eric Traut <erictr@microsoft.com>

commit 47cd514
Author: Eric Traut <eric@traut.com>
Date:   Sat Jun 17 22:21:52 2023 -0700

    Changed auto-variance algorithm to ignore `__new__` and `__init__` methods for purposes of calculating the variance of a TypeVar. This mirrors the behavior of mypy. (microsoft#5327)

    Co-authored-by: Eric Traut <erictr@microsoft.com>

commit 3021b9c
Author: Eric Traut <erictr@microsoft.com>
Date:   Sat Jun 17 21:52:57 2023 -0700

    Minor code cleanup — removed dead code and converted lambda to function. No functional change.

commit 327ce37
Author: Eric Traut <erictr@microsoft.com>
Date:   Sat Jun 17 20:30:36 2023 -0700

    Added test case for microsoft#5027.

commit 0559382
Author: Eric Traut <eric@traut.com>
Date:   Sat Jun 17 14:19:12 2023 -0700

    Fixed false negative when a literal and non-literal are assigned to the same TypeVar in an invariant context. This addresses microsoft#5321. (microsoft#5323)

    Co-authored-by: Eric Traut <erictr@microsoft.com>

commit 2829429
Author: Eric Traut <eric@traut.com>
Date:   Sat Jun 17 12:33:13 2023 -0700

    Fixed bug that led to an incorrect type evaluation for nested call expressions where an inner call expression used a ParamSpec. This addresses microsoft#5281. (microsoft#5322)

    Co-authored-by: Eric Traut <erictr@microsoft.com>
@myselfAbdullah007
Copy link

To address this issue, you can provide an explicit type annotation for the lambda expression. This helps MyPy understand the intended types and avoid the type inference problem.

@andersk
Copy link
Contributor Author

andersk commented Jun 26, 2023

@myselfAbdullah007 No, you can’t: the Python syntax does not allow lambda parameters to be annotated. Anyway, I’m not looking for support; I’m plenty capable of working around this in my own code. I filed this issue so it can be fixed in mypy for everyone.

@AlexWaygood AlexWaygood added the topic-inference When to infer types or require explicit annotations label Jun 26, 2023
andersk added a commit to andersk/zulip that referenced this issue Aug 10, 2023
django-stubs 4.2.1 gives transaction.on_commit a more accurate type
annotation, but this exposed that mypy can’t handle the lambda default
parameters that we use to recapture loop variables such as

    for stream_id in public_stream_ids:
        peer_user_ids = …
        event = …

        transaction.on_commit(
            lambda event=event, peer_user_ids=peer_user_ids: send_event(
                realm, event, peer_user_ids
            )
        )

python/mypy#15459

A workaround that mypy accepts is

        transaction.on_commit(
            (
                lambda event, peer_user_ids: lambda: send_event(
                    realm, event, peer_user_ids
                )
            )(event, peer_user_ids)
        )

But that’s kind of ugly and potentially error-prone, so let’s make a
helper function for this very common pattern.

        send_event_on_commit(realm, event, peer_user_ids)

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 7657cb4)
alexmv pushed a commit to zulip/zulip that referenced this issue Aug 10, 2023
django-stubs 4.2.1 gives transaction.on_commit a more accurate type
annotation, but this exposed that mypy can’t handle the lambda default
parameters that we use to recapture loop variables such as

    for stream_id in public_stream_ids:
        peer_user_ids = …
        event = …

        transaction.on_commit(
            lambda event=event, peer_user_ids=peer_user_ids: send_event(
                realm, event, peer_user_ids
            )
        )

python/mypy#15459

A workaround that mypy accepts is

        transaction.on_commit(
            (
                lambda event, peer_user_ids: lambda: send_event(
                    realm, event, peer_user_ids
                )
            )(event, peer_user_ids)
        )

But that’s kind of ugly and potentially error-prone, so let’s make a
helper function for this very common pattern.

        send_event_on_commit(realm, event, peer_user_ids)

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 7657cb4)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug mypy got something wrong topic-inference When to infer types or require explicit annotations
Projects
None yet
Development

No branches or pull requests

3 participants