Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Possible option: use ParamSpec for lru_cache #6986

Closed
wants to merge 2 commits into from
Closed

Possible option: use ParamSpec for lru_cache #6986

wants to merge 2 commits into from

Conversation

sobolevn
Copy link
Member

@sobolevn sobolevn commented Jan 21, 2022 •
edited
Loading

This one might be controversal. Why?

  • When adding ParamSpec we cannot ensure that args are Hashable, but signature remains the same
  • When not using ParamSpec, we can ensure that args are Hashable, but signature changes to *Any, **Any

What do others think? In my opinion better signature is more important 🤔

Local testing with the latest mypy shows that my version works fine:

from typing import TypeVar, Callable, ParamSpec, Generic

_T = TypeVar("_T")
_P = ParamSpec("_P")

class _lru_cache_wrapper(Generic[_P, _T]):
    __wrapped__: Callable[_P, _T]
    def __call__(self, *args: _P.args, **kwargs: _P.kwargs) -> _T: ...

def lru_cache(maxsize: Callable[_P, _T]) -> _lru_cache_wrapper[_P, _T]:
    pass

@lru_cache
def some(arg: int) -> str:
    pass

reveal_type(some)  # N: Revealed type is "ex._lru_cache_wrapper[def (arg: builtins.int), builtins.str*]"

@sobolevn
Copy link
Member Author

sobolevn commented Jan 21, 2022 •
edited
Loading

This looks like a mypy bug:

 stdlib/functools.pyi:27: error: Argument 1 to "final" has incompatible type "Type[_lru_cache_wrapper[Any, Any]]"; expected "Type[_lru_cache_wrapper[Any, Any]]"  [arg-type]

Update

Yes, I can reproduce it locally. Going to report to mypy (and probably fix it).

@sobolevn sobolevn mentioned this pull request Jan 21, 2022
Closed
@github-actions

This comment has been minimized.

Sign in to view
@sobolevn
Copy link
Member Author

sobolevn commented Jan 21, 2022 •
edited
Loading

Looks like we have another problem, here's an example from flake8:

@functools.lru_cache(maxsize=None)
def style_guide_for(self, filename: str) -> "StyleGuide": pass

It is then used as: guide = self.style_guide_for(filename)
So, it reports:

+ src/flake8/style_guide.py:275: error: Missing positional argument "filename" in call to "__call__" of "_lru_cache_wrapper"
+ src/flake8/style_guide.py:275: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "StyleGuideManager"

I guess that self arg breaks it 😒

@AlexWaygood
Copy link
Member

We had a looong discussion about how to type lru_cache over in #6347, but didn't really come to a conclusion ://

@sobolevn
Copy link
Member Author

Going to try @staticmethod approach!

AlexWaygood
AlexWaygood reviewed Jan 21, 2022
View reviewed changes
stdlib/functools.pyi
@overload
def lru_cache(maxsize: Callable[..., _T], typed: bool = ...) -> _lru_cache_wrapper[_T]: ...
def lru_cache(maxsize: Callable[_P, _T]) -> _lru_cache_wrapper[_P, _T]: ...
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why'd the typed arg get deleted? :)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeap, my bad. It is allowed: https://github.com/python/cpython/blob/54610bb448a9cf5be77d53b66169fca4c11be6cb/Lib/functools.py#L508

@github-actions
Copy link
Contributor

Diff from mypy_primer, showing the effect of this PR on open source code:

rich (https://github.com/willmcgugan/rich)
+ rich/theme.py:25: error: Missing positional argument "style_definition" in call to "__call__" of "_lru_cache_wrapper"
+ rich/theme.py:25: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "Type[Style]"
+ rich/theme.py:54: error: Missing positional argument "style_definition" in call to "__call__" of "_lru_cache_wrapper"
+ rich/theme.py:54: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "Type[Style]"
+ rich/style.py:148: error: Missing positional argument "color" in call to "__call__" of "_lru_cache_wrapper"
+ rich/style.py:148: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "Type[Color]"
+ rich/style.py:393: error: Missing positional argument "system" in call to "__call__" of "_lru_cache_wrapper"
+ rich/style.py:393: error: Missing positional argument "self" in call to "__call__" of "_lru_cache_wrapper"
+ rich/style.py:393: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "ColorSystem"; expected "Color"
+ rich/style.py:396: error: Missing positional argument "system" in call to "__call__" of "_lru_cache_wrapper"
+ rich/style.py:396: error: Missing positional argument "self" in call to "__call__" of "_lru_cache_wrapper"
+ rich/style.py:396: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "ColorSystem"; expected "Color"
+ rich/style.py:416: error: Missing positional argument "style_definition" in call to "__call__" of "_lru_cache_wrapper"
+ rich/style.py:416: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "Type[Style]"
+ rich/style.py:542: error: Missing positional argument "color" in call to "__call__" of "_lru_cache_wrapper"
+ rich/style.py:542: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "Type[Color]"
+ rich/style.py:569: error: Missing positional argument "color" in call to "__call__" of "_lru_cache_wrapper"
+ rich/style.py:569: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "Type[Color]"
+ rich/segment.py:158: error: Missing positional argument "cut" in call to "__call__" of "_lru_cache_wrapper"
+ rich/segment.py:158: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "Segment"; expected "Type[Segment]"
+ rich/segment.py:158: error: Argument 2 to "__call__" of "_lru_cache_wrapper" has incompatible type "int"; expected "Segment"
+ rich/markup.py:155: error: Missing positional argument "style" in call to "__call__" of "_lru_cache_wrapper"
+ rich/markup.py:155: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "Type[Style]"
+ rich/markup.py:212: error: Missing positional argument "style" in call to "__call__" of "_lru_cache_wrapper"
+ rich/markup.py:212: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "Type[Style]"
+ rich/jupyter.py:63: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "TerminalTheme"; expected "Style"
+ rich/console.py:1393: error: Missing positional argument "style_definition" in call to "__call__" of "_lru_cache_wrapper"
+ rich/console.py:1393: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "Type[Style]"
+ rich/console.py:2089: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "TerminalTheme"; expected "Style"
+ rich/console.py:2101: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "TerminalTheme"; expected "Style"
+ rich/color.py:513: error: Missing positional argument "color" in call to "__call__" of "_lru_cache_wrapper"
+ rich/color.py:513: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "ColorTriplet"; expected "Palette"
+ rich/color.py:526: error: Missing positional argument "color" in call to "__call__" of "_lru_cache_wrapper"
+ rich/color.py:526: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "ColorTriplet"; expected "Palette"
+ rich/ansi.py:163: error: Missing positional argument "style_definition" in call to "__call__" of "_lru_cache_wrapper"
+ rich/ansi.py:163: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "Type[Style]"
+ rich/progress_bar.py:142: error: Missing positional argument "no_color" in call to "__call__" of "_lru_cache_wrapper"
+ rich/progress_bar.py:143: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "Style"; expected "ProgressBar"
+ rich/progress_bar.py:143: error: Argument 3 to "__call__" of "_lru_cache_wrapper" has incompatible type "Optional[str]"; expected "Style"
+ rich/progress_bar.py:143: error: Argument 4 to "__call__" of "_lru_cache_wrapper" has incompatible type "bool"; expected "str"

flake8 (https://github.com/pycqa/flake8)
+ src/flake8/style_guide.py:275: error: Missing positional argument "filename" in call to "__call__" of "_lru_cache_wrapper"
+ src/flake8/style_guide.py:275: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "StyleGuideManager"
+ src/flake8/style_guide.py:308: error: Missing positional argument "filename" in call to "__call__" of "_lru_cache_wrapper"
+ src/flake8/style_guide.py:308: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "StyleGuideManager"
+ tests/unit/test_style_guide.py:144: error: Missing positional argument "filename" in call to "__call__" of "_lru_cache_wrapper"

dedupe (https://github.com/dedupeio/dedupe)
+ dedupe/training.py:93: error: Missing positional argument "iterable" in call to "__call__" of "_lru_cache_wrapper"
+ dedupe/training.py:93: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "FrozenSet[int]"; expected "Resampler"

manticore (https://github.com/trailofbits/manticore)
+ manticore/core/smtlib/solver.py:505: error: Signature of "can_be_true" incompatible with supertype "Solver"
+ manticore/core/smtlib/solver.py:605: error: Signature of "get_all_values" incompatible with supertype "Solver"

isort (https://github.com/pycqa/isort)
+ isort/settings.py:335: error: Argument 2 to "__call__" of "_lru_cache_wrapper" has incompatible type "Tuple[str, ...]"; expected "Tuple[str]"
+ isort/settings.py:769: error: Argument 2 to "__call__" of "_lru_cache_wrapper" has incompatible type "Tuple[str, ...]"; expected "Tuple[str]"
+ isort/settings.py:807: error: Argument 2 to "__call__" of "_lru_cache_wrapper" has incompatible type "Tuple[str, ...]"; expected "Tuple[str]"
+ isort/deprecated/finders.py:302: error: Missing positional argument "path" in call to "__call__" of "_lru_cache_wrapper"
+ isort/deprecated/finders.py:302: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "Type[RequirementsFinder]"
+ isort/deprecated/finders.py:333: error: Missing positional argument "path" in call to "__call__" of "_lru_cache_wrapper"
+ isort/deprecated/finders.py:333: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "Type[RequirementsFinder]"

edgedb (https://github.com/edgedb/edgedb)
+ edb/schema/expr.py:258:18: error: Missing positional argument "data" in call to "__call__" of "_lru_cache_wrapper"
+ edb/schema/expr.py:258:53: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "Tuple[str, Union[Tuple[type, ...], type, None], Tuple[UUID, ...], Tuple[Tuple[str, Any], ...]]"; expected "Type[ObjectCollection[Object_T]]"
+ edb/schema/schema.py:1146:16: error: Missing positional argument "stype" in call to "__call__" of "_lru_cache_wrapper"
+ edb/schema/schema.py:1146:32: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "Type"; expected "FlatSchema"
+ edb/schema/schema.py:1156:16: error: Missing positional argument "stype" in call to "__call__" of "_lru_cache_wrapper"
+ edb/schema/schema.py:1156:32: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "Type"; expected "FlatSchema"
+ edb/schema/schema.py:1166:16: error: Missing positional argument "scls" in call to "__call__" of "_lru_cache_wrapper"
+ edb/schema/schema.py:1167:13: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "Object"; expected "FlatSchema"
+ edb/schema/schema.py:1208:5: error: Signature of "get_referrers_ex" incompatible with supertype "Schema"
+ edb/schema/schema.py:1254:19: error: Missing positional argument "data" in call to "__call__" of "_lru_cache_wrapper"
+ edb/schema/schema.py:1254:44: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "Tuple[str, UUID]"; expected "Type[Object]"
+ edb/schema/schema.py:1791:16: error: Missing positional argument "scls" in call to "__call__" of "_lru_cache_wrapper"
+ edb/schema/schema.py:1791:51: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "Object"; expected "FlatSchema"
+ edb/schema/schema.py:1792:15: error: Missing positional argument "scls" in call to "__call__" of "_lru_cache_wrapper"
+ edb/schema/schema.py:1792:49: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "Object"; expected "FlatSchema"
+ edb/schema/schema.py:1793:14: error: Missing positional argument "scls" in call to "__call__" of "_lru_cache_wrapper"
+ edb/schema/schema.py:1793:51: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "Object"; expected "FlatSchema"
+ edb/schema/scalars.py:132:55: error: Argument 2 to "__call__" of "_lru_cache_wrapper" has incompatible type "Object"; expected "Type"
+ edb/schema/scalars.py:132:61: error: Argument 3 to "__call__" of "_lru_cache_wrapper" has incompatible type "Object"; expected "Type"
+ edb/schema/delta.py:1860:19: error: Missing positional argument "field_name" in call to "__call__" of "_lru_cache_wrapper"
+ edb/schema/delta.py:1860:51: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "Type[ObjectCommand[so.Object_T]]"
+ edb/schema/delta.py:3272:53: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "Name"; expected "QualName"
+ edb/schema/indexes.py:74:40: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "Name"; expected "QualName"

pandas (https://github.com/pandas-dev/pandas)
+ pandas/core/dtypes/cast.py:514: error: Unused "type: ignore" comment
+ pandas/core/groupby/ops.py:199: error: Missing positional argument "is_numeric" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ pandas/core/groupby/ops.py:199: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "Type[WrappedCythonOp]"  [arg-type]
+ pandas/core/groupby/ops.py:199: error: Argument 4 to "__call__" of "_lru_cache_wrapper" has incompatible type "bool"; expected "dtype[Any]"  [arg-type]

mypy (https://github.com/python/mypy)
+ mypy/find_sources.py:161: error: Missing positional argument "dir" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ mypy/find_sources.py:161: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "SourceFinder"  [arg-type]
+ mypy/find_sources.py:203: error: Missing positional argument "dir" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ mypy/find_sources.py:203: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "SourceFinder"  [arg-type]

core (https://github.com/home-assistant/core)
+ homeassistant/components/mqtt/__init__.py:1069: error: Missing positional argument "topic" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]

paroxython (https://github.com/laowantong/paroxython)
+ paroxython/assess_costs.py:128: error: Missing positional argument "taxon" in call to "__call__" of "_lru_cache_wrapper"
+ paroxython/assess_costs.py:128: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "TaxonName"; expected "LearningCostAssessor"
+ paroxython/recommend_programs.py:290: error: Missing positional argument "taxon" in call to "__call__" of "_lru_cache_wrapper"
+ paroxython/recommend_programs.py:290: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "TaxonName"; expected "LearningCostAssessor"
+ paroxython/map_taxonomy.py:243: error: Missing positional argument "label_name" in call to "__call__" of "_lru_cache_wrapper"
+ paroxython/map_taxonomy.py:243: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "LabelName"; expected "Taxonomy"

pydantic (https://github.com/samuelcolvin/pydantic)
+ pydantic/tools.py:37: error: Unused "type: ignore" comment

pip (https://github.com/pypa/pip)
+ src/pip/_internal/req/req_install.py:489: error: Missing positional argument "self" in call to "__call__" of "_lru_cache_wrapper"
+ src/pip/_internal/req/req_install.py:514: error: Missing positional argument "self" in call to "__call__" of "_lru_cache_wrapper"
+ src/pip/_internal/index/package_finder.py:863: error: Missing positional argument "project_name" in call to "__call__" of "_lru_cache_wrapper"
+ src/pip/_internal/index/package_finder.py:863: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "PackageFinder"
+ src/pip/_internal/index/package_finder.py:881: error: Missing positional argument "project_name" in call to "__call__" of "_lru_cache_wrapper"
+ src/pip/_internal/index/package_finder.py:882: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "PackageFinder"
+ src/pip/_internal/self_outdated_check.py:145: error: Missing positional argument "project_name" in call to "__call__" of "_lru_cache_wrapper"
+ src/pip/_internal/self_outdated_check.py:145: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "PackageFinder"
+ src/pip/_internal/wheel_builder.py:75: error: Missing positional argument "self" in call to "__call__" of "_lru_cache_wrapper"
+ src/pip/_internal/distributions/sdist.py:100: error: Missing positional argument "self" in call to "__call__" of "_lru_cache_wrapper"
+ src/pip/_internal/resolution/resolvelib/factory.py:269: error: Missing positional argument "self" in call to "__call__" of "_lru_cache_wrapper"
+ src/pip/_internal/resolution/resolvelib/factory.py:581: error: Missing positional argument "project_name" in call to "__call__" of "_lru_cache_wrapper"
+ src/pip/_internal/resolution/resolvelib/factory.py:581: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "NormalizedName"; expected "PackageFinder"
+ src/pip/_internal/commands/list.py:231: error: Missing positional argument "project_name" in call to "__call__" of "_lru_cache_wrapper"
+ src/pip/_internal/commands/list.py:231: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "NormalizedName"; expected "PackageFinder"
+ src/pip/_internal/commands/index.py:118: error: Missing positional argument "project_name" in call to "__call__" of "_lru_cache_wrapper"

pylint (https://github.com/pycqa/pylint)
+ pylint/utils/file_state.py:100: error: Missing positional argument "msgid_or_symbol" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ pylint/utils/file_state.py:100: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "MessageDefinitionStore"  [arg-type]
+ pylint/message/message_definition_store.py:69: error: Missing positional argument "msgid_or_symbol" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ pylint/message/message_definition_store.py:69: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "MessageDefinitionStore"  [arg-type]
+ pylint/message/message_definition_store.py:78: error: Missing positional argument "msgid_or_symbol" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ pylint/message/message_definition_store.py:79: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "MessageDefinitionStore"  [arg-type]

@sobolevn
Copy link
Member Author

sobolevn commented Jan 21, 2022 •
edited
Loading

Ok, it didn't work out. I am going to close it and at least wait for python/mypy#12033 to be merged and Concatenate to be supported.

@sobolevn sobolevn closed this Jan 21, 2022
@tamird
Copy link
Contributor

tamird commented Feb 23, 2024

@sobolevn is it time to revisit this?

@sobolevn
Copy link
Member Author

@tamird please, go ahead and try!

@tamird
Copy link
Contributor

tamird commented Feb 23, 2024

I'm out of my depth on trying, just another frustrated user suffering bad typings :(

@sobolevn
Copy link
Member Author

Looking at some recent discussion, this seems to be rather hard / impossible right now: #10523

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@AlexWaygood AlexWaygood AlexWaygood left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@sobolevn @AlexWaygood @tamird