TYP: enable disallow_untyped_decorators #43828

twoertwein · 2021-10-01T02:29:35Z

closes TYP: enable disallow-untyped-decorators in CI #33455, closes TYP: IntervalIndex.right/left might be infered as function #41588, (closes TYP: IndexEngine.get_indexer_with_fill #43388, closes TYP: RangeIndex._constructor #43387)
tests added / passed
Ensure all linting tests pass, see here for how to run them
whatsnew entry

Some of the many type errors unveiled by cache_readonly might be easily fixable. This PR simply ignores all the uncovered errors to quickly enable mypy's disallow_untyped_decorators to prevent future untyped decorators.

I enabled mypy's disallow_untyped_decorators but pyright's reportUntypedFunctionDecorator finds the following three functions that are not found by mypy, so I didn't enable reportUntypedFunctionDecorator (can't add ignore statements, otherwise mypy will complain about unused ignore statements, edit: using pyright's file-based ignore comments):

pandas/conftest.py:1505
~~pandas/core/util/numba_.py:93~~
pandas/core/window/numba_.py:244

pandas/_libs/properties.pyx

pandas/core/indexes/base.py

pandas/_libs/properties.pyi

twoertwein · 2021-10-13T17:19:06Z

The failure from the web-docs:

pandas/pandas/core/indexes/datetimes.py:docstring of pandas.core.indexes.datetimes.DatetimeIndex:124: WARNING: autosummary: stub file not found 'pandas.DatetimeIndex.std'. Check your autosummary_generate setting.

seems unrelated, but it looks legit. DatetimeIndex inherits the std name through a decorator, but the class from which the method is copied has no doc-string for it (pandas/core/arrays/datetimes.py:1917).

jbrockmendel · 2021-10-13T17:53:08Z

seems unrelated, but it looks legit. DatetimeIndex inherits the std name through a decorator, but the class from which the method is copied has no doc-string for it (pandas/core/arrays/datetimes.py:1917).

yah there are several methods near the top of DatetimeIndex that are there instead of using the decorator for this reason, most of the "# methods that dispatch to DatetimeArray and wrap result" section

twoertwein · 2021-10-13T18:37:42Z

yah there are several methods near the top of DatetimeIndex that are there instead of using the decorator for this reason, most of the "# methods that dispatch to DatetimeArray and wrap result" section

So the solution is to write a doc-string?

I'm surprised that this warning/error occurs only for this PR. The 1.3.3 documentation of DatetimeIndex has the same issue:

jbrockmendel · 2021-10-13T19:04:13Z

So the solution is to write a doc-string?

IIRC the motivation was for mypy, not docstrings. the decorator should handle docstrings correctly.

twoertwein · 2021-10-13T21:16:16Z

The decorator seems to handle doc-strings correctly, all other attributes (for example tz) have their doc-string string in the web-documentation. The main difference between tz and std is that DatetimeArray.tz has a doc-string while DatetimeArray.std has none: DatetimeIndex.tz has therefore a doc-string but DatetimeIndex.std doesn't have a doc-string.

I will write a doc-string in a separate PR.

But I don't understand why this PR causes this issue.

twoertwein · 2021-10-18T21:53:55Z

Should be ready for review :)

jreback · 2021-10-21T01:56:27Z

cc @simonjayhawkins

pandas/core/_numba/executor.py

pandas/core/arrays/categorical.py

simonjayhawkins · 2021-10-21T10:33:34Z

pandas/core/arrays/period.py

-            return result.view(self.dtype)
+            # error: Incompatible return value type (got "Union[ExtensionArray,
+            # ndarray[Any, Any]]", expected "PeriodArray")
+            return result.view(self.dtype)  # type: ignore[return-value]


in pandas/core/arrays/datetimelike.py we overloaded view for DatetimeLikeArrayMixin. maybe could do the same for PeriodArray. (suggestion, not necessarily for this PR)

pandas/core/frame.py

simonjayhawkins · 2021-10-21T10:46:52Z

pandas/core/groupby/generic.py

+        # expected "Index"
+        levels = [ping.group_index for ping in self.grouper.groupings] + [
+            lev  # type: ignore[list-item]
+        ]


hmm, we have a Union return type from factorize.

Over time we should avoid Union return types, either through refactoring code or using overloads.

from https://github.com/python/typeshed/blob/master/CONTRIBUTING.md#conventions

avoid union return types: python/mypy#1693;

(comment, no action needed for this PR)

pyproject.toml

simonjayhawkins

Thanks @twoertwein very nice. some comments and suggestions but none blockers.

simonjayhawkins · 2021-10-21T11:11:35Z

I enabled mypy's disallow_untyped_decorators but pyright's reportUntypedFunctionDecorator finds the following three functions that are not found by mypy, so I didn't enable reportUntypedFunctionDecorator (can't add ignore statements, otherwise mypy will complain about unused ignore statements):

pandas/conftest.py:1505

pandas/core/util/numba_.py:93

pandas/core/window/numba_.py:244

maybe open an issue after this is merged.

simonjayhawkins · 2021-10-21T11:12:48Z

(or use module level pyright comment syntax)

simonjayhawkins · 2021-10-21T11:14:14Z

Some of the many type errors unveiled by cache_readonly might be easily fixable. This PR simply ignores all the uncovered errors to quickly enable mypy's disallow_untyped_decorators to prevent future untyped decorators.

SGTM

twoertwein · 2021-10-21T16:54:50Z

Rebased and addressed the big comments.

If it is green, it should be ready for merging from my perspective.

simonjayhawkins · 2021-10-21T16:55:58Z

If it is green, it should be ready for merging from my perspective.

sure. @jreback @jbrockmendel any further comments?

pandas/_libs/properties.pyi

pandas/util/_decorators.py

pandas/core/window/online.py

twoertwein · 2021-10-21T18:35:17Z

There is something failing related due to the numba wrapper, will debug that

twoertwein · 2021-10-21T19:00:04Z

I think numba analyzes the source code to JIT it. I think the errors are a results of jitting a function that itself has a jitted function where the inner function is jitted with typed_numba_jit.

twoertwein · 2021-10-22T01:37:42Z

I think I have to revert the numba-wrapper: https://numba.pydata.org/numba-doc/dev/reference/pysupported.html#functions-as-arguments. It would be great to clean up all the ignore statement for numba.jit calls but I'm te wrong person for that - not familiar with numba and how pandas uses it.

@simonjayhawkins let me know if you see a simple non-intrusive way that could avoid the ignore statements for this PR. Otherwise, I will revert the commit that added typed_numba_jit.

simonjayhawkins · 2021-10-22T09:43:16Z

@simonjayhawkins let me know if you see a simple non-intrusive way that could avoid the ignore statements for this PR. Otherwise, I will revert the commit that added typed_numba_jit.

no problem leaving that out of this PR. The method I'm working with would not prevent the ignores but does providing typing for the "pandas lib api". I use that term loosely as meaning anything coded in Cython, as this is otherwise not formalized. https://github.com/simonjayhawkins/pandas/blob/ec8f0397b5c4a232e8e3e7abfbec91e9ea33016a/pandas/_libs_numba/algos.py#L245-L255

It'll probably be better to write out own stubs for now and include them in the main repo

twoertwein · 2021-10-22T13:49:31Z

I removed the numba wrapper and rebased

twoertwein · 2021-10-22T20:24:47Z

The isinstance change and rebased after #44144

simonjayhawkins · 2021-10-29T20:21:12Z

@twoertwein can you resolve conflicts

simonjayhawkins · 2021-10-29T20:46:17Z

I removed the numba wrapper and rebased

i've opened #44233 that we could maybe merge after this.

twoertwein · 2021-10-29T20:59:26Z

i've opened #44233 that we could maybe merge after this.

Great! I'll rebase this PR to resolve the conflicts with master.

pep8speaks · 2021-10-30T00:36:15Z

Hello @twoertwein! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2021-10-30 11:55:31 UTC

pandas/core/arrays/categorical.py

pandas/core/indexes/base.py

jreback · 2021-10-30T15:10:19Z

nice thanks @twoertwein and @simonjayhawkins for the review.

twoertwein commented Oct 1, 2021

View reviewed changes

pandas/_libs/properties.pyx Show resolved Hide resolved

twoertwein commented Oct 1, 2021

View reviewed changes

pandas/core/indexes/base.py Outdated Show resolved Hide resolved

twoertwein commented Oct 1, 2021

View reviewed changes

pandas/_libs/properties.pyi Outdated Show resolved Hide resolved

jreback added the Typing type annotations, mypy/pyright type checking label Oct 2, 2021

twoertwein mentioned this pull request Oct 13, 2021

DOC: DatetimeArray.std #44020

Merged

jreback added this to the 1.4 milestone Oct 18, 2021

simonjayhawkins reviewed Oct 21, 2021

View reviewed changes

pandas/core/_numba/executor.py Show resolved Hide resolved

simonjayhawkins reviewed Oct 21, 2021

View reviewed changes

pandas/core/arrays/categorical.py Outdated Show resolved Hide resolved

simonjayhawkins reviewed Oct 21, 2021

View reviewed changes

pandas/core/frame.py Outdated Show resolved Hide resolved

simonjayhawkins reviewed Oct 21, 2021

View reviewed changes

pyproject.toml Show resolved Hide resolved

simonjayhawkins approved these changes Oct 21, 2021

View reviewed changes

jbrockmendel reviewed Oct 21, 2021

View reviewed changes

pandas/_libs/properties.pyi Show resolved Hide resolved

jbrockmendel reviewed Oct 21, 2021

View reviewed changes

pandas/util/_decorators.py Outdated Show resolved Hide resolved

simonjayhawkins reviewed Oct 21, 2021

View reviewed changes

pandas/core/window/online.py Outdated Show resolved Hide resolved

twoertwein mentioned this pull request Oct 27, 2021

Improved the type stubs in the _libs directory to help with type checking #43744

Closed

simonjayhawkins mentioned this pull request Oct 29, 2021

TYP: numba stub #44233

Merged

simonjayhawkins reviewed Oct 30, 2021

View reviewed changes

pandas/core/arrays/categorical.py Show resolved Hide resolved

simonjayhawkins reviewed Oct 30, 2021

View reviewed changes

pandas/core/indexes/base.py Show resolved Hide resolved

twoertwein added 13 commits October 30, 2021 07:55

TYP: enable disallow_untyped_decorators

043b268

one more

1065856

use type instead of typing.Type

456c4ba

black

da798fa

enable pyright's rule but skip two files

590dd41

partial stub

7325a6b

ignore incomplete stub for pyright

89be593

note on lie

c13c0cd

pyright comment after removing numba wrapper

cd61bdd

use isinstance

baf088f

new numbas

966418b

empty line

4e08378

comments

bf8026b

simonjayhawkins approved these changes Oct 30, 2021

View reviewed changes

jreback merged commit e394c53 into pandas-dev:master Oct 30, 2021

twoertwein mentioned this pull request Nov 4, 2021

TYP: NumericIndex._can_hold_na #43390

Closed

twoertwein mentioned this pull request Mar 22, 2022

AttributeError: 'pandas._libs.properties.CachedProperty' object has no attribute 'func'QST: #46474

Closed

2 tasks

Uh oh!

TYP: enable disallow_untyped_decorators #43828

TYP: enable disallow_untyped_decorators #43828

Uh oh!

Conversation

twoertwein commented Oct 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

twoertwein commented Oct 13, 2021

Uh oh!

jbrockmendel commented Oct 13, 2021

Uh oh!

twoertwein commented Oct 13, 2021

Uh oh!

jbrockmendel commented Oct 13, 2021

Uh oh!

twoertwein commented Oct 13, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

twoertwein commented Oct 18, 2021

Uh oh!

jreback commented Oct 21, 2021

Uh oh!

Uh oh!

Uh oh!

simonjayhawkins Oct 21, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

simonjayhawkins Oct 21, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

simonjayhawkins left a comment

Choose a reason for hiding this comment

Uh oh!

simonjayhawkins commented Oct 21, 2021

Uh oh!

simonjayhawkins commented Oct 21, 2021

Uh oh!

simonjayhawkins commented Oct 21, 2021

Uh oh!

twoertwein commented Oct 21, 2021

Uh oh!

simonjayhawkins commented Oct 21, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

twoertwein commented Oct 21, 2021

Uh oh!

twoertwein commented Oct 21, 2021

Uh oh!

twoertwein commented Oct 22, 2021

Uh oh!

simonjayhawkins commented Oct 22, 2021

Uh oh!

twoertwein commented Oct 22, 2021

Uh oh!

twoertwein commented Oct 22, 2021

Uh oh!

simonjayhawkins commented Oct 29, 2021

Uh oh!

simonjayhawkins commented Oct 29, 2021

Uh oh!

twoertwein commented Oct 29, 2021

Uh oh!

pep8speaks commented Oct 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Comment last updated at 2021-10-30 11:55:31 UTC

Uh oh!

Uh oh!

Uh oh!

jreback commented Oct 30, 2021

Uh oh!

Uh oh!

twoertwein commented Oct 1, 2021 •

edited

Loading

twoertwein commented Oct 13, 2021 •

edited

Loading

pep8speaks commented Oct 30, 2021 •

edited

Loading