Unify MYPYINDUCTOR and MYPY #118432

ezyang · 2024-01-26T22:46:24Z

Stack from ghstack (oldest at bottom):

The original motivation for MYPYINDUCTOR was a faster type checking configuration that only checked a subset of files. With the removal of follow_imports = ignore, we are now able to use dmypy to do fast incremental typechecking, eliminating the need for this.

Perhaps erroneously, when I tee'ed up this PR I elected to delete the follow_imports = skip designations in the mypy-inductor.ini. This lead to a number of extra type error suppressions that I manually edited. You will need to review.

Signed-off-by: Edward Z. Yang ezyang@meta.com

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler

Signed-off-by: Edward Z. Yang <ezyang@meta.com> [ghstack-poisoned]

pytorch-bot · 2024-01-26T22:46:27Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/118432

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 5772c25 with merge base ff8e335 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

Signed-off-by: Edward Z. Yang <ezyangmeta.com> cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx peterbell10 ipiszy yf225 chenyang78 kadeng muchulee8 aakhundov ColinPeppler [ghstack-poisoned]

The original motivation for MYPYINDUCTOR was a faster type checking configuration that only checked a subset of files. With the removal of `follow_imports = ignore`, we are now able to use dmypy to do fast incremental typechecking, eliminating the need for this. Perhaps erroneously, when I tee'ed up this PR I elected to delete the `follow_imports = skip` designations in the mypy-inductor.ini. This lead to a number of extra type error suppressions that I manually edited. You will need to review. Signed-off-by: Edward Z. Yang <ezyangmeta.com> cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx peterbell10 ipiszy yf225 chenyang78 kadeng muchulee8 aakhundov ColinPeppler [ghstack-poisoned]

Signed-off-by: Edward Z. Yang <ezyangmeta.com> ghstack-source-id: 3f27751d06d3f4b5d71e79e23858660478b2e194 Pull Request resolved: #118432

ezyang · 2024-01-27T14:23:48Z

@pytorchbot merge

When using dmypy, this setting is enabled and cannot be turned off. Force it for regular mypy too. Signed-off-by: Edward Z. Yang <ezyang@meta.com> Pull Request resolved: #118467 Approved by: https://github.com/Skylion007 ghstack dependencies: #118414, #118418, #118432

Signed-off-by: Edward Z. Yang <ezyang@meta.com> Pull Request resolved: #118468 Approved by: https://github.com/Skylion007 ghstack dependencies: #118414, #118418, #118432, #118467

dmypy silently ignores follow_imports = skip, so to get parity between dmypy and mypy we have to suck it up and type: ignore all of the sympy typing problems. The suppressions were added automatically with the following script generated by GPT-4: ``` import re # Read the error file with open("error_file.txt", "r") as f: errors = f.readlines() # Parse the lines with errors and error types error_lines = {} for error in errors: match = re.match(r"(.*):(\d+):\d+: error:.*\[(.*)\]", error) if match: file_path, line_number, error_type = match.groups() if file_path not in error_lines: error_lines[file_path] = {} error_lines[file_path][int(line_number)] = error_type # Insert ignore comments in the source files for file_path, lines in error_lines.items(): with open(file_path, "r") as f: code = f.readlines() for line_number, error_type in sorted(lines.items(), key=lambda x: x[0], reverse=True): code[line_number - 1] = code[line_number - 1].rstrip() + f" # type: ignore[{error_type}]\n" with open(file_path, "w") as f: f.writelines(code) ``` Signed-off-by: Edward Z. Yang <ezyang@meta.com> Pull Request resolved: #118469 Approved by: https://github.com/Skylion007 ghstack dependencies: #118414, #118418, #118432, #118467, #118468

Signed-off-by: Edward Z. Yang <ezyang@meta.com> Pull Request resolved: #118475 Approved by: https://github.com/suo ghstack dependencies: #118414, #118418, #118432, #118467, #118468, #118469

As we force a specific version of mypy, it's OK to use the agglomerated flag. Signed-off-by: Edward Z. Yang <ezyang@meta.com> Pull Request resolved: #118479 Approved by: https://github.com/Skylion007, https://github.com/albanD ghstack dependencies: #118414, #118418, #118432, #118467, #118468, #118469, #118475

Signed-off-by: Edward Z. Yang <ezyang@meta.com> Pull Request resolved: #118480 Approved by: https://github.com/Skylion007, https://github.com/albanD ghstack dependencies: #118414, #118418, #118432, #118467, #118468, #118469, #118475, #118479

Signed-off-by: Edward Z. Yang <ezyang@meta.com> Pull Request resolved: #118481 Approved by: https://github.com/Skylion007, https://github.com/albanD ghstack dependencies: #118414, #118418, #118432, #118467, #118468, #118469, #118475, #118479, #118480

int3 · 2024-01-29T15:56:11Z

torch/_dynamo/test_case.py

@@ -44,14 +44,14 @@ def run_tests(needs=()):
 class TestCase(TorchTestCase):
    @classmethod
    def tearDownClass(cls):
-        cls._exit_stack.close()
+        cls._exit_stack.close()  # type: ignore[attr-defined]


we should add _exit_stack as an uninitialized class member instead

i.e.

class TestCase(TorchTestCase): _exit_stack: <type>

int3 · 2024-01-29T16:03:39Z

torch/_inductor/fx_passes/serialized_patterns/central_index.py

@@ -1,3 +1,5 @@
+# mypy: ignore-errors


this should probably be autogen'ed as part of gen_attention_patterns.py

int3 · 2024-01-29T16:04:17Z

torch/_inductor/fx_passes/mkldnn_fusion.py

@@ -343,11 +343,11 @@ def _is_valid_binary(match, fn):
        if any(
            not (
                hasattr(n.args[0], "meta")
-                and isinstance(n.args[0].meta.get("val", None), torch.Tensor)
+                and isinstance(n.args[0].meta.get("val", None), torch.Tensor)  # type: ignore[union-attr]


I guess we should probably type meta as a Dict of Any values?

SEND FIXES PLZ!!

The original motivation for MYPYINDUCTOR was a faster type checking configuration that only checked a subset of files. With the removal of `follow_imports = ignore`, we are now able to use dmypy to do fast incremental typechecking, eliminating the need for this. Perhaps erroneously, when I tee'ed up this PR I elected to delete the `follow_imports = skip` designations in the mypy-inductor.ini. This lead to a number of extra type error suppressions that I manually edited. You will need to review. Signed-off-by: Edward Z. Yang <ezyang@meta.com> Pull Request resolved: pytorch#118432 Approved by: https://github.com/Skylion007 ghstack dependencies: pytorch#118414, pytorch#118418

When using dmypy, this setting is enabled and cannot be turned off. Force it for regular mypy too. Signed-off-by: Edward Z. Yang <ezyang@meta.com> Pull Request resolved: pytorch#118467 Approved by: https://github.com/Skylion007 ghstack dependencies: pytorch#118414, pytorch#118418, pytorch#118432

Signed-off-by: Edward Z. Yang <ezyang@meta.com> Pull Request resolved: pytorch#118468 Approved by: https://github.com/Skylion007 ghstack dependencies: pytorch#118414, pytorch#118418, pytorch#118432, pytorch#118467

dmypy silently ignores follow_imports = skip, so to get parity between dmypy and mypy we have to suck it up and type: ignore all of the sympy typing problems. The suppressions were added automatically with the following script generated by GPT-4: ``` import re # Read the error file with open("error_file.txt", "r") as f: errors = f.readlines() # Parse the lines with errors and error types error_lines = {} for error in errors: match = re.match(r"(.*):(\d+):\d+: error:.*\[(.*)\]", error) if match: file_path, line_number, error_type = match.groups() if file_path not in error_lines: error_lines[file_path] = {} error_lines[file_path][int(line_number)] = error_type # Insert ignore comments in the source files for file_path, lines in error_lines.items(): with open(file_path, "r") as f: code = f.readlines() for line_number, error_type in sorted(lines.items(), key=lambda x: x[0], reverse=True): code[line_number - 1] = code[line_number - 1].rstrip() + f" # type: ignore[{error_type}]\n" with open(file_path, "w") as f: f.writelines(code) ``` Signed-off-by: Edward Z. Yang <ezyang@meta.com> Pull Request resolved: pytorch#118469 Approved by: https://github.com/Skylion007 ghstack dependencies: pytorch#118414, pytorch#118418, pytorch#118432, pytorch#118467, pytorch#118468

Signed-off-by: Edward Z. Yang <ezyang@meta.com> Pull Request resolved: pytorch#118475 Approved by: https://github.com/suo ghstack dependencies: pytorch#118414, pytorch#118418, pytorch#118432, pytorch#118467, pytorch#118468, pytorch#118469

As we force a specific version of mypy, it's OK to use the agglomerated flag. Signed-off-by: Edward Z. Yang <ezyang@meta.com> Pull Request resolved: pytorch#118479 Approved by: https://github.com/Skylion007, https://github.com/albanD ghstack dependencies: pytorch#118414, pytorch#118418, pytorch#118432, pytorch#118467, pytorch#118468, pytorch#118469, pytorch#118475

Signed-off-by: Edward Z. Yang <ezyang@meta.com> Pull Request resolved: pytorch#118480 Approved by: https://github.com/Skylion007, https://github.com/albanD ghstack dependencies: pytorch#118414, pytorch#118418, pytorch#118432, pytorch#118467, pytorch#118468, pytorch#118469, pytorch#118475, pytorch#118479

Signed-off-by: Edward Z. Yang <ezyang@meta.com> Pull Request resolved: pytorch#118481 Approved by: https://github.com/Skylion007, https://github.com/albanD ghstack dependencies: pytorch#118414, pytorch#118418, pytorch#118432, pytorch#118467, pytorch#118468, pytorch#118469, pytorch#118475, pytorch#118479, pytorch#118480

Unify MYPYINDUCTOR and MYPY

3b45254

Signed-off-by: Edward Z. Yang <ezyang@meta.com> [ghstack-poisoned]

ezyang requested review from BowenBao, abock, thiagocrepaldi and wschin as code owners January 26, 2024 22:46

ezyang mentioned this pull request Jan 26, 2024

Replace follow_imports = silent with normal #118414

Closed

pytorch-bot bot added the release notes: onnx torch.onnx related changes that should show up in the release notes label Jan 26, 2024

ezyang mentioned this pull request Jan 26, 2024

Upgrade mypy python_version to 3.11 #118418

Closed

github-actions bot added module: inductor module: dynamo ciflow/inductor labels Jan 26, 2024

Skylion007 approved these changes Jan 26, 2024

View reviewed changes

ezyang requested review from int3, malfet and albanD January 26, 2024 23:51

ezyang added the topic: not user facing topic category label Jan 27, 2024

ezyang added a commit that referenced this pull request Jan 27, 2024

Unify MYPYINDUCTOR and MYPY

385df79

Signed-off-by: Edward Z. Yang <ezyangmeta.com> ghstack-source-id: 3f27751d06d3f4b5d71e79e23858660478b2e194 Pull Request resolved: #118432

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Jan 27, 2024

This was referenced Jan 27, 2024

[mypy] Remove colorama ignore_missing_imports #118468

Closed

Remove follow_imports = skip from sympy #118469

Closed

pytorchmergebot closed this in d03173e Jan 27, 2024

pytorchmergebot added Merged and removed merging labels Jan 27, 2024

int3 reviewed Jan 29, 2024

View reviewed changes

facebook-github-bot deleted the gh/ezyang/2527/head branch January 31, 2024 15:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unify MYPYINDUCTOR and MYPY #118432

Unify MYPYINDUCTOR and MYPY #118432

ezyang commented Jan 26, 2024 •

edited

Loading

pytorch-bot bot commented Jan 26, 2024 •

edited

Loading

ezyang commented Jan 27, 2024

int3 Jan 29, 2024

int3 Jan 29, 2024

int3 Jan 29, 2024

ezyang Jan 29, 2024

Unify MYPYINDUCTOR and MYPY #118432

Unify MYPYINDUCTOR and MYPY #118432

Conversation

ezyang commented Jan 26, 2024 • edited Loading

pytorch-bot bot commented Jan 26, 2024 • edited Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/118432

✅ No Failures

ezyang commented Jan 27, 2024

int3 Jan 29, 2024

Choose a reason for hiding this comment

int3 Jan 29, 2024

Choose a reason for hiding this comment

int3 Jan 29, 2024

Choose a reason for hiding this comment

ezyang Jan 29, 2024

Choose a reason for hiding this comment

ezyang commented Jan 26, 2024 •

edited

Loading

pytorch-bot bot commented Jan 26, 2024 •

edited

Loading