[WIP] Python 3.9 parser #137

emmatyping · 2020-05-09T05:25:02Z

Fixes #118

emmatyping · 2020-05-09T05:34:40Z

I'm not really sure how best to get this reviewed. I probably should have created a commit when I imported the files...

My process was basically:

Import files
Update symbols/asdl_c/peg_generator
generate AST c files and Parser/pegen/parse.c
rewrite Custom/typed_ast.c to use the pegen API.
make it build (some extracting of CPython things needed, see for example Custom/pegen_extra.c and Include/pegen_extra.c
Make it build with older Python.

~~I'll take a look at Windows this weekend.~~ Looks like Windows builds are fine! Except 3.5, which this won't support.

gvanrossum · 2020-05-09T16:01:14Z

Hm... Something failing in the tests because it's using the walrus. I guess that's the meta-parser (primarily). Serhiy actually has attempted a fix for that, maybe it's useful?

This should be released as typed_ast 2.0 I suppose.

I will look at it in detail after breakfast. :-)

gvanrossum · 2020-05-09T18:14:26Z

I'm not really sure how best to get this reviewed. I probably should have created a commit when I imported the files...

Yeah, that would have helped. Is it too late to redo it? I guess you could create a temporary branch with that and then rebase this branch over that.

So far I've managed to build and install it using pip install .. When I do this using Python 3.7, and then run mypy, I immediately get this failure:

Traceback (most recent call last):
  File "/Users/guido/v37/bin/mypy", line 6, in <module>
    from mypy.__main__ import console_entry
  File "/Users/guido/v37/lib/python3.7/site-packages/mypy/__main__.py", line 4, in <module>
    from mypy.main import main
  File "mypy/main.py", line 13, in <module>
  File "mypy/build.py", line 33, in <module>
  File "mypy/semanal.py", line 95, in <module>
  File "mypy/typeanal.py", line 30, in <module>
  File "mypy/exprtotype.py", line 10, in <module>
  File "mypy/fastparse.py", line 80, in <module>
AttributeError: module 'typed_ast.ast3' has no attribute 'Ellipsis'

It seems ast.Ellipsis was deprecated and somehow didn't make it into your port? But mypy expects it.

Also, testing with mypy and Python 3.8 is tricky because mypy checks sys.version and uses ast instead of typed_ast if it's >= (3, 8).

gvanrossum · 2020-05-09T18:25:26Z

Running the tests manually (on Mac) is a pain. I tried pytest but that fails with essentially

ImportError: cannot import name '_ast3' from 'typed_ast' (/Users/guido/typed_ast/typed_ast/__init__.py)

Forcing PYTHONPATH to the build/lib* subdirectory where the built files live doesn't help.

Then I tried python setup.py test which succeeded -- with 0 tests run.

But this copies the .so file into the source tree. (!) So I tried pytest again.

Now it segfaults:

(v38) ~/typed_ast$ pytest -v
================================================== test session starts ==================================================
platform darwin -- Python 3.8.3rc1, pytest-5.3.2, py-1.8.0, pluggy-0.13.1 -- /Users/guido/v38/bin/python3.8
cachedir: .pytest_cache
benchmark: 3.2.3 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /Users/guido/typed_ast
plugins: forked-1.0.2, xdist-1.31.0, benchmark-3.2.3, cov-2.8.1
collected 15 items                                                                                                      

ast3/tests/test_basics.py::test_basics FAILED                                                                     [  6%]
ast3/tests/test_basics.py::test_redundantdef PASSED                                                               [ 13%]
ast3/tests/test_basics.py::test_vardecl FAILED                                                                    [ 20%]
ast3/tests/test_basics.py::test_forstmt FAILED                                                                    [ 26%]
ast3/tests/test_basics.py::test_withstmt FAILED                                                                   [ 33%]
ast3/tests/test_basics.py::test_longargs FAILED                                                                   [ 40%]
ast3/tests/test_basics.py::test_ignores FAILED                                                                    [ 46%]
ast3/tests/test_basics.py::test_asyncfunc FAILED                                                                  [ 53%]
ast3/tests/test_basics.py::test_asyncvar FAILED                                                                   [ 60%]
ast3/tests/test_basics.py::test_asynccomp Fatal Python error: Segmentation fault

Current thread 0x0000000107bcfdc0 (most recent call first):
  File "/Users/guido/typed_ast/ast3/tests/test_basics.py", line 262 in test_asynccomp
  File "/Users/guido/v38/lib/python3.8/site-packages/_pytest/python.py", line 166 in pytest_pyfunc_call
  File "/Users/guido/v38/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/Users/guido/v38/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/Users/guido/v38/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/Users/guido/v38/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/Users/guido/v38/lib/python3.8/site-packages/_pytest/python.py", line 1435 in runtest
  File "/Users/guido/v38/lib/python3.8/site-packages/_pytest/runner.py", line 131 in pytest_runtest_call
  File "/Users/guido/v38/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/Users/guido/v38/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/Users/guido/v38/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/Users/guido/v38/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/Users/guido/v38/lib/python3.8/site-packages/_pytest/runner.py", line 207 in <lambda>
  File "/Users/guido/v38/lib/python3.8/site-packages/_pytest/runner.py", line 234 in from_call
  File "/Users/guido/v38/lib/python3.8/site-packages/_pytest/runner.py", line 206 in call_runtest_hook
  File "/Users/guido/v38/lib/python3.8/site-packages/_pytest/runner.py", line 182 in call_and_report
  File "/Users/guido/v38/lib/python3.8/site-packages/_pytest/runner.py", line 96 in runtestprotocol
  File "/Users/guido/v38/lib/python3.8/site-packages/_pytest/runner.py", line 81 in pytest_runtest_protocol
  File "/Users/guido/v38/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/Users/guido/v38/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/Users/guido/v38/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/Users/guido/v38/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/Users/guido/v38/lib/python3.8/site-packages/_pytest/main.py", line 270 in pytest_runtestloop
  File "/Users/guido/v38/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/Users/guido/v38/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/Users/guido/v38/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/Users/guido/v38/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/Users/guido/v38/lib/python3.8/site-packages/_pytest/main.py", line 246 in _main
  File "/Users/guido/v38/lib/python3.8/site-packages/_pytest/main.py", line 196 in wrap_session
  File "/Users/guido/v38/lib/python3.8/site-packages/_pytest/main.py", line 239 in pytest_cmdline_main
  File "/Users/guido/v38/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/Users/guido/v38/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/Users/guido/v38/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/Users/guido/v38/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/Users/guido/v38/lib/python3.8/site-packages/_pytest/config/__init__.py", line 91 in main
  File "/Users/guido/v38/bin/pytest", line 8 in <module>
Segmentation fault: 11
(v38) ~/typed_ast$

So at least we know there's something fishy with async comprehensions...

gvanrossum · 2020-05-09T18:27:14Z

Oh, it's simpler.

(v38) ~/typed_ast$  python
Python 3.8.3rc1 (tags/v3.8.3rc1:802eb676ba, Apr 30 2020, 11:47:35) 
[Clang 11.0.0 (clang-1100.0.33.8)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Cannot read termcap database;
using dumb terminal settings.
>>> from typed_ast import ast3; ast3.parse("pass")
Segmentation fault: 11
(v38) ~/typed_ast$

gvanrossum · 2020-05-09T18:34:40Z

I have a bit more from lldb, but then I'll stop (probably better to try and understand what you did before trying to debug the segfault).

The crash is at line 410 in pegen.c:

        if (t->col_offset == -1) {

It appears t is nonsense. This in turn appears because at line 388, t is initialized like this:

    Token *t = p->known_err_token != NULL ? p->known_err_token : p->tokens[p->fill - 1];

and the problem is that p->known_err_token is NULL, but p->fill is 0, so the expression p->tokens[p->fill - 1] produces nonsense.

All that points to deeper problems. Probably it's easier for you.

emmatyping · 2020-05-09T21:06:38Z

Thank you for looking into this! I will definitely be able to redo it. And I was going to stream some later today anyway, so I will also record it (maybe that will also help 🤷‍♂️)

gvanrossum · 2020-05-10T00:55:32Z

Whatever can be automated in the adaptation of the copied files to work here should be automated (as should the copying itself, perhaps). We will be doing this many times. I wouldn't expect that it's only the grammar that changes (then we could just regenerate parse.c and be done) -- I expect plenty of changes to the support code as well, sometimes systematic changes like python/cpython#19987.

emmatyping · 2020-05-10T10:55:13Z

Superseded by #138

Import Python 3.9 parser

ed9113d

the-knights-who-say-ni added the CLA signed label May 9, 2020

Add notes on the origin of the backports

9a20276

emmatyping mentioned this pull request May 10, 2020

[WIP] Automated 3.9 update #138

Closed

emmatyping closed this May 10, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Python 3.9 parser #137

[WIP] Python 3.9 parser #137

emmatyping commented May 9, 2020

emmatyping commented May 9, 2020 •

edited

Loading

gvanrossum commented May 9, 2020

gvanrossum commented May 9, 2020

gvanrossum commented May 9, 2020

gvanrossum commented May 9, 2020

gvanrossum commented May 9, 2020

emmatyping commented May 9, 2020

gvanrossum commented May 10, 2020

emmatyping commented May 10, 2020

[WIP] Python 3.9 parser #137

[WIP] Python 3.9 parser #137

Conversation

emmatyping commented May 9, 2020

emmatyping commented May 9, 2020 • edited Loading

gvanrossum commented May 9, 2020

gvanrossum commented May 9, 2020

gvanrossum commented May 9, 2020

gvanrossum commented May 9, 2020

gvanrossum commented May 9, 2020

emmatyping commented May 9, 2020

gvanrossum commented May 10, 2020

emmatyping commented May 10, 2020

emmatyping commented May 9, 2020 •

edited

Loading