Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GH-98831: Refactor generate_cases.py #99408

Closed
wants to merge 36 commits into from

Conversation

gvanrossum
Copy link
Member

@gvanrossum gvanrossum commented Nov 12, 2022

Consider this after GH-99313.

gvanrossum and others added 27 commits November 9, 2022 18:38
Had to refactor the parser a bit for this.
Replace Py_INCREF() and Py_XINCREF() with Py_NewRef() and
Py_XNewRef() in C files of the Objects/ directory.
Replace Py_INCREF() and Py_XINCREF() with Py_NewRef() and
Py_XNewRef() in Objects/dictobject.c.
…thon#99280)

Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
… venvs (pythonGH-99206)

Check to see if `base_executable` exists. If it does not, attempt
to use known alternative names of the python binary to find an
executable in the path specified by `home`.

If no alternative is found, previous behavior is preserved.

Signed-off-by: Vincent Fazio <vfazio@gmail.com>

Signed-off-by: Vincent Fazio <vfazio@gmail.com>
python#99271)

Also mark those opcodes that have no stack effect as such.

Co-authored-by: Brandt Bucher <brandtbucher@gmail.com>
@gvanrossum
Copy link
Member Author

My expectation at this point is that converting the majority of instructions to non-legacy form (cache and stack effects) will be a big project, even if we skip opcodes needing arrays or conditional stack effects.

The good news is that it can be parallelized -- each family can be done independently from any others, so anyone who wants to can pick a family and convert it, as long as it doesn't require fixing the generator (it's probably best if I stay on top of that).

I don't know how much we need to do before we can start experimenting with "macro" opcodes (see faster-cpython/ideas#491). Possibly we can start with an example like the one Mark wrote up in the DSL definition.

@markshannon
Copy link
Member

markshannon commented Nov 15, 2022

I won't attempt a review of this PR, but a few comments on how I'd like the tool(s) to look.

  • This is a compiler, so it should be built like one: Dumb IR(s) and multiple passes.
  • Any methods on IR classes should be simple query method. Put all processing in the passes.

A possible design:

  • Tokenizer: Tokenize the whole input file, so that locations are correct.
  • Parser: Parse instructions
  • Analyzer: Group and check families, check stack effects. Compute necessary labels and tables, instruction size, etc.
  • Code gen: Generate C code as stream of tokens and strings
  • Formatter: Format resulting C code, inserting #line directives and whitespace where needed.

Then, when we want to generate a trace interpreter, a register interpreter, compiler front-end, etc. we can re-use the tokenizer, parser, and formatter.

@gvanrossum
Copy link
Member Author

Thanks, I'll try to use that as a north star. In the meantime I am hoping to keep things working the way they were as much as possible to avoid disrupting other work. I have half the analyzer in this refactor, what's missing is a design for the IR to target.

@gvanrossum
Copy link
Member Author

I'm starting over from scratch now GH-99313 is merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants