Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion developer-workflow/grammar.rst
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,8 @@ Note: sometimes things mysteriously don't work. Before giving up, try ``make cl
Then run ``make regen-ast`` to regenerate
:cpy-file:`Include/internal/pycore_ast.h` and :cpy-file:`Python/Python-ast.c`.

* :cpy-file:`Parser/tokenizer.c` contains the tokenization code.
* :cpy-file:`Parser/lexer/` and :cpy-file:`Parser/tokenizer/` contain the
tokenization code.
This is where you would add a new type of comment or string literal, for example.

* :cpy-file:`Python/ast.c` will need changes to validate AST objects
Expand Down
2 changes: 1 addition & 1 deletion internals/compiler.rst
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Abstract

In CPython, the compilation from source code to bytecode involves several steps:

1. Tokenize the source code (:cpy-file:`Parser/tokenizer.c`).
1. Tokenize the source code (:cpy-file:`Parser/lexer/` and :cpy-file:`Parser/tokenizer/`).
2. Parse the stream of tokens into an Abstract Syntax Tree
(:cpy-file:`Parser/parser.c`).
3. Transform AST into an instruction sequence (:cpy-file:`Python/compile.c`).
Expand Down
5 changes: 3 additions & 2 deletions internals/parser.rst
Original file line number Diff line number Diff line change
Expand Up @@ -599,8 +599,9 @@ If you are on Windows you can use the Visual Studio project files to regenerate

./PCbuild/build.bat --regen

How tokens are generated and the rules governing this is completely up to the tokenizer (:cpy-file:`Parser/tokenizer.c`)
and the parser just receives tokens from it.
How tokens are generated and the rules governing this are completely up to the tokenizer
(:cpy-file:`Parser/lexer/` and :cpy-file:`Parser/tokenizer/`);
the parser just receives tokens from it.

Memoization
-----------
Expand Down