gh-130587: Add hand-written docs for non-OP tokens by encukou · Pull Request #130588 · python/cpython

encukou · 2025-02-26T15:40:38Z

Add hand-written docs for non-OP tokens
Make the automation (generate_token.py) check that the hand-written docs are present, and only generate docs for the OP tokens
Switch to list-table for the OP tokens, to make their docs more compact
Add ENDMARKER to the grammar docs where it appears (toplevel components)
Add forgotten versionchanged entry for EXCLAMATION
Remove docs for NT_OFFSET

Issue: token module documentation is incomplete #130587

📚 Documentation preview 📚: https://cpython-previews--130588.org.readthedocs.build/

…e rest

…ARKER to that

Co-authored-by: Blaise Pabon <blaise@gmail.com>

encukou · 2025-02-26T16:44:41Z

The rendered docs are at https://cpython-previews--130588.org.readthedocs.build/en/130588/library/token.html

encukou · 2025-02-26T16:45:20Z

@lysnikolaou, does this look reasonable to you?

Doc/library/token-list.inc

Doc/library/token.rst

Doc/reference/toplevel_components.rst

Tools/build/generate_token.py

Doc/library/token.rst

lysnikolaou

This looks like a good improvement to me! Thanks @encukou!

I've left some inline comments regarding some specifics in the docs.

Doc/library/token.rst

Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>

encukou · 2025-02-27T11:00:52Z

Thank you for the reviews! I addressed some; I'll continue next week.

encukou · 2025-03-05T15:37:57Z

Doc/library/token.rst

+Note that a token's value may depend on tokenizer options. For example, a
+``"+"`` token may be reported as either :data:`PLUS` or :data:`OP`, or
+a ``"match"`` token may be either :data:`NAME` or :data:`SOFT_KEYWORD`.
+


I added this note to clarify the “fuzziness” of these categories.

Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

miss-islington-app · 2025-03-19T15:46:34Z

Thanks @encukou for the PR 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖

miss-islington-app · 2025-03-19T15:46:37Z

Sorry, @encukou, I could not cleanly backport this to 3.13 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 4bced29a74c4dbcf310a47e8202292aaa39b617b 3.13

bedevere-app · 2025-03-19T15:50:01Z

GH-131465 is a backport of this pull request to the 3.13 branch.

…0588) Co-authored-by: Blaise Pabon <blaise@gmail.com> Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com> (cherry picked from commit 4bced29)

…onGH-130588) (cherry picked from commit 4bced29) Co-authored-by: Petr Viktorin <encukou@gmail.com> Co-authored-by: Blaise Pabon <blaise@gmail.com> Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>

…GH-131465) Co-authored-by: Blaise Pabon <blaise@gmail.com> Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>

encukou and others added 12 commits January 22, 2025 17:57

generate_tokens.py: Only generate docs for 'literal' tokens; check th…

7129f00

…e rest

Change docs for the "literal" tokens to a list-table

d400ae7

Merge in the main branch

352611b

Document most of the tokens; improve top-level grammar docs link ENDM…

4411c37

…ARKER to that

Write prose; reorganize the token list

1bf5511

Fixups

e1e498b

Merge in the main branch

56c5d7c

Correct directive name

8bbbb0f

Co-authored-by: Blaise Pabon <blaise@gmail.com>

Don't use the Gather syntax

eed407e

Co-authored-by: Blaise Pabon <blaise@gmail.com>

Revert some changes to the toplevel_components

0dd236f

Merge in the main branch

90cd1e2

Revert an order change

dca268e

encukou requested review from AA-Turner and willingc as code owners February 26, 2025 15:40

bedevere-app bot added the awaiting core review label Feb 26, 2025

bedevere-app bot mentioned this pull request Feb 26, 2025

token module documentation is incomplete #130587

Closed

encukou added docs Documentation in the Doc dir skip news labels Feb 26, 2025

AA-Turner reviewed Feb 26, 2025

View reviewed changes

StanFromIreland reviewed Feb 26, 2025

View reviewed changes

Doc/library/token.rst Outdated Show resolved Hide resolved

Doc/library/token.rst Outdated Show resolved Hide resolved

Doc/library/token.rst Show resolved Hide resolved

lysnikolaou reviewed Feb 27, 2025

View reviewed changes

Doc/library/token.rst Outdated Show resolved Hide resolved

Doc/library/token.rst Outdated Show resolved Hide resolved

Doc/library/token.rst Outdated Show resolved Hide resolved

Doc/library/token.rst Outdated Show resolved Hide resolved

Doc/library/token.rst Outdated Show resolved Hide resolved

encukou commented Feb 27, 2025

View reviewed changes

Doc/library/token.rst Outdated Show resolved Hide resolved

encukou commented Feb 27, 2025

View reviewed changes

Doc/library/token.rst Outdated Show resolved Hide resolved

Apply suggestions from code review

e02ced8

Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>

Add a note and reword OP docs

02fff75

encukou commented Mar 5, 2025

View reviewed changes

Apply suggestions from code review

9b6eb2a

Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

encukou merged commit 4bced29 into python:main Mar 19, 2025
39 checks passed

bedevere-app bot removed the awaiting core review label Mar 19, 2025

github-project-automation bot moved this from Todo to Done in Docs PRs Mar 19, 2025

encukou deleted the token-docs branch March 19, 2025 15:42

encukou added the needs backport to 3.13 bugs and security fixes label Mar 19, 2025

miss-islington-app bot assigned encukou Mar 19, 2025

bedevere-app bot removed the needs backport to 3.13 bugs and security fixes label Mar 19, 2025

encukou mentioned this pull request Mar 24, 2025

Align the grammar documentation with Python's actual grammar #127833

Open

Uh oh!

Conversation

encukou commented Feb 26, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

encukou commented Feb 26, 2025

Uh oh!

encukou commented Feb 26, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lysnikolaou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

encukou commented Feb 27, 2025

Uh oh!

encukou Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

miss-islington-app bot commented Mar 19, 2025

Uh oh!

miss-islington-app bot commented Mar 19, 2025

Uh oh!

bedevere-app bot commented Mar 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

encukou commented Feb 26, 2025 •

edited by github-actions bot

Loading

encukou Mar 5, 2025 •

edited

Loading