bpo-36500: Enable Windows users to regen grammar, opcodes, tokens and symbols from Visual Studio #12654

tonybaloney · 2019-04-02T00:17:06Z

Adds an additional project to the PCbuild.sln that runs the equivalent commands that are in the regen-all stage in the Makefile.

Currently implements:

regen-grammar
regen-opcode
regen-token
regen-keyword
regen-symbol
regen-ast

Has a custom clean stage to delete intermediate files.

Uses the generated python.exe from pythoncore, and so marks pythoncore as a depedency.

The solution file does not mark this project as to be built in the Debug or Release profiles (same as freezeimportlib)

@zooba

TODO:

Remove references to new project versions

https://bugs.python.org/issue36500

…pythoncore as uses python*.exe binary to run the new Pgen module

tonybaloney · 2019-04-02T00:38:54Z

Added regen-keyword and regen-symbol equivalent stages.

tonybaloney · 2019-04-02T01:14:12Z

Added regen-opcode, regen-token. Only regen-ast left (if required?)

tonybaloney · 2019-04-02T01:49:09Z

Added regen-ast and the ASDL/ast related files

zooba

Right now this can't be a part of every build because it touches too many files. That's okay, but then I'd like a PCbuild/regen.bat script for launching it from the command line.

Alternatively, this could regenerate everything but only update source files that have changed (and break the build at that point), and otherwise leave everything clean. Then we could make it part of the regular build and wouldn't need the current freeze_importlib project. But it's got to not touch anything that doesn't need touching and not pollute the source directories.

PCbuild/regen.vcxproj

zooba · 2019-04-02T17:16:14Z

PCbuild/regen.vcxproj

+    <!-- Regenerate Include/Python-ast.h using Parser/asdl_c.py -h -->
+    <Exec Command="&quot;$(OutDir)python$(PyDebugExt).exe&quot; &quot;..\Parser\asdl_c.py&quot; -h &quot;..\Include\Python-ast.h.new&quot; &quot;..\Parser\Python.asdl&quot;" />
+    <Copy SourceFiles="..\Include\Python-ast.h.new" DestinationFiles="..\Include\Python-ast.h">
+      <Output TaskParameter="CopiedFiles" ItemName="_UpdatedH" />


This isn't sufficient for detecting files that were regenerated without changing. Look in the importlib one for the contents check.

(If you want an excuse to learn about target batching in MSBuild - which is ridiculously powerful - you might consider having one target for doing the optional-copy and fail if updated step, then pass it all the target files in an ItemGroup with the generated file as metadata.)

@zooba It copies them regardless in the same way that the Makefile does, which is why this project is an optional configuration and not part of the Debug or Release build dependencies.
Would you like me to add this check and make regen.vcxproj build automatically?

Yep, that'll be most reliable.

If you add all the source files as items in the project then it should skip rebuilding if none of them have changed.

I've tried the target batching, it doesn't seem to be able to determine any better than this revision whether the contents have changed. The docs say that it uses timestamps on the files to determine (which won't work in this case)

Yeah, you'll need to do content comparisons. See what the current freezeimportlib project does.

tonybaloney · 2019-05-09T02:43:43Z

@zooba please could you review, instead of having regen.bat, I've added a --regen option to build.bat so it shares the same configuration flags.
Have made the other changes you requested and looked into target batching (see comment)

tonybaloney · 2019-05-09T02:50:14Z

Actually, I’m going to remove the project from the solution and remove the errors and turn them into warnings

zooba · 2019-05-15T16:56:57Z

PCbuild/regen.vcxproj

+  </Target>
+  <Target Name="_RegenOpcodes" AfterTargets="_RegenAST_C">
+    <!-- Regenerate Include/opcode.h from Lib/opcode.py using Tools/scripts/generate_opcode_h.py-->
+    <Exec Command="&quot;$(PythonExe)&quot; ..\Tools\scripts\generate_opcode_h.py &quot;..\Lib\opcode.py&quot; &quot;$(IntDir)opcode.h&quot;" />


I'm not keen on the relative paths here. We have $(PySourcePath) for the root of the sources.

tonybaloney · 2019-08-21T03:29:21Z

@zooba please can you review with the changes you requested.

Misc/NEWS.d/next/Build/2019-04-02-01-59-26.bpo-36500.fyG6_U.rst

…er must have built the binaries first before using the process. Also fix a leading '.' in the vcxproj

tonybaloney · 2019-08-22T05:28:01Z

Modified the build.bat so that --regen will run after the primary build, it will issue warnings about the source files that it has updated:

C:\Users\anthonyshaw\source\repos\cpython\PCbuild>if not ERRORLEVEL 1 "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin\msbuild.exe" "C:\Users\anthonyshaw\source\repos\cpython\PCbuild\regen.vcxproj" /t:Build /m /nologo /v:m /p:IncludeExternals=true /p:Configuration=Release /p:Platform=Win32 /p:UseTestMarker= /p:GIT="C:\Program Files\Git\cmd\git.exe"
  Killing any running python.exe instances...
  Getting build info from "C:\Program Files\Git\cmd\git.exe"
  Building heads/pcbuildregen-dirty:78639928d0 pcbuildregen
  pythoncore.vcxproj -> C:\Users\anthonyshaw\source\repos\cpython\PCbuild\win32\python39.dll
C:\Users\anthonyshaw\source\repos\cpython\PCbuild\regen.vcxproj(167,5): warning : Grammar updated. You will need to reb
uild pythoncore to see the changes.
C:\Users\anthonyshaw\source\repos\cpython\PCbuild\regen.vcxproj(175,5): warning : Python-ast.h updated. You will need t
o rebuild pythoncore to see the changes.
C:\Users\anthonyshaw\source\repos\cpython\PCbuild\regen.vcxproj(183,5): warning : Python-ast.c updated. You will need t
o rebuild pythoncore to see the changes.
  C:\Users\anthonyshaw\source\repos\cpython\PCbuild\obj\39win32_Release\regen\opcode.h regenerated from C:\Users\anthon
  yshaw\source\repos\cpython\Lib\opcode.py
C:\Users\anthonyshaw\source\repos\cpython\PCbuild\regen.vcxproj(191,5): warning : Opcodes updated. You will need to reb
uild pythoncore to see the changes.
C:\Users\anthonyshaw\source\repos\cpython\PCbuild\regen.vcxproj(209,5): warning : Keywords updated. You will need to re
build pythoncore to see the changes.

Co-Authored-By: Kyle Stanley <aeros167@gmail.com>

tonybaloney · 2019-10-24T20:52:33Z

@zooba please could you look at this for going into 3.9-master?

…an regen grammar, opcodes, tokens and symbols (pythonGH-12654)

Add regen-grammar equivalent as optional build project, dependant on …

f291fb1

…pythoncore as uses python*.exe binary to run the new Pgen module

tonybaloney requested a review from a team as a code owner April 2, 2019 00:17

bedevere-bot added the awaiting review label Apr 2, 2019

the-knights-who-say-ni added the CLA signed label Apr 2, 2019

tonybaloney added 5 commits April 2, 2019 11:20

Downgrade solution file

0f4b23a

Remove win10 toolset configurations manually

759e241

Update pcbuild.sln

192f358

add regen-keywords and regen-symbols build stages

5eb7795

Merge remote-tracking branch 'upstream/pcbuildregen' into pcbuildregen

4017063

tonybaloney changed the title ~~bpo-36500: Enable Windows users to regen parser tables from Visual Studio~~ bpo-36500: Enable Windows users to regen grammar, tokens and symbols from Visual Studio Apr 2, 2019

tonybaloney added 2 commits April 2, 2019 12:02

Copy both C and H for grammar, regen opcodes and order process

0be5632

All regen stages with comments (except regen-ast)

a2c1e0c

tonybaloney changed the title ~~bpo-36500: Enable Windows users to regen grammar, tokens and symbols from Visual Studio~~ bpo-36500: Enable Windows users to regen grammar, opcodes, tokens and symbols from Visual Studio Apr 2, 2019

tonybaloney added 2 commits April 2, 2019 12:44

Added regen-ast in 2 stages

b52329f

Add the other artifacts to the linked files

95eaa9f

📜🤖 Added by blurb_it.

1396fd9

zooba reviewed Apr 2, 2019

View reviewed changes

tonybaloney and others added 3 commits May 9, 2019 11:43

Update with feedback from Steve

80d2d9a

Error on regenerated files in Release configuration

45c7999

Add regen to the build.bat

2162db2

Make regen a seperate project

89246a4

zooba reviewed May 15, 2019

View reviewed changes

tonybaloney added 2 commits August 21, 2019 13:19

Merge branch 'master' into pcbuildregen

546cea5

Swap rel paths with $(PySourcePath)

eb87863

tonybaloney added 2 commits August 21, 2019 16:06

Bind project path correctly

9fca9db

merge upstream

7863992

aeros added the OS-windows label Aug 22, 2019

aeros reviewed Aug 22, 2019

View reviewed changes

Misc/NEWS.d/next/Build/2019-04-02-01-59-26.bpo-36500.fyG6_U.rst Outdated Show resolved Hide resolved

aeros added the type-feature A feature request or enhancement label Aug 22, 2019

Change the regen to be a stage after the primary build so that the us…

370fcba

…er must have built the binaries first before using the process. Also fix a leading '.' in the vcxproj

tonybaloney and others added 2 commits August 22, 2019 15:28

Update Misc/NEWS.d/next/Build/2019-04-02-01-59-26.bpo-36500.fyG6_U.rst

8f93907

Co-Authored-By: Kyle Stanley <aeros167@gmail.com>

Update 2019-04-02-01-59-26.bpo-36500.fyG6_U.rst

0a6ab75

zooba merged commit 9e36589 into python:master Dec 17, 2019

bedevere-bot removed the awaiting review label Dec 17, 2019

shihai1991 pushed a commit to shihai1991/cpython that referenced this pull request Jan 31, 2020

bpo-36500: Add --regen option to PCbuild/build.bat so Windows users c…

fa86028

…an regen grammar, opcodes, tokens and symbols (pythonGH-12654)

tonybaloney deleted the pcbuildregen branch May 31, 2021 00:09

ammaraskar mentioned this pull request Jul 27, 2021

Mention build.bat counterpart in GH workflows error message #26770

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bpo-36500: Enable Windows users to regen grammar, opcodes, tokens and symbols from Visual Studio #12654

bpo-36500: Enable Windows users to regen grammar, opcodes, tokens and symbols from Visual Studio #12654

tonybaloney commented Apr 2, 2019 •

edited

Loading

tonybaloney commented Apr 2, 2019

tonybaloney commented Apr 2, 2019

tonybaloney commented Apr 2, 2019

zooba left a comment

zooba Apr 2, 2019

tonybaloney Apr 2, 2019 •

edited

Loading

zooba Apr 12, 2019

tonybaloney May 9, 2019

zooba May 15, 2019

tonybaloney commented May 9, 2019

tonybaloney commented May 9, 2019

zooba May 15, 2019

tonybaloney commented Aug 21, 2019

tonybaloney commented Aug 22, 2019

tonybaloney commented Oct 24, 2019

bpo-36500: Enable Windows users to regen grammar, opcodes, tokens and symbols from Visual Studio #12654

bpo-36500: Enable Windows users to regen grammar, opcodes, tokens and symbols from Visual Studio #12654

Conversation

tonybaloney commented Apr 2, 2019 • edited Loading

tonybaloney commented Apr 2, 2019

tonybaloney commented Apr 2, 2019

tonybaloney commented Apr 2, 2019

zooba left a comment

Choose a reason for hiding this comment

zooba Apr 2, 2019

Choose a reason for hiding this comment

tonybaloney Apr 2, 2019 • edited Loading

Choose a reason for hiding this comment

zooba Apr 12, 2019

Choose a reason for hiding this comment

tonybaloney May 9, 2019

Choose a reason for hiding this comment

zooba May 15, 2019

Choose a reason for hiding this comment

tonybaloney commented May 9, 2019

tonybaloney commented May 9, 2019

zooba May 15, 2019

Choose a reason for hiding this comment

tonybaloney commented Aug 21, 2019

tonybaloney commented Aug 22, 2019

tonybaloney commented Oct 24, 2019

tonybaloney commented Apr 2, 2019 •

edited

Loading

tonybaloney Apr 2, 2019 •

edited

Loading