build: ongoing list of actions for Python 3 compatibility

We have several threads in issues and pull requests related to Python 3 compatibility. We need a task list to keep in sync and to show our progress

QA:: Our CI machines are mix of machines that have Python 2, have Python 3, have both, have a python symlink, have it not, etc. By preferring Python 3 in configure/make/test some set of our current machines will start to test against Python 3, but many will continue to test against Python 2. The more modern machines will use 3, as will our users running more up-to-date operating systems in the next year.

Upstream V8 issues:

Neither Chromium or depot_tools are Python 3 compatible yet. This doesn't effect the main Node.js builds, but does affect our V8 integration builds, where v8 is built and tested.

migration of Chromium to Python 3: https://bugs.chromium.org/p/chromium/issues/detail?id=942720
specific one for depot_tools: https://bugs.chromium.org/p/chromium/issues/detail?id=984182

Miscellaneous non-blocking cleanups

~~Travis CI does not test ./configure --ninja test test: adding test for configure ninja #30033~~ reverted
configure.py: upgrade from optparse to argparse build: replace optparse for argparse in configure.py #29814
Tests do not test ./configure --ninja Tests do not test ./configure --ninja #29415 reopened
sort((MSVSFolder, MSVSProject)) node-gyp Python 3 support node-addon-api#445 (comment)
sort out the home for tap2junit: Transfer jenkins CI tool tap2junit into Node.js org admin#413
investigate using nodejs/node-gyp:gyp/ as up-stream for nodejs/node:tools/gyp/, see The GYP issue: where, how and who to maintain node-gyp#1791 for more information and alternate proposals.

Please use the "python" label!

@cclauss

@cclauss This is one neatly compiled list. Thanks for doing that 👍 I am just wondering if we can start dealing with GYP's Python 3 compatibility.

GYP got a few patches for Python 3 compatibility upstream: https://chromium.googlesource.com/external/gyp/+log

@refack

Oh, that's great. @refack and @cclauss have been working on @refack's fork of GYP at https://github.com/refack/GYP in this regard. Perhaps we can consolidate our efforts.

I will like to talk ci involved this. I know we are current introduce docker in node ci system.

We can introduce a docker image only with python3 to test our code.

One more thing, not sure this is the right place. We may need to pre-install some python3 package.
Package utils not include in python3.

node/tools/test.py

Line 43 in 86a49ce

import utils

node/tools/utils.py the file in question. Perhaps we need to start adding from __future__ import absolute_import.

https://stackoverflow.com/questions/33743880/what-does-from-future-import-absolute-import-actually-do

Maybe need to install python3 in this file too https://github.com/nodejs/node/blob/26cfaf4c1803d3d95c875865a1c671b914a16f4b/tools/bootstrap/windows_boxstarter.

tools: windows_boxstarter "choco install python -y" for Python 3

@sam-github

@sam-github @cclauss The switch to preferring Python 3 seems to have broken our V8 CI on centos7-ppcle.

See https://ci.nodejs.org/job/node-test-commit-v8-linux/2590/nodes=centos7-ppcle,v8test=v8test/console

07:58:31 + ./configure
07:58:32 Node configure: Found Python 3.7.3...
07:58:32 Traceback (most recent call last):
07:58:32   File "./configure", line 25, in <module>
07:58:32     import configure
07:58:32   File "/home/iojs/build/workspace/node-test-commit-v8-linux/nodes/centos7-ppcle/v8test/v8test/configure.py", line 14, in <module>
07:58:32     import bz2
07:58:32   File "/usr/local/lib/python3.7/bz2.py", line 19, in <module>
07:58:32     from _bz2 import BZ2Compressor, BZ2Decompressor
07:58:32 ModuleNotFoundError: No module named '_bz2'

https://stackoverflow.com/questions/12806122/missing-python-bz2-module

This one related: nodejs/node-gyp#1947

# ./configure --debug -C

Node configure: Found Python 3.7.5...
INFO: Using floating patch "tools/icu/patches/64/source/common/putil.cpp" from "tools/icu"
INFO: Using floating patch "tools/icu/patches/64/source/i18n/dtptngen.cpp" from "tools/icu"
Traceback (most recent call last):
  File "./configure", line 25, in <module>
    import configure
  File "/root/node/configure.py", line 1755, in <module>
    run_gyp(gyp_args)
  File "tools/gyp_node.py", line 54, in run_gyp
    rc = gyp.main(args)
  File "tools/gyp/pylib/gyp/__init__.py", line 547, in main
    return gyp_main(args)
  File "tools/gyp/pylib/gyp/__init__.py", line 532, in gyp_main
    generator.GenerateOutput(flat_list, targets, data, params)
  File "tools/gyp/pylib/gyp/generator/compile_commands_json.py", line 95, in GenerateOutput
    for qualified_target, target in target_dicts.iteritems():
AttributeError: 'dict' object has no attribute 'iteritems'

@cclauss

@cclauss I haven't heard of anyone having troubles with Python 3, have you, or anyone else?

Are there next steps we can take now?

I've been building node with Python 3 for a while however the following workaround is needed for Python >= 3.8. It's a simple "hack" but it enables node to build on Python 3.8.

It seems to be limited to Python < 3.5(?). Not sure if this is true for Master or not.

sed -i 's/(3, 5)/(3, 8)/' configure

@dylanaraps

@dylanaraps with what node.js version do you do that?

master:

node/configure

Line 24 in 20fd123

acceptable_pythons = ((3, 8), (3, 7), (3, 6), (3, 5), (2, 7))

Apologies, I see that this is fixed in the latest version. Disregard my prior comment. That being said, Python 3 support works flawlessly here. 👍

@cclauss

Talked to @cclauss, we came up with a plan. We'll open new issues for this, but for the moment, here's the notes.

Reminder:

https://python3statement.org/ still says Jan1 was EOL for Python2, though @cclauss says it was recently extended by 4 months.
Node.js does not support a platform version if a vendor has expired support for it.
LTS schedule: https://github.com/nodejs/Release#release-schedule

1. Update npm to node-gyp@6.x on the next npm major, npm@7

This needs coordination with the npm team, @nodejs/npm

node-gyp@6 "prefers" py3 over py2 (node-gyp@5, what npm@6 uses, uses py3 only if py2 is not found). Changing the default is considered semver-major, which might be excessively paranoid, but nobody wants to break the ecosystem.

ATM, note that node 10.x and greater all have npm@6, which means that right now py3-only systems should build node.js addons with py3 on all currently supported node.js release lines.

2. node.js 12.x must support being built with py3

node@12.x will outlive py2 EOL by a long time, it must get non-experimental support for building with py3

We will identify the commits/PRs that need backporting.

3. node.js 10.x should support being built with py3

nodejs@10 outlives py2 EOL, so should be buildable with py3. Whether we can achieve that will take some analysis, maybe its easy, maybe its drifted too far from 12.x. If its way too hard/destabilizing, we might need to make an exception for it to our "unsupported platform" policy.

Note that lack of support for py3 means that https://github.com/felixrieseberg/windows-build-tools cannot yet install py3 instead of py2 - if it did that, users wouldn't be able to build node.js 10.x on Windows with using it. I mention, because its one example of the kind of downstream pain caused by not having py3 support. It might become a problem on non-Windows platforms, too, as py2 becomes less commonly packaged, or perhaps not packaged at all.

/cc @nodejs/tsc @nodejs/python @nodejs/node-gyp

#31562 after this patch.

python3 configure.py --debug --ninja failed with:

Traceback (most recent call last):
  File "configure.py", line 1770, in <module>
    run_gyp(gyp_args)
  File "tools/gyp_node.py", line 54, in run_gyp
    rc = gyp.main(args)
  File "tools/gyp/pylib/gyp/__init__.py", line 547, in main
    return gyp_main(args)
  File "tools/gyp/pylib/gyp/__init__.py", line 532, in gyp_main
    generator.GenerateOutput(flat_list, targets, data, params)
  File "tools/gyp/pylib/gyp/generator/ninja.py", line 2426, in GenerateOutput
    config_name)
  File "tools/gyp/pylib/gyp/generator/ninja.py", line 2317, in GenerateOutputForConfig
    hash_for_rules = hashlib.md5(qualified_target_for_hash).hexdigest()
TypeError: Unicode-objects must be encoded before hashing

On master this code is at line 2385 and the proper encoding is done on the line above.

Please see: https://github.com/nodejs/node/search?q=qualified_target_for_hash

@targos

Thanks to @targos for #30563, many compatibility issue should be resolved (like ninja issue).

Closing — Thanks to everyone who helped to make Node.js on Python 3 happen!

build: remove support for Python 2

Makefile: Upgrade to Python 3

Navigation Menu

Repository navigation

build: ongoing list of actions for Python 3 compatibility #25789

Upstream V8 issues:

Miscellaneous non-blocking cleanups

Please use the "python" label!

30 remaining items

1. Update npm to node-gyp@6.x on the next npm major, npm@7

2. node.js 12.x must support being built with py3

3. node.js 10.x should support being built with py3

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

build: ongoing list of actions for Python 3 compatibility #25789

Description

Upstream V8 issues:

Miscellaneous non-blocking cleanups

Please use the "python" label!

Activity

thefourtheye commented on Jan 30, 2019

targos commented on Jan 30, 2019

thefourtheye commented on Jan 30, 2019

gengjiawen commented on Jan 30, 2019

gengjiawen commented on Jan 30, 2019

cclauss commented on Jan 30, 2019

gengjiawen commented on Mar 4, 2019

30 remaining items

targos commented on Oct 28, 2019

cclauss commented on Oct 28, 2019

gengjiawen commented on Oct 29, 2019

sam-github commented on Jan 6, 2020

dylanaraps commented on Jan 6, 2020

sam-github commented on Jan 6, 2020

dylanaraps commented on Jan 6, 2020

sam-github commented on Jan 6, 2020

1. Update npm to node-gyp@6.x on the next npm major, npm@7

2. node.js 12.x must support being built with py3

3. node.js 10.x should support being built with py3

gengjiawen commented on Jan 29, 2020

cclauss commented on Jan 29, 2020

gengjiawen commented on Feb 29, 2020

cclauss commented on Feb 29, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions