bpo-38234: Add test_init_setpath_config() to test_embed #16402

vstinner · 2019-09-25T22:00:18Z

Add test_init_setpath_config() to test_embed: test Py_SetPath()
with PyConfig.
test_init_setpath() and test_init_setpythonhome() no longer call
Py_SetProgramName(), but use the default program name.
_PyPathConfig: only define isolated, site_import
and base_executable on Windows.
If executable is set explicitly in the configuration, ignore
calculated base_executable: _PyConfig_InitPathConfig() will copy
executable to base_executable.

https://bugs.python.org/issue38234

Python/pathconfig.c

zooba · 2019-09-25T22:39:27Z

It's nice that you separate out all the ._pth related stuff under a build flag for Windows, but there have been requests to add support on other platforms as well (I forget the bpo numbers now, or maybe it was in person at PyCon). So maybe make sure you don't make it too inaccessible or we'll have more work to do if/when we make that happen.

vstinner · 2019-09-25T22:56:33Z

It's nice that you separate out all the ._pth related stuff under a build flag for Windows, but there have been requests to add support on other platforms as well (I forget the bpo numbers now, or maybe it was in person at PyCon). So maybe make sure you don't make it too inaccessible or we'll have more work to do if/when we make that happen.

_PyPathConfig is an internal structure, so it's just an internal cleanup. It will be easy to enable the code on other platforms.

Right now, I'm trying to document the current implementation and write tests for it. But I would be interested to support also base_executable and __PYVENV_LAUNCHER__ environment variable in Modules/getpath.c.

* Add test_init_setpath_config() to test_embed: test Py_SetPath() with PyConfig. * test_init_setpath() and test_init_setpythonhome() no longer call Py_SetProgramName(), but use the default program name. * _PyPathConfig: only define isolated, site_import and base_executable on Windows. * If executable is set explicitly in the configuration, ignore calculated base_executable: _PyConfig_InitPathConfig() copies executable to base_executable. * Complete path config documentation.

bedevere-bot · 2019-09-26T00:47:26Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 FreeBSD 10-STABLE Non-Debug 3.x has failed when building commit 8bf39b6.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/167/builds/1619) and take a look at the build logs.
Check if the failure is related to this commit (8bf39b6) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/167/builds/1619

Failed tests:

test_embed

Failed subtests:

test_init_setpath_config - test.test_embed.InitConfigTests

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

396 tests OK.

10 slowest tests:

test_multiprocessing_spawn: 5 min 58 sec
test_tokenize: 5 min 45 sec
test_concurrent_futures: 4 min 53 sec
test_tools: 4 min 5 sec
test_lib2to3: 3 min 32 sec
test_multiprocessing_forkserver: 3 min 29 sec
test_asyncio: 3 min 17 sec
test_subprocess: 2 min 18 sec
test_unicodedata: 2 min 14 sec
test_multiprocessing_fork: 2 min 6 sec

1 test failed:
test_embed

22 tests skipped:
test_dbm_gnu test_devpoll test_epoll test_gdb test_idle test_ioctl
test_msilib test_ossaudiodev test_smtpnet test_spwd test_ssl
test_startfile test_tcl test_tix test_tk test_ttk_guionly
test_ttk_textonly test_turtle test_winconsoleio test_winreg
test_winsound test_zipfile64

1 re-run test:
test_embed

Total duration: 20 min 56 sec

Click to see traceback logs

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10.nondebug/build/Lib/test/test_embed.py", line 1059, in test_init_setpath_config
    self.check_all_configs("test_init_setpath_config", config,
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10.nondebug/build/Lib/test/test_embed.py", line 684, in check_all_configs
    self.check_config(configs, expected_config)
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10.nondebug/build/Lib/test/test_embed.py", line 613, in check_config
    self.assertEqual(config, expected)
AssertionError: {'_co[286 chars]g': 'ascii', 'filesystem_errors': 'surrogatees[1087 chars]': 1} != {'_co[286 chars]g': 'utf-8', 'filesystem_errors': 'surrogatees[1087 chars]': 1}
  {'_config_init': 2,
   '_init_main': 1,
   '_install_importlib': 1,
   'argv': [''],
   'base_exec_prefix': '',
   'base_executable': 'conf_executable',
   'base_prefix': '',
   'buffered_stdio': 1,
   'bytes_warning': 0,
   'check_hash_pycs_mode': 'default',
   'configure_c_stdio': 1,
   'dev_mode': 0,
   'dump_refs': 0,
   'exec_prefix': '',
   'executable': 'conf_executable',
   'faulthandler': 0,
-  'filesystem_encoding': 'ascii',
?                          ^^^^^

bedevere-bot · 2019-09-26T00:54:03Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot x86-64 High Sierra 3.x has failed when building commit 8bf39b6.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/145/builds/2486) and take a look at the build logs.
Check if the failure is related to this commit (8bf39b6) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/145/builds/2486

Failed tests:

test_embed

Failed subtests:

test_init_setpath - test.test_embed.InitConfigTests
test_init_setpythonhome - test.test_embed.InitConfigTests

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

402 tests OK.

10 slowest tests:

test_multiprocessing_spawn: 4 min 12 sec
test_concurrent_futures: 3 min 41 sec
test_tokenize: 3 min 25 sec
test_tools: 3 min 23 sec
test_multiprocessing_forkserver: 2 min 33 sec
test_lib2to3: 2 min 27 sec
test_unicodedata: 2 min 542 ms
test_asyncio: 1 min 54 sec
test_io: 1 min 8 sec
test_pickle: 1 min 8 sec

1 test failed:
test_embed

16 tests skipped:
test_devpoll test_epoll test_gdb test_ioctl test_msilib
test_multiprocessing_fork test_ossaudiodev test_spwd
test_startfile test_tix test_tk test_ttk_guionly test_winconsoleio
test_winreg test_winsound test_zipfile64

1 re-run test:
test_embed

Total duration: 28 min 21 sec

Click to see traceback logs

Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.billenstein-sierra/build/Lib/test/test_embed.py", line 1035, in test_init_setpath
    self.check_all_configs("test_init_setpath", config,
  File "/Users/buildbot/buildarea/3.x.billenstein-sierra/build/Lib/test/test_embed.py", line 684, in check_all_configs
    self.check_config(configs, expected_config)
  File "/Users/buildbot/buildarea/3.x.billenstein-sierra/build/Lib/test/test_embed.py", line 613, in check_config
    self.assertEqual(config, expected)
AssertionError: {'_co[418 chars]ne, 'module_search_paths': ['/Users/buildbot/b[913 chars]': 1} != {'_co[418 chars]ne, 'executable': '/Users/buildbot/buildbot/bi[837 chars]': 1}
  {'_config_init': 1,
   '_init_main': 1,
   '_install_importlib': 1,
   'argv': [''],
   'base_exec_prefix': '',
-  'base_executable': '/Users/buildbot/buildarea/3.x.billenstein-sierra/build/Programs/_testembed',
+  'base_executable': '/Users/buildbot/buildbot/bin/python3',
   'base_prefix': '',
   'buffered_stdio': 1,
   'bytes_warning': 0,
   'check_hash_pycs_mode': 'default',
   'configure_c_stdio': 0,
   'dev_mode': 0,
   'dump_refs': 0,
   'exec_prefix': '',
-  'executable': '/Users/buildbot/buildarea/3.x.billenstein-sierra/build/Programs/_testembed',
+  'executable': '/Users/buildbot/buildbot/bin/python3',
   'faulthandler': 0,
   'hash_seed': 0,
   'home': None,
   'import_time': 0,
   'inspect': 0,
   'install_signal_handlers': 1,
   'interactive': 0,
   'isolated': 0,
   'malloc_stats': 0,
   'module_search_paths': ['/Users/buildbot/buildarea/3.x.billenstein-sierra/build/target/lib/python39.zip',
                           '/Users/buildbot/buildarea/3.x.billenstein-sierra/build/Lib',
                           '/Users/buildbot/buildarea/3.x.billenstein-sierra/build/build/lib.macosx-10.13-x86_64-3.9-pydebug'],
   'optimization_level': 0,
   'parse_argv': 0,
   'parser_debug': 0,
   'pathconfig_warnings': 1,
   'prefix': '',
   'program_name': 'python3',
   'pycache_prefix': None,
   'pythonpath_env': None,
   'quiet': 0,
   'run_command': None,
   'run_filename': None,
   'run_module': None,
   'show_alloc_count': 0,
   'show_ref_count': 0,
   'site_import': 1,
   'skip_source_first_line': 0,
   'tracemalloc': 0,
   'use_environment': 1,
   'use_hash_seed': 0,
   'user_site_directory': 1,
   'verbose': 0,
   'warnoptions': [],
   'write_bytecode': 1,
   'xoptions': []}


Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.billenstein-sierra/build/Lib/test/test_embed.py", line 1142, in test_init_setpythonhome
    self.check_all_configs("test_init_setpythonhome", config,
  File "/Users/buildbot/buildarea/3.x.billenstein-sierra/build/Lib/test/test_embed.py", line 684, in check_all_configs
    self.check_config(configs, expected_config)
  File "/Users/buildbot/buildarea/3.x.billenstein-sierra/build/Lib/test/test_embed.py", line 613, in check_config
    self.assertEqual(config, expected)
AssertionError: {'_co[702 chars]d', 'module_search_paths': ['/Users/buildbot/b[1360 chars]': 1} != {'_co[702 chars]d', 'executable': '/Users/buildbot/buildbot/bi[1284 chars]': 1}
  {'_config_init': 1,
   '_init_main': 1,
   '_install_importlib': 1,
   'argv': [''],
   'base_exec_prefix': '/Users/buildbot/buildarea/3.x.billenstein-sierra/build',
-  'base_executable': '/Users/buildbot/buildarea/3.x.billenstein-sierra/build/Programs/_testembed',
+  'base_executable': '/Users/buildbot/buildbot/bin/python3',
   'base_prefix': '/Users/buildbot/buildarea/3.x.billenstein-sierra/build',
   'buffered_stdio': 1,
   'bytes_warning': 0,
   'check_hash_pycs_mode': 'default',
   'configure_c_stdio': 0,
   'dev_mode': 0,
   'dump_refs': 0,
   'exec_prefix': '/Users/buildbot/buildarea/3.x.billenstein-sierra/build',
-  'executable': '/Users/buildbot/buildarea/3.x.billenstein-sierra/build/Programs/_testembed',
+  'executable': '/Users/buildbot/buildbot/bin/python3',
   'faulthandler': 0,
   'hash_seed': 0,
   'home': '/Users/buildbot/buildarea/3.x.billenstein-sierra/build',
   'import_time': 0,
   'inspect': 0,
   'install_signal_handlers': 1,
   'interactive': 0,
   'isolated': 0,
   'malloc_stats': 0,
   'module_search_paths': ['/Users/buildbot/buildarea/3.x.billenstein-sierra/build/target/lib/python39.zip',
                           '/Users/buildbot/buildarea/3.x.billenstein-sierra/build/Lib',
                           '/Users/buildbot/buildarea/3.x.billenstein-sierra/build/build/lib.macosx-10.13-x86_64-3.9-pydebug',
                           '/Users/buildbot/buildarea/3.x.billenstein-sierra/build/lib/python39.zip',
                           '/Users/buildbot/buildarea/3.x.billenstein-sierra/build/lib/python3.9',
                           '/Users/buildbot/buildarea/3.x.billenstein-sierra/build/lib/python3.9/lib-dynload'],
   'optimization_level': 0,
   'parse_argv': 0,
   'parser_debug': 0,
   'pathconfig_warnings': 1,
   'prefix': '/Users/buildbot/buildarea/3.x.billenstein-sierra/build',
   'program_name': 'python3',
   'pycache_prefix': None,
   'pythonpath_env': '/Users/buildbot/buildarea/3.x.billenstein-sierra/build/target/lib/python39.zip:/Users/buildbot/buildarea/3.x.billenstein-sierra/build/Lib:/Users/buildbot/buildarea/3.x.billenstein-sierra/build/build/lib.macosx-10.13-x86_64-3.9-pydebug',
   'quiet': 0,
   'run_command': None,
   'run_filename': None,
   'run_module': None,
   'show_alloc_count': 0,
   'show_ref_count': 0,
   'site_import': 1,
   'skip_source_first_line': 0,
   'tracemalloc': 0,
   'use_environment': 1,
   'use_hash_seed': 0,
   'user_site_directory': 1,
   'verbose': 0,
   'warnoptions': [],
   'write_bytecode': 1,
   'xoptions': []}

) * bpo-38234: Py_SetPath() uses the program full path (GH-16357) Py_SetPath() now sets sys.executable to the program full path (Py_GetProgramFullPath()), rather than to the program name (Py_GetProgramName()). Fix also memory leaks in pathconfig_set_from_config(). (cherry picked from commit 1ce152a) * bpo-38234: Add tests for Python init path config (GH-16358) (cherry picked from commit bb6bf7d) * bpo-38234: test_embed: test pyvenv.cfg and pybuilddir.txt (GH-16366) Add test_init_pybuilddir() and test_init_pyvenv_cfg() to test_embed to test pyvenv.cfg and pybuilddir.txt configuration files. Fix sysconfig._generate_posix_vars(): pybuilddir.txt uses UTF-8 encoding, not ASCII. (cherry picked from commit 52ad33a) * bpo-38234: Cleanup getpath.c (GH-16367) * search_for_prefix() directly calls reduce() if found is greater than 0. * Add calculate_pybuilddir() subfunction. * search_for_prefix(): add path string buffer for readability. * Fix some error handling code paths: release resources on error. * calculate_read_pyenv(): rename tmpbuffer to filename. * test.pythoninfo now also logs windows.dll_path (cherry picked from commit 221fd84) * bpo-38234: Fix test_embed pathconfig tests (GH-16390) bpo-38234: On macOS and FreeBSD, the temporary directory can be symbolic link. For example, /tmp can be a symbolic link to /var/tmp. Call realpath() to resolve all symbolic links. (cherry picked from commit 00508a7) * bpo-38234: Add test_init_setpath_config() to test_embed (GH-16402) * Add test_embed.test_init_setpath_config(): test Py_SetPath() with PyConfig. * test_init_setpath() and test_init_setpythonhome() no longer call Py_SetProgramName(), but use the default program name. * _PyPathConfig: isolated, site_import and base_executable fields are now only available on Windows. * If executable is set explicitly in the configuration, ignore calculated base_executable: _PyConfig_InitPathConfig() copies executable to base_executable. * Complete path config documentation. (cherry picked from commit 8bf39b6) * bpo-38234: Complete init config documentation (GH-16404) (cherry picked from commit 88feaec) * bpo-38234: Fix test_embed.test_init_setpath_config() on FreeBSD (GH-16406) Explicitly preinitializes with a Python preconfiguration to avoid Py_SetPath() implicit preinitialization with a compat preconfiguration. Fix also test_init_setpath() and test_init_setpythonhome() on macOS: use self.test_exe as the executable (and base_executable), rather than shutil.which('python3'). (cherry picked from commit 49d99f0) * bpo-38234: Py_Initialize() sets global path configuration (GH-16421) * Py_InitializeFromConfig() now writes PyConfig path configuration to the global path configuration (_Py_path_config). * Add test_embed.test_get_pathconfig(). * Fix typo in _PyWideStringList_Join(). (cherry picked from commit 12f2f17)

* Add test_embed.test_init_setpath_config(): test Py_SetPath() with PyConfig. * test_init_setpath() and test_init_setpythonhome() no longer call Py_SetProgramName(), but use the default program name. * _PyPathConfig: isolated, site_import and base_executable fields are now only available on Windows. * If executable is set explicitly in the configuration, ignore calculated base_executable: _PyConfig_InitPathConfig() copies executable to base_executable. * Complete path config documentation.

vstinner added the skip news label Sep 25, 2019

the-knights-who-say-ni added the CLA signed label Sep 25, 2019

bedevere-bot added the awaiting core review label Sep 25, 2019

zooba reviewed Sep 25, 2019

View reviewed changes

Python/pathconfig.c Outdated Show resolved Hide resolved

vstinner requested a review from a team as a code owner September 25, 2019 22:48

vstinner merged commit 8bf39b6 into python:master Sep 26, 2019

vstinner deleted the pathconfig_tests branch September 26, 2019 00:22

bedevere-bot removed the awaiting core review label Sep 26, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

bpo-38234: Add test_init_setpath_config() to test_embed #16402

bpo-38234: Add test_init_setpath_config() to test_embed #16402

Uh oh!

vstinner commented Sep 25, 2019 •

edited by bedevere-bot

Loading

Uh oh!

Uh oh!

zooba commented Sep 25, 2019

Uh oh!

vstinner commented Sep 25, 2019

Uh oh!

bedevere-bot commented Sep 26, 2019

Uh oh!

bedevere-bot commented Sep 26, 2019

Uh oh!

Uh oh!

Uh oh!

bpo-38234: Add test_init_setpath_config() to test_embed #16402

bpo-38234: Add test_init_setpath_config() to test_embed #16402

Uh oh!

Conversation

vstinner commented Sep 25, 2019 • edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

zooba commented Sep 25, 2019

Uh oh!

vstinner commented Sep 25, 2019

Uh oh!

bedevere-bot commented Sep 26, 2019

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

bedevere-bot commented Sep 26, 2019

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

Uh oh!

vstinner commented Sep 25, 2019 •

edited by bedevere-bot

Loading