Fatal Python error: _PySys_BeginInit: can't initialize sys module

[Tasy]

BPO	35713
Nosy	@vstinner, @tirkarthi
PRs	bpo-35713: Rework Python initialization #11647 bpo-35713: Rework Python initialization #11647 bpo-35713: Rework Python initialization #11647 bpo-35713: Split _Py_InitializeCore into subfunctions #11650 bpo-35713: Split _Py_InitializeCore into subfunctions #11650 bpo-35713: Split _Py_InitializeCore into subfunctions #11650 bpo-35713: Reorganize sys module initialization #11658 bpo-35713: Reorganize sys module initialization #11658 bpo-35713: Reorganize sys module initialization #11658

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-01-18.17:52:46.947>
created_at = <Date 2019-01-10.22:35:31.620>
labels = ['build', '3.7']
title = "Fatal Python error: _PySys_BeginInit: can't initialize sys module"
updated_at = <Date 2019-01-23.14:06:29.967>
user = 'https://bugs.python.org/Tasy'

bugs.python.org fields:

activity = <Date 2019-01-23.14:06:29.967>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-01-18.17:52:46.947>
closer = 'Tasy'
components = ['Build']
creation = <Date 2019-01-10.22:35:31.620>
creator = 'Tasy'
dependencies = []
files = []
hgrepos = []
issue_num = 35713
keywords = []
message_count = 14.0
messages = ['333423', '333425', '333427', '333428', '333429', '333733', '333853', '333984', '333992', '333999', '334219', '334233', '334258', '334259']
nosy_count = 3.0
nosy_names = ['vstinner', 'xtreak', 'Tasy']
pr_nums = ['11647', '11647', '11647', '11650', '11650', '11650', '11658', '11658', '11658']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue35713'
versions = ['Python 3.7']

[Tasy]

.
.
.
./python -E -S -m sysconfig --generate-posix-vars ;\
if test $? -ne 0 ; then \
echo "generate-posix-vars failed" ; \
rm -f ./pybuilddir.txt ; \
exit 1 ; \
fi
Fatal Python error: _PySys_BeginInit: can't initialize sys module

Current thread 0x00002b4e5f9bf400 (most recent call first):
Aborted (core dumped)
generate-posix-vars failed
make[1]: *** [pybuilddir.txt] Error 1
make[1]: Leaving directory `/usr/local/mysoftware/Python-3.7.2/build'
make: *** [profile-opt] Error 2

[vstinner]

.
.
.

Hello. Can you elaborate this part?

[Tasy]

Sorry for the confusion. The three dots were for many lines of successful compilation output from make. The rest is the final few lines where the compilation fails.

[vstinner]

Because it's late, I will say it shortly: if you don't elaborate how you get this error, I will simply close the issue. You have to describe what you are trying to do, your OS, etc.

(It works for me!)

[tirkarthi]

Please attach the full build log too along with the command executed.

[Tasy]

Configuration Options:

../configure --prefix=$HOME --enable-shared --enable-optimizations --with-system-expat --with-system-ffi --with-ensurepip=yes

Make throws the following warning:

*** WARNING: renaming "_curses_panel" since importing it failed: No module named '_curses'

Python build finished successfully!
The necessary bits to build these optional modules were not found:
_ssl _uuid
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc atexit pwd
time

Failed to build these modules:
_curses

Following modules built successfully but were removed because they could not be imported:
_curses_panel

Could not build the ssl module!
Python requires an OpenSSL 1.0.2 or 1.1 compatible libssl with X509_VERIFY_PARAM_set1_host().
LibreSSL 2.6.4 and earlier do not provide the necessary APIs, libressl/portable#381

There ther is a following error...

0:06:18 load avg: 0.55 [171/416] test_hashlib
*** Error in `./python': corrupted size vs. prev_size: 0x000000000276b7a0 ***
Fatal Python error: Aborted

Current thread 0x00002ba4468c7bc0 (most recent call first):
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/test_hashlib.py", line 904 in _test_pbkdf2_hmac
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/test_hashlib.py", line 935 in test_pbkdf2_hmac_c
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/unittest/case.py", line 615 in run
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/unittest/case.py", line 663 in call
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/unittest/suite.py", line 122 in run
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/unittest/suite.py", line 84 in call
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/unittest/suite.py", line 122 in run
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/unittest/suite.py", line 84 in call
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/unittest/suite.py", line 122 in run
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/unittest/suite.py", line 84 in call
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/support/testresult.py", line 162 in run
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/support/init.py", line 1895 in _run_suite
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/support/init.py", line 1991 in run_unittest
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/libregrtest/runtest.py", line 178 in test_runner
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/libregrtest/runtest.py", line 182 in runtest_inner
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/libregrtest/runtest.py", line 137 in runtest
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/libregrtest/main.py", line 407 in run_tests_sequential
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/libregrtest/main.py", line 514 in run_tests
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/libregrtest/main.py", line 615 in _main
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/libregrtest/main.py", line 582 in main
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/libregrtest/main.py", line 636 in main
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/regrtest.py", line 46 in _main
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/regrtest.py", line 50 in
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/runpy.py", line 85 in _run_code
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/runpy.py", line 193 in _run_module_as_main
Aborted (core dumped)
make[1]: Leaving directory /usr/local/data/mySoftware/Python-3.7.2/build' make build_all_merge_profile make[1]: Entering directory /usr/local/data/mySoftware/Python-3.7.2/build'
true
make[1]: Leaving directory `/usr/local/data/mySoftware/Python-3.7.2/build'

Remove profile generation binary since we are done with it.

make clean
make[1]: Entering directory /usr/local/data/mySoftware/Python-3.7.2/build' find .. -depth -name '__pycache__' -exec rm -rf {} ';' find .. -name '*.py[co]' -exec rm -f {} ';' find . -name '*.[oa]' -exec rm -f {} ';' find . -name '*.s[ol]' -exec rm -f {} ';' find . -name '*.so.[0-9]*.[0-9]*' -exec rm -f {} ';' find build -name 'fficonfig.h' -exec rm -f {} ';' || true find build -name '*.py' -exec rm -f {} ';' || true find build -name '*.py[co]' -exec rm -f {} ';' || true rm -f pybuilddir.txt rm -f [Lib/lib2to3/](https://github.com/python/cpython/blob/main/Lib/lib2to3)*Grammar*.pickle rm -f [Programs/_testembed](https://github.com/python/cpython/blob/main/Programs/_testembed) [Programs/_freeze_importlib](https://github.com/python/cpython/blob/main/Programs/_freeze_importlib) find build -type f -a ! -name '*.gc??' -exec rm -f {} ';' rm -f [Include/pydtrace_probes.h](https://github.com/python/cpython/blob/main/Include/pydtrace_probes.h) rm -f profile-gen-stamp make[1]: Leaving directory /usr/local/data/mySoftware/Python-3.7.2/build'

This is an expensive target to build and it does not have proper

makefile dependency information. So, we create a "stamp" file

to record its completion and avoid re-running it.

touch profile-run-stamp
Rebuilding with profile guided optimizations:
rm -f profile-clean-stamp
make build_all CFLAGS_NODIST=" -fprofile-use -fprofile-correction" LDFLAGS_NODIST=""
make[1]: Entering directory `/usr/local/data/mySoftware/Python-3.7.2/build'

And then finally :

Modules/_localemodule.o Modules/_iomodule.o Modules/iobase.o Modules/fileio.o Modules/bytesio.o Modules/bufferedio.o Modules/textio.o Modules/stringio.o Modules/zipimport.o Modules/faulthandler.o Modules/_tracemalloc.o Modules/hashtable.o Modules/symtablemodule.o Modules/xxsubtype.o Python/frozen.o
gcc -pthread -shared -Wl,--no-as-needed -o libpython3.so -Wl,-hlibpython3.so libpython3.7m.so
gcc -pthread -Xlinker -export-dynamic -o python Programs/python.o -L. -lpython3.7m -lpthread -ldl -lutil -lm
gcc -pthread -Xlinker -export-dynamic -o Programs/_testembed Programs/_testembed.o -L. -lpython3.7m -lpthread -ldl -lutil -lm
LD_LIBRARY_PATH=/usr/local/data/mySoftware/Python-3.7.2/build:$HOME/lib:/usr/local/cuda/lib64:$HOME/lib/cuda/lib64:$HOME/lib/cuda/extras/CUPTI/lib64:/usr/local/data/lib:/usr/local/lib/:/usr/lib/ ./python -E -S -m sysconfig --generate-posix-vars ;\
if test $? -ne 0 ; then \
echo "generate-posix-vars failed" ; \
rm -f ./pybuilddir.txt ; \
exit 1 ; \
fi
Fatal Python error: _PySys_BeginInit: can't initialize sys module

Current thread 0x00002adc936069c0 (most recent call first):
Aborted (core dumped)
generate-posix-vars failed
make[1]: *** [pybuilddir.txt] Error 1
make[1]: Leaving directory `/usr/local/data/mySoftware/Python-3.7.2/build'
make: *** [profile-opt] Error 2

[vstinner]

What is your OS (name and version)?

What is your compiler (name and version)?

Fatal Python error: _PySys_BeginInit: can't initialize sys module

I have no idea why you get this error. You should try to run this function in a debugger like gdb and run the code step by step to see what happens.

"""

0:06:18 load avg: 0.55 [171/416] test_hashlib
*** Error in `./python': corrupted size vs. prev_size: 0x000000000276b7a0 ***
Fatal Python error: Aborted

Current thread 0x00002ba4468c7bc0 (most recent call first):
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/test_hashlib.py", line 904 in _test_pbkdf2_hmac
"""

That's maybe unrelated, but _test_pbkdf2_hmac() is not supposed to crash.

[Tasy]

Compiler:
$ gcc --version
gcc (Ubuntu 4.8.4-2ubuntu1~14.04.4) 4.8.4

OS:
$ uname -a
Linux machine 4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec 10 13:20:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Home directory is of type nfs in case that is relevant.
I'll try to do the gdb thing over the weekend.

[vstinner]

Hum. Are you aware that PGO with GCC is broken on such old and unsupported Ubuntu version?

My old note about that:

"PGO is broken on Ubuntu 14.04 LTS with GCC 4.8.4-2ubuntu1~14.04: Modules/socketmodule.c:7743:1: internal compiler error: in edge_badness, at ipa-inline.c:895"

https://pyperformance.readthedocs.io/usage.html#compile

See also https://bugs.python.org/issue31963

I suggest you to not use PGO compilation on old Ubuntu. Maybe upgrade to latest Ubuntu LTS?

[Tasy]

compiling without optimizations worked.

[vstinner]

New changeset bf4ac2d by Victor Stinner in branch 'master':
bpo-35713: Rework Python initialization (GH-11647)
bf4ac2d