Use -Wl,--dynamic-list=x.list, not -Xlinker -export-dynamic

[jankratochvil]

BPO	10112
Nosy	@loewis, @pitrou, @vstinner, @merwok, @davidmalcolm, @meadori
Files	hidden4.patch: Fixed patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2010-10-15.10:16:24.300>
labels = ['build', 'performance']
title = 'Use -Wl,--dynamic-list=x.list, not -Xlinker -export-dynamic'
updated_at = <Date 2019-01-23.15:47:02.674>
user = 'https://bugs.python.org/jankratochvil'

bugs.python.org fields:

activity = <Date 2019-01-23.15:47:02.674>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Build']
creation = <Date 2010-10-15.10:16:24.300>
creator = 'jankratochvil'
dependencies = []
files = ['24575']
hgrepos = []
issue_num = 10112
keywords = ['patch']
message_count = 7.0
messages = ['118756', '151146', '153394', '153459', '153764', '153810', '237136']
nosy_count = 9.0
nosy_names = ['loewis', 'pitrou', 'vstinner', 'eric.araujo', 'Arfrever', 'dmalcolm', 'meador.inge', 'jankratochvil', 'tromey']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue10112'
versions = ['Python 3.3']

[jankratochvil]

FSF GDB (and future Fedora GDBs) became 250KB smaller than before. Python recently disabled this GDB build optimization so GDB is 250KB larger again.

-rwxr-xr-x 1 4524488 gdb-7.1-19.fc13.x86_64/usr/bin/gdb
-rwxr-xr-x 1 4266728 gdb-7.1-19dynamiclist.fc13.x86_64/usr/bin/gdb
-rw-r--r-- 1 2364656 gdb-7.1-19.fc13.x86_64.rpm
-rw-r--r-- 1 2274300 gdb-7.1-19dynamiclist.fc13.x86_64.rpm

Some Python binaries/libraries could probably also benefit from smaller pageable code image.

GDB regressed due to /usr/lib*/python*/config/Makefile contains:
LINKFORSHARED=-Xlinker -export-dynamic

and GDB thus has to use it even for its own link, effectively disabling its:
-Wl,--dynamic-list=./proc-service.list

AFAIK Python already contains the required exports list but it is used only on MS-Windows. It should be utilized even for GNU/Linux builds.

[jankratochvil]

Here is the implementation.

Python/getargs.c was modified for:
ImportError: /usr/lib64/python2.7/lib-dynload/fcntlmodule.so: undefined symbol: _PyArg_ParseTuple_SizeT
but I guess that patch part should be different. There is no need to do #ifdef HAVE_DECLSPEC_DLL as PyAPI_FUNC is properly defined even without it. So the declaration+export block should be unified there.

The Modules/ files were modified for:
*** WARNING: renaming "_ctypes_test" since importing it failed: dynamic module does not define init function (init_ctypes_test)
*** WARNING: renaming "_elementtree" since importing it failed: dynamic module does not define init function (init_elementtree)
*** WARNING: renaming "_hotshot" since importing it failed: dynamic module does not define init function (init_hotshot)
*** WARNING: renaming "_json" since importing it failed: dynamic module does not define init function (init_json)
*** WARNING: renaming "ossaudiodev" since importing it failed: dynamic module does not define init function (initossaudiodev)

without this patch:
-rw-r--r-- 1 jkratoch jkratoch 5775829 Jan 12 17:42 python-libs-2.7.2-18.fc17.x86_64.rpm
-r-xr-xr-x 1 jkratoch jkratoch 1738264 Jan 12 17:44 ./clean/python-libs-2.7.2-18.fc17.x86_64/usr/lib64/libpython2.7.so.1.0*
1226 libpython2.7.so.1.0 .dynsym exports
+
-rw-r--r-- 1 jkratoch jkratoch 1986161 Jan 12 17:48 gdb-7.4.50.20120103-8.fc17.x86_64.rpm
-rwxr-xr-x 1 jkratoch jkratoch 5018800 Jan 12 19:13 clean/gdb-7.4.50.20120103-8.fc17.x86_64/usr/bin/gdb*

with this patch:
-rw-r--r-- 1 jkratoch jkratoch 5762537 Jan 12 19:04 python-libs-2.7.2-18hidden3.fc17.x86_64.rpm
-r-xr-xr-x 1 jkratoch jkratoch 1720920 Jan 12 19:06 ./hidden3/python-libs-2.7.2-18hidden3.fc17.x86_64/usr/lib64/libpython2.7.so.1.0*
1046 libpython2.7.so.1.0 .dynsym exports
+
-rw-r--r-- 1 jkratoch jkratoch 1886781 Jan 12 19:11 gdb-7.4.50.20120103-8.fc17.x86_64.rpm
-rwxr-xr-x 1 jkratoch jkratoch 4732080 Jan 12 19:13 hidden3/gdb-7.4.50.20120103-8.fc17.x86_64/usr/bin/gdb*

[jankratochvil]

What more can be done to get it fixed? I do not mind Python but GDB linking is broken due to it. And while I can workaround it in GDB I am not used for workarounding one Free package in another Free package. Free software has the advantage problems can be fixed at the right place.

[meadori]

When I try to build with the attached patch on the 2.7 branch many (if not all) of the modules fail to build:

Failed to build these modules:
_bisect _bsddb _codecs_cn
_codecs_hk _codecs_iso2022 _codecs_jp
_codecs_kr _codecs_tw _collections
_csv _ctypes _ctypes_test
_curses _curses_panel _elementtree
_functools _hashlib _heapq
_hotshot _io _json
_locale _lsprof _multibytecodec
_multiprocessing _random _socket
_sqlite3 _ssl _struct
_testcapi _tkinter array
audioop binascii bz2
cmath cPickle crypt
cStringIO datetime dbm
fcntl future_builtins gdbm
grp itertools linuxaudiodev
math mmap nis
operator ossaudiodev parser
pyexpat readline resource
select spwd strop
syslog termios time
unicodedata zlib

I see several warnings in the build output:

*** WARNING: renaming "_codecs_iso2022" since importing it failed: build/lib.lin
ux-x86_64-2.7/_codecs_iso2022.so: undefined symbol: PyExc_ValueError
..
*** WARNING: renaming "linuxaudiodev" since importing it failed: build/lib.linux
-x86_64-2.7/linuxaudiodev.so: undefined symbol: Py_Py3kWarningFlag
...
*** WARNING: renaming "_tkinter" since importing it failed: build/lib.linux-x86_64-2.7/_tkinter.so: undefined symbol: _Py_ZeroStruct

[jankratochvil]

There was a bug for non-shared build of Python, thanks.

This patch is an improvement on its but it is not yet complete.
One should also replace "-Xlinker -export-dynamic" by "-Wl,--dynamic-list" so that linking application with libpython is no longer intrusive.

But I found out GDB does not need to be affected by this Bug. When I use -fvisibility=hidden for GDB .c files then "-Xlinker -export-dynamic" has no longer any negative effect (GDB was using only "-Wl,--dynamic-list" now which works without libpython but not with libpython). Therefore I no longer mind much about this Bug.

Still at least:
1771024 libpython2.7.so.1.0
->
1757776 libpython2.7.so.1.0

Even with clean tree the upstream build generates for me:
Python build finished, but the necessary bits to build these modules were not found:
_bsddb _ssl bsddb185
dl imageop sunaudiodev

(so sunaudiodev.c was changed without any testing of it)

[loewis]

The patch is fairly incomplete: it assumes that the compiler supports __attribute__((visibility())), but it really needs to check for that.

Also, there is no chance that this can go into Python 2.7; retargetting for 3.3.

[jankratochvil]

It even crashes applications due to pollution of dynamic symbols namespace by application symbols as seen in:
https://bugzilla.redhat.com/show_bug.cgi?id=1198158

[iritkatriel]

@jankratochvil Is this issue still relevant?

[jankratochvil]

I am no longer involved with GDB. But I also think it has been fixed for python2->3:

$ /usr/bin/python2-config --ldflags
-lpython2.7 -ldl -lm -Xlinker -export-dynamic
$ /usr/bin/python3-config --ldflags
 -L/usr/lib64  -lcrypt -ldl  -lm -lm