Compress::Raw::Zlib doesn't apply symbol versioning when building zlib-src

[monkburger]

There's a rare chance that when using the C:R:Z XS module under Linux (possibly others, like Solaris), there may be a symbol collision depending on your backend environment (perlcc, mod_perl, or other modules).

Zlib.so exports these functions, globally, without any symbol versions (via readelf -s blib/arch/auto/Compress/Raw/Zlib/Zlib.so | egrep flate)

57: 0000000000012ce0   198 FUNC    GLOBAL DEFAULT   14 inflateResetKeep
61: 0000000000015650    80 FUNC    GLOBAL DEFAULT   14 inflateCodesUsed
62: 00000000000152d0    75 FUNC    GLOBAL DEFAULT   14 inflateSyncPoint
64: 000000000000f310   228 FUNC    GLOBAL DEFAULT   14 deflateResetKeep
65: 0000000000012580    62 FUNC    GLOBAL DEFAULT   14 inflateBackEnd
66: 0000000000015530    70 FUNC    GLOBAL DEFAULT   14 inflateUndermine
67: 0000000000012ee0   192 FUNC    GLOBAL DEFAULT   14 inflateInit2_
69: 0000000000014ec0   212 FUNC    GLOBAL DEFAULT   14 inflateSetDictionary
70: 0000000000012e10   198 FUNC    GLOBAL DEFAULT   14 inflateReset2
77: 000000000000f590   210 FUNC    GLOBAL DEFAULT   14 deflatePrime
78: 000000000000f800  4985 FUNC    GLOBAL DEFAULT   14 deflate

On ELF based systems (Linux, etc), unversioned symbols preempt versioned ones, so you could run into issues such as what was described in this bug report with CryptX, as well as these upstream reports relating to symbol versions, as well as this

He's what LD_DEBUG=all shows on some test code:

foo.1358336: 1358336: symbol=inflate; lookup in file=/usr/lib/x86_64-linux-gnu/perl/5.30/auto/Compress/Raw/Zlib/Zlib.so [0]
foo.1358336: 1358336: symbol=inflate; lookup in file=/lib/x86_64-linux-gnu/libz.so.1 [0]

Which yields the following:

foo.1358336:   1358336:	binding file /usr/lib/x86_64-linux-gnu/perl/5.30/auto/Compress/Raw/Zlib/Zlib.so [0] to /lib/x86_64-linux-gnu/libz.so.1 [0]: normal symbol `inflate'

A small rant: ELF systems (Solaris, Linux) situation is ever worse because they allow runtime symbol interposition (i.e. libraries compete for who wins over symbol names at program startup). There's no telling what symbol you will get, as the flat namespace model shows.

Some ideas to avoid any issues:

• Use zlib.map from upstream when building zlib-src
• -Bsymbolic to the linker/compiler flags? <-- not tested, unsure of how it would work unless you linked libperl.so / libperl.a at link time
• Avoid building zlib-src and just depend on the system libz

[FGasper]

One way to do this would be to build libz.a, then link libz.a into the C::R::Zlib shared library. The linker would need --exclude-libs=ALL, too.

[pmqs]

Hey @monkburger

thanks for the feedback.

shout if I'm inferring too much here, but as things stand this is still a potential issue that hasn't been triggered anywhere that you know of?

Getting back to the issue, can I make sure I understand it properly?

This is only going to be an issue where Perl & C::R::Zlib are embedded in a larger application. Let's assume mod_perl for now. In the mod_perl setup they may be another component that wants to use zlib. That other component has been linked against the OS version of zlib (that has included versioned symbols for a while).

At load time the unversioned zlib symbols in C::R::Zlib will always be used in preference to the OS copy. That is a possible issue if the zlib source embedded in C::R::Zlib & the OS versions of zlib don't match.

Is that about it?

thanks
Paul

[monkburger]

Hi Paul.

That is correct.

I may have a demonstration of this behavior very soon - with a crash. I'll keep you posted.

[toddr]

libperl links against distro zlib on AlmaLinux 8 because libnsl needs the distro zlib . As best I can tell,

$>lddtree /usr/local/cpanel/3rdparty/perl/532/lib/perl5/532/x86_64-linux-64int/CORE/libperl.so 
libperl.so => /usr/local/cpanel/3rdparty/perl/532/lib/perl5/532/x86_64-linux-64int/CORE/libperl.so (interpreter => none)
    libpthread.so.0 => /lib64/libpthread.so.0
        ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2
    libnsl.so.2 => /lib64/libnsl.so.2
        libtirpc.so.3 => /lib64/libtirpc.so.3
            libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2
                libkrb5support.so.0 => /lib64/libkrb5support.so.0
                    libselinux.so.1 => /lib64/libselinux.so.1
                        libpcre2-8.so.0 => /lib64/libpcre2-8.so.0
                libkeyutils.so.1 => /lib64/libkeyutils.so.1
                libcrypto.so.1.1 => /lib64/libcrypto.so.1.1
                    libz.so.1 => /lib64/libz.so.1
                libresolv.so.2 => /lib64/libresolv.so.2
            libkrb5.so.3 => /lib64/libkrb5.so.3
            libk5crypto.so.3 => /lib64/libk5crypto.so.3
            libcom_err.so.2 => /lib64/libcom_err.so.2
    libdl.so.2 => /lib64/libdl.so.2
    libm.so.6 => /lib64/libm.so.6
    libcrypt.so.1 => /lib64/libcrypt.so.1
    libutil.so.1 => /lib64/libutil.so.1
    libc.so.6 => /lib64/libc.so.6

[toddr]

Right now I cannot build blead on AlmaLinux 8.

[FGasper]

As best I can tell, a reasonable fix would be, instead of slurping all of zlib’s .o files directly into Zlib.so, create libz.a from the .o files, then slurp libz.a into Zlib.so, with the --exclude-libs=ALL linker flag. That’ll prevent the symbol conflict because Zlib.so will no longer export zlib’s symbols.

You’ll need to exclude platforms where the linker doesn’t support --exclude-libs; AFAICT only GNU’s linker recognizes it.

[pmqs]

Right now I cannot build blead on AlmaLinux 8.

Is that because of this issue?

[FGasper]

echo 'int main() { return 0; }' | gcc '-Wl,--exclude-libs,ALL' -xc -o /dev/null -

^^ One-liner to determine if the linker supports the relevant option.

[toddr]

Right now I cannot build blead on AlmaLinux 8.

Is that because of this issue?

Correct.

[2022-05-02 17:30:00] LOG       #   Failed test 'ZLIB_VERSION (1.2.12) matches Compress::Zlib::zlib_version'
[2022-05-02 17:30:00] LOG       #   at t/cz-01version.t line 34.
[2022-05-02 17:30:00] LOG       #          got: '1.2.12'
[2022-05-02 17:30:00] LOG       #     expected: '1.2.11'
[2022-05-02 17:30:00] LOG       # 
[2022-05-02 17:30:00] LOG       # The version of zlib.h does not match the version of libz
[2022-05-02 17:30:00] LOG       # 
[2022-05-02 17:30:00] LOG       # You have zlib.h version 1.2.12
[2022-05-02 17:30:00] LOG       #      and libz   version 1.2.11
[2022-05-02 17:30:00] LOG       # 
[2022-05-02 17:30:00] LOG       # You probably have two versions of zlib installed on your system.
[2022-05-02 17:30:00] LOG       # Try removing the one you don't want to use and rebuild.
[2022-05-02 17:30:00] LOG       # Looks like you failed 1 test of 2.
[2022-05-02 17:30:00] LOG       ../cpan/IO-Compress/t/cz-01version.t ............................... 

[pmqs]

Right now I cannot build blead on AlmaLinux 8.

Is that because of this issue?

Correct.

[2022-05-02 17:30:00] LOG       #   Failed test 'ZLIB_VERSION (1.2.12) matches Compress::Zlib::zlib_version'
[2022-05-02 17:30:00] LOG       #   at t/cz-01version.t line 34.
[2022-05-02 17:30:00] LOG       #          got: '1.2.12'
[2022-05-02 17:30:00] LOG       #     expected: '1.2.11'
[2022-05-02 17:30:00] LOG       # 
[2022-05-02 17:30:00] LOG       # The version of zlib.h does not match the version of libz
[2022-05-02 17:30:00] LOG       # 
[2022-05-02 17:30:00] LOG       # You have zlib.h version 1.2.12
[2022-05-02 17:30:00] LOG       #      and libz   version 1.2.11
[2022-05-02 17:30:00] LOG       # 
[2022-05-02 17:30:00] LOG       # You probably have two versions of zlib installed on your system.
[2022-05-02 17:30:00] LOG       # Try removing the one you don't want to use and rebuild.
[2022-05-02 17:30:00] LOG       # Looks like you failed 1 test of 2.
[2022-05-02 17:30:00] LOG       ../cpan/IO-Compress/t/cz-01version.t ............................... 

OK - means this isn't a hypothetical issue anymore

[pmqs]

I think there are two robust approaches to make this problem go away forever

1. Statically link the object files, as you suggest @FGasper
   Concerns here are coming up with something that will work on all versions of Perl and all platforms.
2. Drop support for building with local copies of the zlib sources as @monkburger suggests.
   My recollection on why this feature exists dates from a time when some platforms didn't have easy access to zlib. These days the main Linux distros don't build C:R:Z with the local zlib sources at all. No idea what the Windows/Mac distros do. Need to see if this is likely to break Strawberry Perl & Activestate?

[FGasper]

Is it possible/feasible to include zlib as a github submodule? If so, you maybe could piggyback on top of zlib’s own build process.