t/porting/libperl.t failure with GCC 12 and -flto

[ntyni]

This is a bug report for perl from ntyni@debian.org,
generated with the help of perlbug 1.43 running under perl 5.37.6.

---

Description

As reported by Matthias Klose in https://bugs.debian.org/985884
t/porting/libperl.t fails when perl is built with link time optimization
(LTO).

Steps to Reproduce

On current Debian unstable, with gcc (Debian 12.2.0-9) 12.2.0
and perl v5.37.5-162-g52917b368f

$ ./Configure -des -Dusedevel -Dccflags=-flto -Dldflags=-flto && make -j4 && make test
[...]
t/porting/libperl ................................................ # Failed test 3 - has data const symbols at porting/libperl.t line 322
# Failed test 4 - has PL_no_mem at porting/libperl.t line 323
FAILED at test 3
[...]
Failed 1 test out of 2502, 99.96% okay.
	porting/libperl.t


$ ./perl -Ilib t/porting/libperl.t 
# $^O = linux
# $Config{archname} = x86_64-linux
# $Config{cc} = cc
# libperl = ../libperl.a
# nm = /usr/bin/nm
# nm_style = gnu
# nm_opt = 
# command: "/usr/bin/nm  ../libperl.a 2>libperl564590 |"
ok 1 - has object util.o
ok 2 - has text Perl_croak in util.o
not ok 3 - has data const symbols
# Failed test 3 - has data const symbols at t/porting/libperl.t line 322
not ok 4 - has PL_no_mem
# Failed test 4 - has PL_no_mem at t/porting/libperl.t line 323
# nocommon = 0
ok 5 - has PL_hash_seed_w
ok 6 - has PL_ppaddr
ok 7 - has undefined symbols
ok 8 - uses chmod (doio.o)
ok 9 - uses dlopen (DynaLoader.o)
ok 10 - uses exp (pp.o)
ok 11 - uses getenv (DynaLoader.o locale.o perl.o perlio.o regcomp.o toke.o util.o)
ok 12 - uses sigaction (mg.o util.o)
ok 13 - uses socket (doio.o)
ok 14 - uses time (perl.o pp_sys.o util.o)
ok 15 - uses no atoi ()
ok 16 - uses no atol ()
ok 17 - uses no atoll ()
ok 18 - uses no fgets ()
ok 19 - uses no gets ()
ok 20 - uses no sprintf ()
ok 21 - uses no strcat ()
ok 22 - uses no strcpy ()
ok 23 - uses no strncat ()
ok 24 - uses no strncpy ()
ok 25 - uses no strtol ()
ok 26 - uses no strtoq ()
ok 27 - uses no strtoul ()
ok 28 - uses no system ()
ok 29 - uses no tmpfile ()
ok 30 - uses no vsprintf ()
ok 31 - no S_ exports
1..31

Ubuntu is currently disabling the failing tests FWIW.

---

Flags

• category=core
• severity=low

---

Perl configuration

Site configuration information for perl 5.37.6:

Configured by ntyni at Wed Nov 16 19:23:28 UTC 2022.

Summary of my perl5 (revision 5 version 37 subversion 6) configuration:
  Commit id: 52917b368fe204d0670f020d5f0f3ad9ec236e01
  Platform:
    osname=linux
    osvers=5.19.0-2-amd64
    archname=x86_64-linux
    uname='linux carme 5.19.0-2-amd64 #1 smp preempt_dynamic debian 5.19.11-1 (2022-09-24) x86_64 gnulinux '
    config_args='-des -Dusedevel -Dccflags=-flto -Dldflags=-flto'
    hint=recommended
    useposix=true
    d_sigaction=define
    useithreads=undef
    usemultiplicity=undef
    use64bitint=define
    use64bitall=define
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
  Compiler:
    cc='cc'
    ccflags ='-flto -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2'
    optimize='-O2'
    cppflags='-flto -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include'
    ccversion=''
    gccversion='12.2.0'
    gccosandvers=''
    intsize=4
    longsize=8
    ptrsize=8
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=16
    longdblkind=3
    ivtype='long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='off_t'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='cc'
    ldflags ='-flto -fstack-protector-strong -L/usr/local/lib'
    libpth=/usr/local/lib /usr/lib/x86_64-linux-gnu /usr/lib /usr/lib64
    libs=-lpthread -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc -lgdbm_compat
    perllibs=-lpthread -ldl -lm -lcrypt -lutil -lc
    libc=/lib/x86_64-linux-gnu/libc.so.6
    so=so
    useshrplib=false
    libperl=libperl.a
    gnulibc_version='2.36'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs
    dlext=so
    d_dlsymun=undef
    ccdlflags='-Wl,-E'
    cccdlflags='-fPIC'
    lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector-strong'


---
@INC for perl 5.37.6:
    lib
    /usr/local/lib/perl5/site_perl/5.37.6/x86_64-linux
    /usr/local/lib/perl5/site_perl/5.37.6
    /usr/local/lib/perl5/5.37.6/x86_64-linux
    /usr/local/lib/perl5/5.37.6

---
Environment for perl 5.37.6:
    HOME=/home/ntyni
    LANG (unset)
    LANGUAGE (unset)
    LD_LIBRARY_PATH=/usr/lib/libeatmydata
    LOGDIR (unset)
    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
    PERL_BADLANG (unset)
    SHELL=/bin/bash

[ntyni]

I haven't tested if this is GCC 12 specific. It was reported quite a while ago so it might well happen with earlier GCC versions too.

[jkeenan]

I was able to reproduce these failures on Linux (Debian 11) using gcc-10, the default C-compiler:

$ git describe
v5.37.5-171-gd47ed502d5
$ sh ./Configure -des -Dusedevel -Dccflags=-flto -Dldflags=-flto && make test_porting
...
$ cd t;./perl harness -v porting/libperl.t; cd -

ok 1 - has object util.o
ok 2 - has text Perl_croak in util.o
not ok 3 - has data const symbols
# Failed test 3 - has data const symbols at porting/libperl.t line 322
not ok 4 - has PL_no_mem
# Failed test 4 - has PL_no_mem at porting/libperl.t line 323
ok 5 - has PL_hash_seed_w
...
ok 31 - no S_ exports
Failed 2/31 subtests 

Test Summary Report
-------------------
porting/libperl.t (Wstat: 0 Tests: 31 Failed: 2)
  Failed tests:  3-4
Files=1, Tests=31,  0 wallclock secs ( 0.02 usr  0.00 sys +  0.05 cusr  0.05 csys =  0.12 CPU)
Result: FAIL

And, for what it's worth, I tried this with gcc10 as the C-compiler on FreeBSD-12. make failed on both blead and at v5.36.0. Example:

[perlmonger: perl] $ git describe
v5.37.5-171-gd47ed502d5
[perlmonger: perl] $ sh ./Configure -des -Dusedevel -Dccflags=-flto -Dldflags=-flto -Dcc=gcc10 && make test_porting
...
Updating 'mktables.lst'
./miniperl -Ilib -MExtUtils::Miniperl -e 'writemain(\"perlmain.c", @ARGV)' DynaLoader 
gcc10 -c -DPERL_CORE -flto -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_FORTIFY_SOURCE=2 -std=c99 -O2 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings perlmain.c
gcc10 -o perl -flto -fstack-protector-strong -L/usr/local/lib  perlmain.o   libperl.a `cat ext.libs` -lpthread -ldl -lm -lcrypt -lutil -lc
/usr/local/bin/ld: /tmp//perl.Qqd7KG.ltrans0.ltrans.o: in function `xs_init':
<artificial>:(.text+0x6): undefined reference to `boot_DynaLoader'
/usr/local/bin/ld: <artificial>:(.text+0x10): undefined reference to `Perl_newXS'
/usr/local/bin/ld: /tmp//perl.Qqd7KG.ltrans0.ltrans.o: in function `main':
<artificial>:(.text.startup+0x23): undefined reference to `Perl_sys_init3'
/usr/local/bin/ld: <artificial>:(.text.startup+0x29): undefined reference to `PL_do_undump'
/usr/local/bin/ld: <artificial>:(.text.startup+0x31): undefined reference to `perl_alloc'
/usr/local/bin/ld: <artificial>:(.text.startup+0x49): undefined reference to `perl_construct'
/usr/local/bin/ld: <artificial>:(.text.startup+0x4f): undefined reference to `PL_perl_destruct_level'
/usr/local/bin/ld: <artificial>:(.text.startup+0x6c): undefined reference to `PL_exit_flags'
/usr/local/bin/ld: <artificial>:(.text.startup+0x72): undefined reference to `perl_parse'
/usr/local/bin/ld: <artificial>:(.text.startup+0x7d): undefined reference to `PL_sig_name'
/usr/local/bin/ld: <artificial>:(.text.startup+0x9a): undefined reference to `PL_sig_name'
/usr/local/bin/ld: <artificial>:(.text.startup+0xa5): undefined reference to `PL_sig_num'
/usr/local/bin/ld: <artificial>:(.text.startup+0xad): undefined reference to `Perl_rsignal_state'
/usr/local/bin/ld: <artificial>:(.text.startup+0xb4): undefined reference to `PL_csighandlerp'
/usr/local/bin/ld: <artificial>:(.text.startup+0xc0): undefined reference to `Perl_rsignal'
/usr/local/bin/ld: <artificial>:(.text.startup+0xd8): undefined reference to `perl_destruct'
/usr/local/bin/ld: <artificial>:(.text.startup+0xe7): undefined reference to `perl_free'
/usr/local/bin/ld: <artificial>:(.text.startup+0xec): undefined reference to `Perl_sys_term'
/usr/local/bin/ld: <artificial>:(.text.startup+0x100): undefined reference to `perl_run'
collect2: error: ld returned 1 exit status
*** Error code 1

Stop.
make: stopped in /usr/home/jkeenan/gitwork/perl

(I've never tried those 2 switches myself, so I don't know what to expect.)

[jkeenan]

This is a bug report for perl from ntyni@debian.org, generated with the help of perlbug 1.43 running under perl 5.37.6.

Description

As reported by Matthias Klose in https://bugs.debian.org/985884 t/porting/libperl.t fails when perl is built with link time optimization (LTO).

Steps to Reproduce

On current Debian unstable, with gcc (Debian 12.2.0-9) 12.2.0 and perl v5.37.5-162-g52917b368f

$ ./Configure -des -Dusedevel -Dccflags=-flto -Dldflags=-flto && make -j4 && make test

Can you describe, in lay person's terms, what -flto is intended to accomplish? I can't recall seeing that used in any of our smoke-testing, nor can I recall previous bug reports with that.

[jkeenan]

On Linux (Debian 11), using gcc-10 as the C-compiler, I checked out various tags and built with this configuration:

sh ./Configure -des -Dusedevel -Dccflags=-flto -Dldflags=-flto

As was the case with HEAD and v5.36.0, I got two failures in t/porting/libperl.t at each of v5.32.1 and v5.28.3. At v5.24.1 the build failed to complete.

So that leads me to ask: What evidence do you have that these configuration switches ever built and tested successfully?

[gregoa]

On Wed, 16 Nov 2022 15:26:22 -0800, James E Keenan wrote: Can you describe, in lay person's terms, what `-flto` is intended to accomplish? I can't recall seeing that used in any of our smoke-testing, nor can I recall previous bug reports with that.

I'm sure that @ntyni can explain this in a better way, but for starters, the announcement at <https://lists.debian.org/debian-devel/2022/06/msg00092.html> and the wiki link there might be helpful. Cheers, gregor

…

-- .''`. https://info.comodo.priv.at -- Debian Developer https://www.debian.org : :' : OpenPGP fingerprint D1E1 316E 93A7 60A8 104D 85FA BB3A 6801 8649 AA06 `. `' Member VIBE!AT & SPI Inc. -- Supporter Free Software Foundation Europe `- BOFH excuse #11: magnetic interference from money/credit cards

[iabyn]

On Wed, Nov 16, 2022 at 12:17:18PM -0800, Niko Tyni wrote: $ ./Configure -des -Dusedevel -Dccflags=-flto -Dldflags=-flto && make -j4 && make test [...] t/porting/libperl ................................................ # Failed test 3 - has data const symbols at porting/libperl.t line 322 # Failed test 4 - has PL_no_mem at porting/libperl.t line 323 FAILED at test 3 [...] Failed 1 test out of 2502, 99.96% okay. porting/libperl.t not ok 3 - has data const symbols # Failed test 3 - has data const symbols at t/porting/libperl.t line 322 not ok 4 - has PL_no_mem # Failed test 4 - has PL_no_mem at t/porting/libperl.t line 323 # nocommon = 0

The proximate cause of these two failures is a lack of readonly symbols in libperl.a. In a normal build: $ /usr/bin/nm libperl.a | grep ' [Rr] ' 0000000000000e00 r array_passed_to_stat 00000000000003e0 r bodies_by_type 00000000000087c0 r custom_op_register_vtbl ... $ /usr/bin/nm libperl.a | grep PL_no_mem U PL_no_mem 0000000000000850 R PL_no_mem $ In a -flto build: $ /usr/bin/nm libperl.a | grep ' [Rr] ' $ $ /usr/bin/nm libperl.a | grep PL_no_mem U PL_no_mem 00000000 D PL_no_mem $ (Note that PL_no_mem has changed from R to D.) I don't know enough about LTO to know whether not having such symbols is to be expected and thus whether libperl.t should just skip those two tests when perl is configured with -flto.

…

-- More than any other time in history, mankind faces a crossroads. One path leads to despair and utter hopelessness. The other, to total extinction. Let us pray we have the wisdom to choose correctly. -- Woody Allen

[jkeenan]

On Wed, 16 Nov 2022 15:26:22 -0800, James E Keenan wrote: Can you describe, in lay person's terms, what -flto is intended to accomplish? I can't recall seeing that used in any of our smoke-testing, nor can I recall previous bug reports with that.
I'm sure that @ntyni can explain this in a better way, but for starters, the announcement at https://lists.debian.org/debian-devel/2022/06/msg00092.html and the wiki link there might be helpful. Cheers, gregor

@gregoa, thank you very much for those URLs. Here is my summary/paraphrase of what they say:

Link time optimizations are an optimization that helps with a single digit percent number optimizing both for smaller size, and better speed. ... The proposal is to turn on LTO by default on most 64bit release architectures. In test rebuilds, there were 373 packages (dd-list in the wiki page) found not to build with link time optimizations for various reasons. [Perl is one of those with test failures.] The idea is to fix as many of these as possible, and then change the packaging for the others to just turn off LTO in the package build.

So this ticket is only incidentally a report of test failures. It's actually a request for a new functionality, i.e., build perl in a way not previously requested so that Debian (and presumably other distributions) can move in a new direction. (I note this because the original posting led me to believe this was a regression in perl; it's not.)

Until reading this ticket I had never heard of link-time optimizations, so I don't know how we should proceed. @ntyni, @gregoa, @jmdh, could one of you write up a post for the perl5-porters mailing list about Debian's move to LTO? That list has much wider visibility than these GH issues and it would be good for our readership to learn more about this new initiative.

Thank you very much.
Jim Keenan

[ntyni]

So this ticket is only incidentally a report of test failures. It's actually a request for a new functionality, i.e., build perl in a way not previously requested so that Debian (and presumably other distributions) can move in a new direction. (I note this because the original posting led me to believe this was a regression in perl; it's not.)

Indeed. Apologies for the misunderstanding.

Until reading this ticket I had never heard of link-time optimizations, so I don't know how we should proceed. @ntyni, @gregoa, @jmdh, could one of you write up a post for the perl5-porters mailing list about Debian's move to LTO? That list has much wider visibility than these GH issues and it would be good for our readership to learn more about this new initiative.

Okay; mail sent.

Niko

[Leont]

I don't know enough about LTO to know whether not having such symbols is
to be expected and thus whether libperl.t should just skip those two tests
when perl is configured with -flto.

I think we should change the test. We should not make assumptions about in which section the compiler puts that variable.

[jkeenan]

I don't know enough about LTO to know whether not having such symbols is
to be expected and thus whether libperl.t should just skip those two tests
when perl is configured with -flto.

I think we should change the test. We should not make assumptions about in which section the compiler puts that variable.

Could you explain what these "sections" are? Also, would it be possible to get a p.r. for skipping the tests?

[Leont]

Could you explain what these "sections" are?

They're the different parts of the ELF binary format for executables and libraries. There is one section that is readonly and pre-initialized at compile time and the test expects a the variable to be put in there.

Apparently when using link-time optimization it shows up in the "initialized data section" (used for static writable variables) for the library, even if it ends up read-only in the resulting executable.

I don't know if this is a bug in nm or a side-effect of this decision being made at link time. In either case, we should either:

• skip() if $ccflags =~ /-flto/
• remove the test entirely