Fix test failure in POSIX/t/mb.t with semi-broken locales

[p5pRT]

Migrated from rt.perl.org#134182 (status was 'open')

Searchable as RT134182$

[p5pRT]

From @jmdh

Created by @jmdh

This is a bug report for perl from dom@​earth.li,
generated with the help of perlbug 1.40 running under perl 5.24.1.

-----------------------------------------------------------------
The test introduced at 25d7b7a
fails in environments where LANG is set to a locale which is not
installed and LC_ALL is set to a valid locale. Such an environment
is arguably broken, but arises in a common use case in Debian
build tools. This modification takes a more robust approach to
modifying the environment.

Perl Info

Flags:
    category=library
    severity=low
    module=POSIX

Site configuration information for perl 5.24.1:

Configured by Debian Project at Thu Nov 29 11:11:57 UTC 2018.

Summary of my perl5 (revision 5 version 24 subversion 1) configuration:
   
  Platform:
    osname=linux, osvers=3.16.0, archname=x86_64-linux-gnu-thread-multi
    uname='linux localhost 3.16.0 #1 smp debian 3.16.0 x86_64 gnulinux '
    config_args='-Dusethreads -Duselargefiles -Dcc=x86_64-linux-gnu-gcc -Dcpp=x86_64-linux-gnu-cpp -Dld=x86_64-linux-gnu-gcc -Dccflags=-DDEBIAN -Wdate-time -D_FORTIFY_SOURCE=2 -g -O2 -fdebug-prefix-map=/build/perl-CWhbRh/perl-5.24.1=. -fstack-protector-strong -Wformat -Werror=format-security -Dldflags= -Wl,-z,relro -Dlddlflags=-shared -Wl,-z,relro -Dcccdlflags=-fPIC -Darchname=x86_64-linux-gnu -Dprefix=/usr -Dprivlib=/usr/share/perl/5.24 -Darchlib=/usr/lib/x86_64-linux-gnu/perl/5.24 -Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5 -Dvendorarch=/usr/lib/x86_64-linux-gnu/perl5/5.24 -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl/5.24.1 -Dsitearch=/usr/local/lib/x86_64-linux-gnu/perl/5.24.1 -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3 -Dusesitecustomize -Duse64bitint -Dman1ext=1 -Dman3ext=3perl
-Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Ud_ualarm -Uusesfio -Uusenm -Ui_libutil -Uversiononly -DDEBUGGING=-g -Doptimize=-O2 -dEs -Duseshrplib -Dlibperl=libperl.so.5.24.1'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    use64bitint=define, use64bitall=define, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='x86_64-linux-gnu-gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O2 -g',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-aliasing -pipe -I/usr/local/include'
    ccversion='', gccversion='6.3.0 20170516', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678, doublekind=3
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16, longdblkind=3
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='x86_64-linux-gnu-gcc', ldflags =' -fstack-protector-strong -L/usr/local/lib'
    libpth=/usr/local/lib /usr/lib/gcc/x86_64-linux-gnu/6/include-fixed /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib
    libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt
    perllibs=-ldl -lm -lpthread -lc -lcrypt
    libc=libc-2.24.so, so=so, useshrplib=true, libperl=libperl.so.5.24
    gnulibc_version='2.24'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib -fstack-protector-strong'

Locally applied patches:
    DEBPKG:debian/cpan_definstalldirs - Provide a sensible INSTALLDIRS default for modules installed from CPAN.
    DEBPKG:debian/db_file_ver - https://bugs.debian.org/340047 Remove overly restrictive DB_File version check.
    DEBPKG:debian/doc_info - Replace generic man(1) instructions with Debian-specific information.
    DEBPKG:debian/enc2xs_inc - https://bugs.debian.org/290336 Tweak enc2xs to follow symlinks and ignore missing @INC directories.
    DEBPKG:debian/errno_ver - https://bugs.debian.org/343351 Remove Errno version check due to upgrade problems with long-running processes.
    DEBPKG:debian/libperl_embed_doc - https://bugs.debian.org/186778 Note that libperl-dev package is required for embedded linking
    DEBPKG:fixes/respect_umask - Respect umask during installation
    DEBPKG:debian/writable_site_dirs - Set umask approproately for site install directories
    DEBPKG:debian/extutils_set_libperl_path - EU:MM: set location of libperl.a under /usr/lib
    DEBPKG:debian/no_packlist_perllocal - Don't install .packlist or perllocal.pod for perl or vendor
    DEBPKG:debian/fakeroot - Postpone LD_LIBRARY_PATH evaluation to the binary targets.
    DEBPKG:debian/instmodsh_doc - Debian policy doesn't install .packlist files for core or vendor.
    DEBPKG:debian/ld_run_path - Remove standard libs from LD_RUN_PATH as per Debian policy.
    DEBPKG:debian/libnet_config_path - Set location of libnet.cfg to /etc/perl/Net as /usr may not be writable.
    DEBPKG:debian/mod_paths - Tweak @INC ordering for Debian
    DEBPKG:debian/prune_libs - https://bugs.debian.org/128355 Prune the list of libraries wanted to what we actually need.
    DEBPKG:fixes/net_smtp_docs - [rt.cpan.org #36038] https://bugs.debian.org/100195 Document the Net::SMTP 'Port' option
    DEBPKG:debian/perlivp - https://bugs.debian.org/510895 Make perlivp skip include directories in /usr/local
    DEBPKG:debian/deprecate-with-apt - https://bugs.debian.org/747628 Point users to Debian packages of deprecated core modules
    DEBPKG:debian/squelch-locale-warnings - https://bugs.debian.org/508764 Squelch locale warnings in Debian package maintainer scripts
    DEBPKG:debian/skip-upstream-git-tests - Skip tests specific to the upstream Git repository
    DEBPKG:debian/patchlevel - https://bugs.debian.org/567489 List packaged patches for 5.24.1-3+deb9u5 in patchlevel.h
    DEBPKG:debian/skip-kfreebsd-crash - https://bugs.debian.org/628493 [perl #96272] Skip a crashing test case in t/op/threads.t on GNU/kFreeBSD
    DEBPKG:fixes/document_makemaker_ccflags - https://bugs.debian.org/628522 [rt.cpan.org #68613] Document that CCFLAGS should include $Config{ccflags}
    DEBPKG:debian/find_html2text - https://bugs.debian.org/640479 Configure CPAN::Distribution with correct name of html2text
    DEBPKG:debian/perl5db-x-terminal-emulator.patch - https://bugs.debian.org/668490 Invoke x-terminal-emulator rather than xterm in perl5db.pl
    DEBPKG:debian/cpan-missing-site-dirs - https://bugs.debian.org/688842 Fix CPAN::FirstTime defaults with nonexisting site dirs if a parent is writable
    DEBPKG:fixes/memoize_storable_nstore - [rt.cpan.org #77790] https://bugs.debian.org/587650 Memoize::Storable: respect 'nstore' option not respected
    DEBPKG:debian/regen-skip - Skip a regeneration check in unrelated git repositories
    DEBPKG:debian/makemaker-pasthru - https://bugs.debian.org/758471 Pass LD settings through to subdirectories
    DEBPKG:debian/makemaker-manext - https://bugs.debian.org/247370 Make EU::MakeMaker honour MANnEXT settings in generated manpage headers
    DEBPKG:debian/devel-ppport-reproducibility - https://bugs.debian.org/801523 Sort the list of XS code files when generating RealPPPort.xs
    DEBPKG:debian/encode-unicode-bom-doc - https://bugs.debian.org/798727 Document Debian backport of Encode::Unicode fix
    DEBPKG:debian/kfreebsd-softupdates - https://bugs.debian.org/796798 Work around Debian Bug#796798
    DEBPKG:fixes/autodie-scope - https://bugs.debian.org/798096 Fix a scoping issue with "no autodie" and the "system" sub
    DEBPKG:fixes/crosscompile-no-targethost - [23695c0] [perl #127234] Fix the Configure escape with usecrosscompile but no targethost
    DEBPKG:fixes/memoize-pod - [rt.cpan.org #89441] Fix POD errors in Memoize
    DEBPKG:fixes/ok-pod - Added encoding for pod.
    DEBPKG:debian/hurd-softupdates - https://bugs.debian.org/822735 Fix t/op/stat.t failures on hurd
    DEBPKG:fixes/nntp_docs - https://bugs.debian.org/51962 Net::NNTP: Correct innd/nnrpd confusion in relation to Reader option
    DEBPKG:fixes/math_complex_doc_great_circle - https://bugs.debian.org/697567 [rt.cpan.org #114104] Math::Trig: clarify definition of great_circle_midpoint
    DEBPKG:fixes/math_complex_doc_see_also - https://bugs.debian.org/697568 [rt.cpan.org #114105] Math::Trig: add missing SEE ALSO
    DEBPKG:fixes/math_complex_doc_angle_units - https://bugs.debian.org/731505 [rt.cpan.org #114106] Math::Trig: document angle units
    DEBPKG:fixes/cpan_web_link - https://bugs.debian.org/367291 CPAN: Add link to main CPAN web site
    DEBPKG:fixes/time_piece_doc - https://bugs.debian.org/817925 Time::Piece: Improve documentation for add_months and add_years
    DEBPKG:fixes/perlbug-refactor - https://bugs.debian.org/822463 [perl #128020] perlbug: Refactor duplicated file reading code
    DEBPKG:fixes/perlbug-linewrap - https://bugs.debian.org/822463 [perl #128020] perlbug: wrap overly long lines
    DEBPKG:fixes/hurd_sigaction - https://bugs.debian.org/825016 [d54f4ed] ext/POSIX/t/sigaction.t: Skip uid and pid tests on GNU/Hurd
    DEBPKG:fixes/hurd_hints - [4694301] https://bugs.debian.org/825020 [perl #128279] Modify hints for Hurd per Debian ticket 825020.
    DEBPKG:fixes/extutils-parsexs-reproducibility - [perl #128517] https://bugs.debian.org/829296 Make the output of ExtUtils::ParseXS reproducible
    DEBPKG:debian/CVE-2016-1238/sitecustomize-in-etc - Look for sitecustomize.pl in /etc/perl rather than sitelib on Debian systems
    DEBPKG:debian/CVE-2016-1238/test-suite-without-dot - [perl #127810] Patch unit tests to explicitly insert "." into @INC when needed.
    DEBPKG:debian/CVE-2016-1238/eumm-without-dot - [perl #127810] Add PERL_USE_UNSAFE_INC support to EU::MM for fortify_inc support.
    DEBPKG:debian/CVE-2016-1238/cpan-without-dot - [perl #127810] Set PERL_USE_UNSAFE_INC for cpan usage
    DEBPKG:debian/document_inc_removal - Document in perlvar that we remove '.' from @INC by default
    DEBPKG:fixes/extutils_makemaker_reproducible - https://bugs.debian.org/835815 https://bugs.debian.org/834190 Make perllocal.pod files reproducible
    DEBPKG:debian/CVE-2016-1238/remove-inc-test - Remove test for '.' in @INC as it might not be
    DEBPKG:fixes/file_path_hurd_errno - File-Path: Fix test failure in Hurd due to hard-coded ENOENT
    DEBPKG:debian/hppa_op_optimize_workaround - https://bugs.debian.org/838613 Temporarily lower the optimization of op.c on hppa due to gcc-6 problems
    DEBPKG:fixes/test-builder-warning - https://bugs.debian.org/840968 Silence a 'used only once' warning in Test::Builder
    DEBPKG:fixes/longdblinf-randomness - [dd68853] [perl #130133] https://bugs.debian.org/844752 Configure: fix garbage filtering with 80-bit long doubles
    DEBPKG:debian/installman-utf8 - https://bugs.debian.org/840211 Generate man pages with UTF-8 characters
    DEBPKG:fixes/list_assign_leak - [1050723] [perl #130766] https://bugs.debian.org/855064 avoid a leak in list assign from/to magic values
    DEBPKG:fixes/perlfunc_inc_doc - [a03e9f8] https://bugs.debian.org/839536 [perl #130832] Documentation fixes for '.' possibly no longer being in @INC
    DEBPKG:fixes/file_path_chmod_race - https://bugs.debian.org/863870 [rt.cpan.org #121951] Prevent directory chmod race attack.
    DEBPKG:fixes/extutils_file_path_compat - Correct the order of tests of chmod(). (#294)
    DEBPKG:debian/customized - Update customized.dat for files patched in Debian
    DEBPKG:fixes/getopt-long-1 - https://bugs.debian.org/855532 [rt.cpan.org #114999] Fix bug RT#114999
    DEBPKG:fixes/getopt-long-2 - [rt.cpan.org #120300] Withdraw part of commit 5d9947fb445327c7299d8beb009d609bc70066c0, which tries to implement more GNU getopt_long campatibility. GNU
    DEBPKG:fixes/getopt-long-3 - provide a default value for optional arguments
    DEBPKG:fixes/getopt-long-4 - https://bugs.debian.org/864544 [rt.cpan.org #122068] Fix issue #122068.
    DEBPKG:fixes/fbm-instr-crash - [bb152a4] [perl #131575] https://bugs.debian.org/864782 don't call Perl_fbm_instr() with negative length
    DEBPKG:debian/CVE-2016-1238/base-pm-amends-pt2 - [1afa289] Limit dotless-INC effect on base.pm with guard:
    DEBPKG:fixes/CVE-2017-12837 - https://bugs.debian.org/875596 [perl #131582] [f7e5417] regcomp [perl #131582]
    DEBPKG:fixes/CVE-2017-12883 - https://bugs.debian.org/875597 [perl #131598] [40b3cda] PATCH: [perl #131598]
    DEBPKG:fixes/CVE-2018-6797 - [perl #132227] (perl #132227) restart a node if we change to uni rules within the node and encounter a sharp S
    DEBPKG:fixes/CVE-2018-6798/pt1 - [perl #132063] Heap buffer overflow
    DEBPKG:fixes/CVE-2018-6798/pt2 - [perl #132063] v5.24.3: fix TRIE_READ_CHAR and DECL_TRIE_TYPE to account for non-utf8 target
    DEBPKG:fixes/CVE-2018-6798/pt3 - [perl #132063] (perl #132063) we should no longer warn for this code
    DEBPKG:fixes/CVE-2018-6913 - [perl #131844] (perl #131844) fix various space calculation issues in pp_pack.c
    DEBPKG:fixes/CVE-2018-12015-Archive-Tar-directory-traversal - https://bugs.debian.org/900834 [rt.cpan.org #125523] Remove existing files before overwriting them
    DEBPKG:fixes/CVE-2018-18311 - Perl_my_setenv(); handle integer wrap
    DEBPKG:fixes/CVE-2018-18312 - for 5.26 maint
    DEBPKG:fixes/CVE-2018-18313 - regcomp.c: Convert some strchr to memchr
    DEBPKG:fixes/CVE-2018-18314 - fix #131649 - extended charclass can trigger assert


@INC for perl 5.24.1:
    /etc/perl
    /usr/local/lib/x86_64-linux-gnu/perl/5.24.1
    /usr/local/share/perl/5.24.1
    /usr/lib/x86_64-linux-gnu/perl5/5.24
    /usr/share/perl5
    /usr/lib/x86_64-linux-gnu/perl/5.24
    /usr/share/perl/5.24
    /usr/local/lib/site_perl
    /usr/lib/x86_64-linux-gnu/perl-base


Environment for perl 5.24.1:
    HOME=/home/dom
    LANG=en_GB.UTF-8
    LANGUAGE=en_GB:en
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/home/dom/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/sbin:/usr/sbin
    PERL_BADLANG (unset)
    SHELL=/bin/bash

[p5pRT]

From @jmdh

0001-Fix-edge-case-test-failure-in-ext-POSIX-t-mb.t.patch

From ba80ce1f59e6aa82532d84627b8c5d094eeda1a4 Mon Sep 17 00:00:00 2001
From: Dominic Hargreaves <dom@earth.li>
Date: Fri, 7 Jun 2019 10:04:26 +0100
Subject: [PATCH] Fix edge case test failure in ext/POSIX/t/mb.t

This new test fails in an environment where LANG is set to one thing and
LC_ALL is set to another, and where LANG is set to a locale which is
not installed in the environment in question.

Such a test environment is arguably broken, but appears in common
chroot setups such as Debian's sbuild tool where LANG is inherited from
the parent environment, and LC_ALL is used to override it.
---
 ext/POSIX/t/mb.t | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/ext/POSIX/t/mb.t b/ext/POSIX/t/mb.t
index 053693e611..3312b0d737 100644
--- a/ext/POSIX/t/mb.t
+++ b/ext/POSIX/t/mb.t
@@ -34,9 +34,13 @@ SKIP: {
     my $utf8_locale = find_utf8_ctype_locale();
     skip("no utf8 locale available", 3) unless $utf8_locale;
 
+    # Here we need to influence LC_CTYPE, but it's not enough to just
+    # set this because LC_ALL could override it. It's also not enough
+    # to delete LC_ALL because it could be used to override other
+    # variables such as LANG in the underlying test environment.
+    # Continue to set LC_CTYPE just in case...
     local $ENV{LC_CTYPE} = $utf8_locale;
-    local $ENV{LC_ALL};
-    delete $ENV{LC_ALL};
+    local $ENV{LC_ALL} = $utf8_locale;
 
     fresh_perl_like(
         'use POSIX; print &POSIX::MB_CUR_MAX',
-- 
2.11.0

[p5pRT]

From @jkeenan

On Fri, 07 Jun 2019 10​:06​:58 GMT, dom wrote​:

This is a bug report for perl from dom@​earth.li,
generated with the help of perlbug 1.40 running under perl 5.24.1.

-----------------------------------------------------------------
The test introduced at 25d7b7a
fails in environments where LANG is set to a locale which is not
installed and LC_ALL is set to a valid locale. Such an environment
is arguably broken, but arises in a common use case in Debian
build tools. This modification takes a more robust approach to
modifying the environment.

Pushed to blead in commit 69b89a0, with one committer's edit -- I had to remove a non-printing character in the patch​:

#####
  #<A0>Continue to set LC_CTYPE just in case...
#####

Dom, since I doubt any of our smoke-testing rigs are set up to reproduce this problem, could you send us some sort of evidence that the problem has been fixed?

Thank you very much.
--
James E Keenan (jkeenan@​cpan.org)

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @jmdh

On Fri, Jun 07, 2019 at 05​:11​:10AM -0700, James E Keenan via RT wrote​:

Dom, since I doubt any of our smoke-testing rigs are set up to reproduce this problem, could you send us some sort of evidence that the problem has been fixed?

Before the patch was applied, this test failed in my Debian sbuild
environment. Afterwards, it succeeded.

(The relevant detail here is that outside sbuild, my LANG is en_GB.UTF-8.
This is progagated to within sbuild (which sets up a chroot) but that
environment does not have the en_GB.UTF-8 locale data installed (since it's
supposed to be a minimal environment).

sbuild itself corrects for this problem by setting LC_ALL, so the fact
that the test overrode LC_ALL caused the non-working en_GB.UTF-8 locale
to be used.)

Thanks,
Domninic.

[p5pRT]

From @jkeenan

Dom,

Unfortunately I have to call your attention to 2 smoke-test failures in ext/POSIX/t/mb.t which were recorded *after* I applied your patch.

http​://perl5.test-smoke.org/report/89146
logs at​: http​://perl5.test-smoke.org/logfile/89146

http​://perl5.test-smoke.org/report/89211
logs at​: http​://perl5.test-smoke.org/logfile/89211

(These can be tracked via this search​:
http​://perl5.test-smoke.org/submatrix?test=../ext/POSIX/t/mb.t&pversion=5.31.1)

In each case the failures in mb.t occurred when blead was configured as follows​:

[stdio] -Dcc=clang -Accflags="-Werror=declaration-after-statement -g -fno-omit-frame-pointer -fsanitize=address -fno-common -fsanitize-blacklist=`pwd`/asan_ignore" -Aldflags="-fsanitize=address"

With and without -DDEBUGGING.

A couple of points​:

1. Tester is using what I would guess is an advanced version of the Linux kernel​: 5.0.9-200 versus my own 4.15.0-51 (Ubuntu 18.04 LTS). OTOH, we are getting smoke-test reports from rigs with even higher-numbered Linux kernels.

2. I myself don't understand all those compiler switches the tester is using. In particular, 'make' fails for me on FreeBSD-11.2 when I use those compiler switches.

3. Nonetheless, when I build a perl with all those switches (except -DDEBUGGING), I get those test same failures. See attachments.

4. When I build blead with those same compiler switches at the commit immediately prior to the one where I applied your patch, I get a PASS.

#####
$ git show | head -1
commit fb55ce6
$ cd t;./perl harness -v ../ext/POSIX/t/mb.t; cd -

ok 1 - mblen() basically works
ok 2 - MB_CUR_MAX is at least 4 in a UTF-8 locale
ok 3 - mblen() recognizes invalid multibyte characters
ok 4 - mblen() works on UTF-8 characters
ok
All tests successful.
Files=1, Tests=4, 0 wallclock secs ( 0.02 usr 0.00 sys + 0.26 cusr 0.12 csys = 0.40 CPU)
Result​: PASS
#####

So your patch has triggered test failures, albeit under these very obscure conditions. I'm going to revert your patch from blead and then re-apply it in a branch so that we can continue to gather smoke-test reports.

Thank you very much.

--
James E Keenan (jkeenan@​cpan.org)

[p5pRT]

From @jkeenan

# Failed test 3 - mblen() recognizes invalid multibyte characters at ../../t/test.pl line 1062
# got "=================================================================\n==1656==ERROR​: AddressSanitizer​: heap-use-after-free on address 0x6020000006f0 at pc 0x0000004b166d bp 0x7fff1853e910 sp 0x7fff1853e0c0\nREAD of size 2 at 0x6020000006f0 thread T0\n #0 0x4b166c (/home/jkeenan/gitwork/perl/perl+0x4b166c)\n #1 0x6f0758 (/home/jkeenan/gitwork/perl/perl+0x6f0758)\n #2 0x6efb1f (/home/jkeenan/gitwork/perl/perl+0x6efb1f)\n #3 0x81bd6f (/home/jkeenan/gitwork/perl/perl+0x81bd6f)\n #4 0x73e3c9 (/home/jkeenan/gitwork/perl/perl+0x73e3c9)\n #5 0x5a27b1 (/home/jkeenan/gitwork/perl/perl+0x5a27b1)\n #6 0x598b20 (/home/jkeenan/gitwork/perl/perl+0x598b20)\n #7 0x56ca35 (/home/jkeenan/gitwork/perl/perl+0x56ca35)\n #8 0x554aff (/home/jkeenan/gitwork/perl/perl+0x554aff)\n #9 0x558399 (/home/jkeenan/gitwork/perl/perl+0x558399)\n #10 0x6230bd (/home/jkeenan/gitwork/perl/perl+0x6230bd)\n #11 0x823bdb (/home/jkeenan/gitwork/perl/perl+0x823bdb)\n #12 0x81f02c (/home/jkeenan/gitwork/perl/perl+0x81f02c)\n #13 0x73e3c9 (/home/jkeenan/gitwork/perl/perl+0x73e3c9)\n #14 0x5a27b1 (/home/jkeenan/gitwork/perl/perl+0x5a27b1)\n #15 0x598b20 (/home/jkeenan/gitwork/perl/perl+0x598b20)\n #16 0x56ca35 (/home/jkeenan/gitwork/perl/perl+0x56ca35)\n #17 0x554aff (/home/jkeenan/gitwork/perl/perl+0x554aff)\n #18 0x558399 (/home/jkeenan/gitwork/perl/perl+0x558399)\n #19 0x6230bd (/home/jkeenan/gitwork/perl/perl+0x6230bd)\n #20 0x59d88c (/home/jkeenan/gitwork/perl/perl+0x59d88c)\n #21 0x52f0fc (/home/jkeenan/gitwork/perl/perl+0x52f0fc)\n #22 0x7f57db6adb96 (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)\n #23 0x436b69 (/home/jkeenan/gitwork/perl/perl+0x436b69)\n\n0x6020000006f0 is located 0 bytes inside of 8-byte region [0x6020000006f0,0x6020000006f8)\nfreed by thread T0 here​:\n #0 0x4f6850 (/home/jkeenan/gitwork/perl/perl+0x4f6850)\n #1 0x7f57db6b94cf (/lib/x86_64-linux-gnu/libc.so.6+0x2d4cf)\n\npreviously allocated by thread T0 here​:\n #0 0x4f6a20 (/home/jkeenan/gitwork/perl/perl+0x4f6a20)\n #1 0x7f57db7299b9 (/lib/x86_64-linux-gnu/libc.so.6+0x9d9b9)\n\nSUMMARY​: AddressSanitizer​: heap-use-after-free (/home/jkeenan/gitwork/perl/perl+0x4b166c) \nShadow bytes around the buggy address​:\n 0x0c047fff8080​: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa 00 fa\n 0x0c047fff8090​: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa 00 fa\n 0x0c047fff80a0​: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa 00 fa\n 0x0c047fff80b0​: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa 00 fa\n 0x0c047fff80c0​: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa 00 fa\n=>0x0c047fff80d0​: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa[fd]fa\n 0x0c047fff80e0​: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa 00 fa\n 0x0c047fff80f0​: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa\n 0x0c047fff8100​: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa\n 0x0c047fff8110​: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa\n 0x0c047fff8120​: fa fa fd fa fa fa 06 fa fa fa 00 fa fa fa fd fa\nShadow byte legend (one shadow byte represents 8 application bytes)​:\n Addressable​: 00\n Partially addressable​: 01 02 03 04 05 06 07 \n Heap left redzone​: fa\n Freed heap region​: fd\n Stack left redzone​: f1\n Stack mid redzone​: f2\n Stack right redzone​: f3\n Stack after return​: f5\n Stack use after scope​: f8\n Global redzone​: f9\n Global init order​: f6\n Poisoned by user​: f7\n Container overflow​: fc\n Array cookie​: ac\n Intra object redzone​: bb\n ASan internal​: fe\n Left alloca redzone​: ca\n Right alloca redzone​: cb\n==1656==ABORTING"
# expected "-1"
# PROG​:
# use POSIX; print &POSIX​::mblen("Ã(", 2)
# STATUS​: 256
# Failed test 4 - mblen() works on UTF-8 characters at ../../t/test.pl line 1062
# got "=================================================================\n==1658==ERROR​: AddressSanitizer​: heap-use-after-free on address 0x6020000006f0 at pc 0x0000004b166d bp 0x7ffe19094070 sp 0x7ffe19093820\nREAD of size 2 at 0x6020000006f0 thread T0\n #0 0x4b166c (/home/jkeenan/gitwork/perl/perl+0x4b166c)\n #1 0x6f0758 (/home/jkeenan/gitwork/perl/perl+0x6f0758)\n #2 0x6efb1f (/home/jkeenan/gitwork/perl/perl+0x6efb1f)\n #3 0x81bd6f (/home/jkeenan/gitwork/perl/perl+0x81bd6f)\n #4 0x73e3c9 (/home/jkeenan/gitwork/perl/perl+0x73e3c9)\n #5 0x5a27b1 (/home/jkeenan/gitwork/perl/perl+0x5a27b1)\n #6 0x598b20 (/home/jkeenan/gitwork/perl/perl+0x598b20)\n #7 0x56ca35 (/home/jkeenan/gitwork/perl/perl+0x56ca35)\n #8 0x554aff (/home/jkeenan/gitwork/perl/perl+0x554aff)\n #9 0x558399 (/home/jkeenan/gitwork/perl/perl+0x558399)\n #10 0x6230bd (/home/jkeenan/gitwork/perl/perl+0x6230bd)\n #11 0x823bdb (/home/jkeenan/gitwork/perl/perl+0x823bdb)\n #12 0x81f02c (/home/jkeenan/gitwork/perl/perl+0x81f02c)\n #13 0x73e3c9 (/home/jkeenan/gitwork/perl/perl+0x73e3c9)\n #14 0x5a27b1 (/home/jkeenan/gitwork/perl/perl+0x5a27b1)\n #15 0x598b20 (/home/jkeenan/gitwork/perl/perl+0x598b20)\n #16 0x56ca35 (/home/jkeenan/gitwork/perl/perl+0x56ca35)\n #17 0x554aff (/home/jkeenan/gitwork/perl/perl+0x554aff)\n #18 0x558399 (/home/jkeenan/gitwork/perl/perl+0x558399)\n #19 0x6230bd (/home/jkeenan/gitwork/perl/perl+0x6230bd)\n #20 0x59d88c (/home/jkeenan/gitwork/perl/perl+0x59d88c)\n #21 0x52f0fc (/home/jkeenan/gitwork/perl/perl+0x52f0fc)\n #22 0x7fdf1d3bfb96 (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)\n #23 0x436b69 (/home/jkeenan/gitwork/perl/perl+0x436b69)\n\n0x6020000006f0 is located 0 bytes inside of 8-byte region [0x6020000006f0,0x6020000006f8)\nfreed by thread T0 here​:\n #0 0x4f6850 (/home/jkeenan/gitwork/perl/perl+0x4f6850)\n #1 0x7fdf1d3cb4cf (/lib/x86_64-linux-gnu/libc.so.6+0x2d4cf)\n\npreviously allocated by thread T0 here​:\n #0 0x4f6a20 (/home/jkeenan/gitwork/perl/perl+0x4f6a20)\n #1 0x7fdf1d43b9b9 (/lib/x86_64-linux-gnu/libc.so.6+0x9d9b9)\n\nSUMMARY​: AddressSanitizer​: heap-use-after-free (/home/jkeenan/gitwork/perl/perl+0x4b166c) \nShadow bytes around the buggy address​:\n 0x0c047fff8080​: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa 00 fa\n 0x0c047fff8090​: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa 00 fa\n 0x0c047fff80a0​: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa 00 fa\n 0x0c047fff80b0​: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa 00 fa\n 0x0c047fff80c0​: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa 00 fa\n=>0x0c047fff80d0​: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa[fd]fa\n 0x0c047fff80e0​: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa 00 fa\n 0x0c047fff80f0​: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa\n 0x0c047fff8100​: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa\n 0x0c047fff8110​: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa\n 0x0c047fff8120​: fa fa fd fa fa fa 06 fa fa fa 00 fa fa fa fd fa\nShadow byte legend (one shadow byte represents 8 application bytes)​:\n Addressable​: 00\n Partially addressable​: 01 02 03 04 05 06 07 \n Heap left redzone​: fa\n Freed heap region​: fd\n Stack left redzone​: f1\n Stack mid redzone​: f2\n Stack right redzone​: f3\n Stack after return​: f5\n Stack use after scope​: f8\n Global redzone​: f9\n Global init order​: f6\n Poisoned by user​: f7\n Container overflow​: fc\n Array cookie​: ac\n Intra object redzone​: bb\n ASan internal​: fe\n Left alloca redzone​: ca\n Right alloca redzone​: cb\n==1658==ABORTING"
# expected "2"
# PROG​:
# use POSIX; print &POSIX​::mblen("\N{GREEK SMALL LETTER SIGMA}", 2)
# STATUS​: 256
../ext/POSIX/t/mb.t ..
1..4
ok 1 - mblen() basically works
ok 2 - MB_CUR_MAX is at least 4 in a UTF-8 locale
not ok 3 - mblen() recognizes invalid multibyte characters
not ok 4 - mblen() works on UTF-8 characters
Failed 2/4 subtests

Test Summary Report

---

../ext/POSIX/t/mb.t (Wstat​: 0 Tests​: 4 Failed​: 2)
  Failed tests​: 3-4
Files=1, Tests=4, 0 wallclock secs ( 0.01 usr 0.01 sys + 0.17 cusr 0.07 csys = 0.26 CPU)
Result​: FAIL

[p5pRT]

From @jkeenan

Summary of my perl5 (revision 5 version 31 subversion 1) configuration​:
  Commit id​: 7c21f00
  Platform​:
  osname=linux
  osvers=4.15.0-51-generic
  archname=x86_64-linux
  uname='linux zareason 4.15.0-51-generic #55-ubuntu smp wed may 15 14​:27​:21 utc 2019 x86_64 x86_64 x86_64 gnulinux '
  config_args='-des -Dusedevel -Dcc=clang -Accflags=-Werror=declaration-after-statement -g -fno-omit-frame-pointer -fsanitize=address -fno-common -fsanitize-blacklist=/home/jkeenan/gitwork/perl/asan_ignore -Aldflags=-fsanitize=address'
  hint=recommended
  useposix=true
  d_sigaction=define
  useithreads=undef
  usemultiplicity=undef
  use64bitint=define
  use64bitall=define
  uselongdouble=undef
  usemymalloc=n
  default_inc_excludes_dot=define
  bincompat5005=undef
  Compiler​:
  cc='clang'
  ccflags ='-Werror=declaration-after-statement -g -fno-omit-frame-pointer -fsanitize=address -fno-common -fsanitize-blacklist=/home/jkeenan/gitwork/perl/asan_ignore -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2'
  optimize='-O2'
  cppflags='-Werror=declaration-after-statement -g -fno-omit-frame-pointer -fsanitize=address -fno-common -fsanitize-blacklist=/home/jkeenan/gitwork/perl/asan_ignore -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include'
  ccversion=''
  gccversion='4.2.1 Compatible Clang 6.0.0 (tags/RELEASE_600/final)'
  gccosandvers=''
  intsize=4
  longsize=8
  ptrsize=8
  doublesize=8
  byteorder=12345678
  doublekind=3
  d_longlong=define
  longlongsize=8
  d_longdbl=define
  longdblsize=16
  longdblkind=3
  ivtype='long'
  ivsize=8
  nvtype='double'
  nvsize=8
  Off_t='off_t'
  lseeksize=8
  alignbytes=8
  prototype=define
  Linker and Libraries​:
  ld='clang'
  ldflags =' -fsanitize=address -fstack-protector-strong -L/usr/local/lib'
  libpth=/usr/local/lib /usr/lib/llvm-6.0/lib/clang/6.0.0/lib /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib /lib64 /usr/lib64
  libs=-lpthread -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc
  perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
  libc=libc-2.27.so
  so=so
  useshrplib=false
  libperl=libperl.a
  gnulibc_version='2.27'
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs
  dlext=so
  d_dlsymun=undef
  ccdlflags='-Wl,-E'
  cccdlflags='-fPIC'
  lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector-strong'

Characteristics of this binary (from libperl)​:
  Compile-time options​:
  HAS_TIMES
  PERLIO_LAYERS
  PERL_COPY_ON_WRITE
  PERL_DONT_CREATE_GVSV
  PERL_MALLOC_WRAP
  PERL_OP_PARENT
  PERL_PRESERVE_IVUV
  PERL_USE_DEVEL
  USE_64_BIT_ALL
  USE_64_BIT_INT
  USE_LARGE_FILES
  USE_LOCALE
  USE_LOCALE_COLLATE
  USE_LOCALE_CTYPE
  USE_LOCALE_NUMERIC
  USE_LOCALE_TIME
  USE_PERLIO
  USE_PERL_ATOF
  Built under linux
  Compiled at Jun 8 2019 17​:09​:02
  %ENV​:
  PERL2DIR="/home/jkeenan/gitwork/perl2"
  PERLBREW_HOME="/home/jkeenan/.perlbrew"
  PERLBREW_MANPATH="/home/jkeenan/perl5/perlbrew/perls/perl-5.30.0/man"
  PERLBREW_PATH="/home/jkeenan/perl5/perlbrew/bin​:/home/jkeenan/perl5/perlbrew/perls/perl-5.30.0/bin"
  PERLBREW_PERL="perl-5.30.0"
  PERLBREW_ROOT="/home/jkeenan/perl5/perlbrew"
  PERLBREW_SHELLRC_VERSION="0.84"
  PERLBREW_VERSION="0.84"
  PERL_WORKDIR="/home/jkeenan/gitwork/perl"
  @​INC​:
  lib
  /usr/local/lib/perl5/site_perl/5.31.1/x86_64-linux
  /usr/local/lib/perl5/site_perl/5.31.1
  /usr/local/lib/perl5/5.31.1/x86_64-linux
  /usr/local/lib/perl5/5.31.1

[p5pRT]

From @jkeenan

On Sat, 08 Jun 2019 21​:43​:16 GMT, jkeenan wrote​:

Dom,

Unfortunately I have to call your attention to 2 smoke-test failures
in ext/POSIX/t/mb.t which were recorded *after* I applied your patch.

http​://perl5.test-smoke.org/report/89146
logs at​: http​://perl5.test-smoke.org/logfile/89146

http​://perl5.test-smoke.org/report/89211
logs at​: http​://perl5.test-smoke.org/logfile/89211

(These can be tracked via this search​:
http​://perl5.test-
smoke.org/submatrix?test=../ext/POSIX/t/mb.t&pversion=5.31.1)

In each case the failures in mb.t occurred when blead was configured
as follows​:

[stdio] -Dcc=clang -Accflags="-Werror=declaration-after-statement -g
-fno-omit-frame-pointer -fsanitize=address -fno-common -fsanitize-
blacklist=`pwd`/asan_ignore" -Aldflags="-fsanitize=address"

With and without -DDEBUGGING.

A couple of points​:

1. Tester is using what I would guess is an advanced version of the
Linux kernel​: 5.0.9-200 versus my own 4.15.0-51 (Ubuntu 18.04 LTS).
OTOH, we are getting smoke-test reports from rigs with even higher-
numbered Linux kernels.

2. I myself don't understand all those compiler switches the tester is
using. In particular, 'make' fails for me on FreeBSD-11.2 when I use
those compiler switches.

3. Nonetheless, when I build a perl with all those switches (except
-DDEBUGGING), I get those test same failures. See attachments.

4. When I build blead with those same compiler switches at the commit
immediately prior to the one where I applied your patch, I get a PASS.

#####
$ git show | head -1
commit fb55ce6
$ cd t;./perl harness -v ../ext/POSIX/t/mb.t; cd -

ok 1 - mblen() basically works
ok 2 - MB_CUR_MAX is at least 4 in a UTF-8 locale
ok 3 - mblen() recognizes invalid multibyte characters
ok 4 - mblen() works on UTF-8 characters
ok
All tests successful.
Files=1, Tests=4, 0 wallclock secs ( 0.02 usr 0.00 sys + 0.26 cusr
0.12 csys = 0.40 CPU)
Result​: PASS
#####

So your patch has triggered test failures, albeit under these very
obscure conditions. I'm going to revert your patch from blead and
then re-apply it in a branch so that we can continue to gather smoke-
test reports.

Thank you very much.

The smoke-test branch is​:

smoke-me/jkeenan/dom/134182-mb

--
James E Keenan (jkeenan@​cpan.org)

[p5pRT]

From @jmdh

On Sat, Jun 08, 2019 at 02​:43​:16PM -0700, James E Keenan via RT wrote​:

Dom,

Unfortunately I have to call your attention to 2 smoke-test failures in ext/POSIX/t/mb.t which were recorded *after* I applied your patch.

http​://perl5.test-smoke.org/report/89146
logs at​: http​://perl5.test-smoke.org/logfile/89146

http​://perl5.test-smoke.org/report/89211
logs at​: http​://perl5.test-smoke.org/logfile/89211

(These can be tracked via this search​:
http​://perl5.test-smoke.org/submatrix?test=../ext/POSIX/t/mb.t&pversion=5.31.1)

In each case the failures in mb.t occurred when blead was configured as follows​:

[stdio] -Dcc=clang -Accflags="-Werror=declaration-after-statement -g -fno-omit-frame-pointer -fsanitize=address -fno-common -fsanitize-blacklist=`pwd`/asan_ignore" -Aldflags="-fsanitize=address"

With and without -DDEBUGGING.

Very curious, this looks like the original bug that the test was
written for. Niko, do you understand what's happening here?

Cheers,
Dominic,

[p5pRT]

From @ntyni

On Fri, Jun 14, 2019 at 06​:21​:16PM +0100, Dominic Hargreaves wrote​:

On Sat, Jun 08, 2019 at 02​:43​:16PM -0700, James E Keenan via RT wrote​:

Unfortunately I have to call your attention to 2 smoke-test failures in ext/POSIX/t/mb.t which were recorded *after* I applied your patch.

In each case the failures in mb.t occurred when blead was configured as follows​:

[stdio] -Dcc=clang -Accflags="-Werror=declaration-after-statement -g -fno-omit-frame-pointer -fsanitize=address -fno-common -fsanitize-blacklist=`pwd`/asan_ignore" -Aldflags="-fsanitize=address"

With and without -DDEBUGGING.

Very curious, this looks like the original bug that the test was
written for. Niko, do you understand what's happening here?

It's a different thing that just happened to get triggered here; this
is with non-threaded builds for starters.

I can reproduce it on 5.30.0. It seems to be related to version strings
and LC_NUMERIC. I reduced it to this​:

$ LC_NUMERIC=C.UTF-8 ./perl -l -Ilib -e 'require 5.006;'

==21403==ERROR​: AddressSanitizer​: heap-use-after-free on address 0x602000000190 at pc 0x0000004813aa bp 0x7fff4f62ea90 sp 0x7fff4f62e230
READ of size 2 at 0x602000000190 thread T0
  #0 0x4813a9 in __interceptor_setlocale (/tmp/perl-5.30.0/perl+0x4813a9)
  #1 0x6d7feb in Perl_upg_version /tmp/perl-5.30.0/./vutil.c​:717​:17
  #2 0x6d73bf in Perl_new_version /tmp/perl-5.30.0/./vutil.c​:551​:12
  #3 0x8019a4 in S_require_version /tmp/perl-5.30.0/pp_ctl.c​:3719​:10
  #4 0x8019a4 in Perl_pp_require /tmp/perl-5.30.0/pp_ctl.c​:4345
  #5 0x725bf9 in Perl_runops_standard /tmp/perl-5.30.0/run.c​:41​:26
  #6 0x588f71 in S_run_body /tmp/perl-5.30.0/perl.c
  #7 0x588381 in perl_run /tmp/perl-5.30.0/perl.c​:2639​:2
  #8 0x516e1c in main /tmp/perl-5.30.0/perlmain.c​:127​:9
  #9 0x7f073082a09a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2409a)
  #10 0x43fc49 in _start (/tmp/perl-5.30.0/perl+0x43fc49)

0x602000000190 is located 0 bytes inside of 8-byte region [0x602000000190,0x602000000198)
freed by thread T0 here​:
  #0 0x4e7712 in __interceptor_free (/tmp/perl-5.30.0/perl+0x4e7712)
  #1 0x7f0730833963 in setlocale (/lib/x86_64-linux-gnu/libc.so.6+0x2d963)

previously allocated by thread T0 here​:
  #0 0x4e7a93 in malloc (/tmp/perl-5.30.0/perl+0x4e7a93)
  #1 0x7f073088ddb9 in __strdup (/lib/x86_64-linux-gnu/libc.so.6+0x87db9)

SUMMARY​: AddressSanitizer​: heap-use-after-free (/tmp/perl-5.30.0/perl+0x4813a9) in __interceptor_setlocale
Shadow bytes around the buggy address​:
  0x0c047fff7fe0​: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c047fff7ff0​: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c047fff8000​: fa fa 01 fa fa fa 00 02 fa fa 00 02 fa fa 00 02
  0x0c047fff8010​: fa fa 00 02 fa fa 00 02 fa fa 00 02 fa fa 06 fa
  0x0c047fff8020​: fa fa 00 02 fa fa fd fa fa fa fd fa fa fa 00 fa
=>0x0c047fff8030​: fa fa[fd]fa fa fa fd fa fa fa fd fa fa fa fd fa
  0x0c047fff8040​: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa
  0x0c047fff8050​: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa
  0x0c047fff8060​: fa fa fd fa fa fa 02 fa fa fa 00 fa fa fa 02 fa
  0x0c047fff8070​: fa fa fd fa fa fa fd fa fa fa 00 00 fa fa 00 06
  0x0c047fff8080​: fa fa 00 03 fa fa 00 03 fa fa 00 fa fa fa 00 04
Shadow byte legend (one shadow byte represents 8 application bytes)​:
  Addressable​: 00
  Partially addressable​: 01 02 03 04 05 06 07
  Heap left redzone​: fa
  Freed heap region​: fd
  Stack left redzone​: f1
  Stack mid redzone​: f2
  Stack right redzone​: f3
  Stack after return​: f5
  Stack use after scope​: f8
  Global redzone​: f9
  Global init order​: f6
  Poisoned by user​: f7
  Container overflow​: fc
  Array cookie​: ac
  Intra object redzone​: bb
  ASan internal​: fe
  Left alloca redzone​: ca
  Right alloca redzone​: cb
  Shadow gap​: cc
==21403==ABORTING

--
Niko Tyni ntyni@​debian.org

[p5pRT]

From @ntyni

On Mon, Jun 17, 2019 at 09​:47​:04AM +0300, Niko Tyni wrote​:

I can reproduce it on 5.30.0. It seems to be related to version strings
and LC_NUMERIC. I reduced it to this​:

$ LC_NUMERIC=C.UTF-8 ./perl -l -Ilib -e 'require 5.006;'

And further to this. It's not clear to me if this is a problem with asan
or the code.

$ cat t.c; clang -g -fsanitize=address t.c; ./a.out
#include <locale.h>
int main(void) {
  char *l;
  setlocale(LC_NUMERIC, "C.UTF-8");
  l = setlocale(LC_NUMERIC, NULL);
  setlocale(LC_NUMERIC, "C");
  setlocale(LC_NUMERIC, l);
}

==17625==ERROR​: AddressSanitizer​: heap-use-after-free on address 0x602000000070 at pc 0x00000045ea3a bp 0x7ffce1e85f70 sp 0x7ffce1e85710
READ of size 2 at 0x602000000070 thread T0
  #0 0x45ea39 in __interceptor_setlocale (/home/ntyni/a.out+0x45ea39)
  #1 0x4f4327 in main /home/ntyni/t.c​:7​:5
  #2 0x7fd77885209a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2409a)
  #3 0x41d2d9 in _start (/home/ntyni/a.out+0x41d2d9)

0x602000000070 is located 0 bytes inside of 8-byte region [0x602000000070,0x602000000078)
freed by thread T0 here​:
  #0 0x4c4da2 in __interceptor_free (/home/ntyni/a.out+0x4c4da2)
  #1 0x7fd77885b963 in setlocale (/lib/x86_64-linux-gnu/libc.so.6+0x2d963)

previously allocated by thread T0 here​:
  #0 0x4c5123 in malloc (/home/ntyni/a.out+0x4c5123)
  #1 0x7fd7788b5db9 in __strdup (/lib/x86_64-linux-gnu/libc.so.6+0x87db9)

SUMMARY​: AddressSanitizer​: heap-use-after-free (/home/ntyni/a.out+0x45ea39) in __interceptor_setlocale
Shadow bytes around the buggy address​:
  0x0c047fff7fb0​: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c047fff7fc0​: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c047fff7fd0​: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c047fff7fe0​: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c047fff7ff0​: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c047fff8000​: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa[fd]fa
  0x0c047fff8010​: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff8020​: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff8030​: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff8040​: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff8050​: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes)​:
  Addressable​: 00
  Partially addressable​: 01 02 03 04 05 06 07
  Heap left redzone​: fa
  Freed heap region​: fd
  Stack left redzone​: f1
  Stack mid redzone​: f2
  Stack right redzone​: f3
  Stack after return​: f5
  Stack use after scope​: f8
  Global redzone​: f9
  Global init order​: f6
  Poisoned by user​: f7
  Container overflow​: fc
  Array cookie​: ac
  Intra object redzone​: bb
  ASan internal​: fe
  Left alloca redzone​: ca
  Right alloca redzone​: cb
  Shadow gap​: cc
==17625==ABORTING

--
Niko

[p5pRT]

From @ntyni

On Mon, Jun 17, 2019 at 02​:49​:43PM +0300, Niko Tyni wrote​:

On Mon, Jun 17, 2019 at 09​:47​:04AM +0300, Niko Tyni wrote​:

I can reproduce it on 5.30.0. It seems to be related to version strings
and LC_NUMERIC. I reduced it to this​:

$ LC_NUMERIC=C.UTF-8 ./perl -l -Ilib -e 'require 5.006;'

And further to this. It's not clear to me if this is a problem with asan
or the code.

$ cat t.c; clang -g -fsanitize=address t.c; ./a.out
#include <locale.h>
int main(void) {
char *l;
setlocale(LC_NUMERIC, "C.UTF-8");
l = setlocale(LC_NUMERIC, NULL);
setlocale(LC_NUMERIC, "C");
setlocale(LC_NUMERIC, l);
}

Presumably the intervening setlocale() call clobbers the buffer
that l points to.

The attached patch to vutil.c seems to fix this issue for me,
but eyeballs appreciated of course.
--
Niko

[p5pRT]

From @ntyni

0001-Copy-setlocale-return-value-in-case-it-gets-clobbere.patch

From 2357c65fd9559dd0852d1cf3febb3a4e468151ed Mon Sep 17 00:00:00 2001
From: Niko Tyni <ntyni@debian.org>
Date: Mon, 17 Jun 2019 16:21:20 +0300
Subject: [PATCH] Copy setlocale() return value in case it gets clobbered by
 later calls

Flagged by AddressSanitizer in [perl #134182]

Quoting IEEE Std 1003.1, 2004 Edition

 https://pubs.opengroup.org/onlinepubs/009695399/functions/setlocale.html

 The string returned by setlocale() is such that a subsequent call with
 that string and its associated category shall restore that part of the
 program's locale. The application shall not modify the string returned
 which may be overwritten by a subsequent call to setlocale().

Bug: https://rt.perl.org/Public/Bug/Display.html?id=134182
---
 vutil.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/vutil.c b/vutil.c
index 236748915..6814e59b0 100644
--- a/vutil.c
+++ b/vutil.c
@@ -639,7 +639,7 @@ VER_NV:
 
             LC_NUMERIC_LOCK(0);    /* Start critical section */
 
-            locale_name_on_entry = setlocale(LC_NUMERIC, NULL);
+            locale_name_on_entry = savepv(setlocale(LC_NUMERIC, NULL));
             if (   strNE(locale_name_on_entry, "C")
                 && strNE(locale_name_on_entry, "POSIX"))
             {
@@ -647,6 +647,7 @@ VER_NV:
             }
             else {  /* This value indicates to the restore code that we didn't
                        change the locale */
+                Safefree(locale_name_on_entry);
                 locale_name_on_entry = NULL;
             }
 
@@ -715,6 +716,7 @@ VER_NV:
 
             if (locale_name_on_entry) {
                 setlocale(LC_NUMERIC, locale_name_on_entry);
+                Safefree(locale_name_on_entry);
             }
 
             LC_NUMERIC_UNLOCK;  /* End critical section */
-- 
2.20.1

[p5pRT]

From @ntyni

On Mon, Jun 17, 2019 at 07​:20​:32PM +0300, Niko Tyni wrote​:

On Mon, Jun 17, 2019 at 02​:49​:43PM +0300, Niko Tyni wrote​:

On Mon, Jun 17, 2019 at 09​:47​:04AM +0300, Niko Tyni wrote​:

#include <locale.h>
int main(void) {
char *l;
setlocale(LC_NUMERIC, "C.UTF-8");
l = setlocale(LC_NUMERIC, NULL);
setlocale(LC_NUMERIC, "C");
setlocale(LC_NUMERIC, l);
}

Presumably the intervening setlocale() call clobbers the buffer
that l points to.

The attached patch to vutil.c seems to fix this issue for me,
but eyeballs appreciated of course.

I see vutil.c comes from the version.pm distribution so I've submitted
the patch there as

  Perl/version.pm#7

--
Niko