panic: regcomp.c: 13346: negative offset: -68719492257

[p5pRT]

Migrated from rt.perl.org#134059 (status was 'pending release')

Searchable as RT134059$

[p5pRT]

From @dur-randir

Created by @dur-randir

While fuzzing perl v5.29.9-63-g2496d8f3f7 built with afl and run
under libdislocator, I found the following program

/0000000000000000[\N{U+0.00}0000/

to emit 'panic​: regcomp.c​: 13346​: negative offset​: -68719492257 trying
to output message for pattern ?​:\N{U+0.00}|[\N{U+0.00}0000)'
diagnostics.
This seems to be a regression in blead, but I wasn't able to bisect it.

Perl Info

Flags:
    category=core
    severity=medium

Site configuration information for perl 5.29.9:

Configured by dur-randir at Wed Feb 27 14:51:01 MSK 2019.

Summary of my perl5 (revision 5 version 29 subversion 9) configuration:
  Commit id: c1e47bad34ce1d9c84ed57c9b8978bcbd5a02e98
  Platform:
    osname=darwin
    osvers=13.4.0
    archname=darwin-thread-multi-2level
    uname='darwin isengard.local 13.4.0 darwin kernel version 13.4.0:
mon jan 11 18:17:34 pst 2016; root:xnu-2422.115.15~1release_x86_64
x86_64 '
    config_args='-de -Dusedevel -DDEBUGGING -Dusethreads'
    hint=recommended
    useposix=true
    d_sigaction=define
    useithreads=define
    usemultiplicity=define
    use64bitint=define
    use64bitall=define
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
    bincompat5005=undef
  Compiler:
    cc='cc'
    ccflags ='-fno-common -DPERL_DARWIN -mmacosx-version-min=10.9
-DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector
-I/usr/local/include -DPERL_USE_SAFE_PUTENV'
    optimize='-O3 -g'
    cppflags='-fno-common -DPERL_DARWIN -mmacosx-version-min=10.9
-DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector
-I/usr/local/include'
    ccversion=''
    gccversion='4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)'
    gccosandvers=''
    intsize=4
    longsize=8
    ptrsize=8
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=16
    longdblkind=3
    ivtype='long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='off_t'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='cc'
    ldflags =' -mmacosx-version-min=10.9 -fstack-protector -L/usr/local/lib'
    libpth=/usr/local/lib
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/6.0/lib
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib
/usr/lib
    libs=-lpthread -lgdbm -ldbm -ldl -lm -lutil -lc
    perllibs=-lpthread -ldl -lm -lutil -lc
    libc=
    so=dylib
    useshrplib=false
    libperl=libperl.a
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_dlopen.xs
    dlext=bundle
    d_dlsymun=undef
    ccdlflags=' '
    cccdlflags=' '
    lddlflags=' -mmacosx-version-min=10.9 -bundle -undefined
dynamic_lookup -L/usr/local/lib -fstack-protector'



@INC for perl 5.29.9:
    lib
    /usr/local/lib/perl5/site_perl/5.29.9/darwin-thread-multi-2level
    /usr/local/lib/perl5/site_perl/5.29.9
    /usr/local/lib/perl5/5.29.9/darwin-thread-multi-2level
    /usr/local/lib/perl5/5.29.9


Environment for perl 5.29.9:
    DYLD_LIBRARY_PATH (unset)
    HOME=/Users/dur-randir
    LANG=en_US.UTF-8
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/Users/dur-randir/perlbrew/bin:/Users/dur-randir/perlbrew/perls/perl-5.22.1/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/texbin
    PERLBREW_HOME=/Users/dur-randir/.perlbrew
    PERLBREW_MANPATH=/Users/dur-randir/perlbrew/perls/perl-5.22.1/man
    PERLBREW_PATH=/Users/dur-randir/perlbrew/bin:/Users/dur-randir/perlbrew/perls/perl-5.22.1/bin
    PERLBREW_PERL=perl-5.22.1
    PERLBREW_ROOT=/Users/dur-randir/perlbrew
    PERLBREW_SHELLRC_VERSION=0.84
    PERLBREW_VERSION=0.84
    PERL_BADLANG (unset)
    SHELL=/usr/local/bin/zsh

[p5pRT]

From @dur-randir

I was finally able to bisect this to

d62e749 is the first bad commit
commit d62e749
Author​: Karl Williamson <khw@​cpan.org>
Date​: Sun Oct 14 12​:01​:07 2018 -0600

  regcomp.c​: Add ability to not warn during substitute parse

  Under certain conditions, regcomp.c will pretend something other than
  the input pattern is to be parsed. There is a mechanism to seamlessly
  show the original code when that substitute expression contains the
  original as a subset. But there are cases where the entire substitute
  is constructed by regcomp.c, and has none of the original pattern in
  it. Since it is our construction, it should be legal, devoid of
  warnings, but if somehow something happened to generate a warning, it
  could lead to seg faults, etc.

  This commit adds and uses a mechanism to turn off warnings while parsing
  these constructs. Should a warning attempt to be output, instead of a
  seg fault, a panic error message giving debugging details is output.

The stand-alone reproducer also turned out to be much more elaborate​:

eval q!m'000000000000000000[0\N{U+00.0}0'!; warn $@​; 1 if $@​ =~ /panic/;

GDB stacktrace to the panic() point is​:

#1 0x00007ffff7c25535 in __GI_abort () at abort.c​:79
#2 0x0000555555712bf2 in Perl_vcroak (pat=0x555555a83c60 "panic​: %s​: %d​: negative offset​: %ld trying to output message for pattern %.*s",
  args=0x7fffffffb860) at util.c​:1701
#3 0x0000555555712f02 in Perl_croak (pat=0x555555a83c60 "panic​: %s​: %d​: negative offset​: %ld trying to output message for pattern %.*s") at util.c​:1750
#4 0x00005555556c3618 in S_regatom (pRExC_state=0x7fffffffd440, flagp=0x7fffffffbc04, depth=9) at regcomp.c​:13346
#5 0x00005555556bcb3a in S_regpiece (pRExC_state=0x7fffffffd440, flagp=0x7fffffffbd20, depth=8) at regcomp.c​:12457
#6 0x00005555556bc43a in S_regbranch (pRExC_state=0x7fffffffd440, flagp=0x7fffffffbdc8, first=0, depth=7) at regcomp.c​:12377
#7 0x00005555556ba180 in S_reg (pRExC_state=0x7fffffffd440, paren=58, flagp=0x7fffffffc350, depth=6) at regcomp.c​:12132
#8 0x00005555556e1d87 in S_regclass (pRExC_state=0x7fffffffd440, flagp=0x7fffffffcb04, depth=5, stop_at_1=false, allow_mutiple_chars=true,
  silence_non_portable=false, strict=false, optimizable=true, ret_invlist=0x0) at regcomp.c​:17870
#9 0x00005555556c31a9 in S_regatom (pRExC_state=0x7fffffffd440, flagp=0x7fffffffcb04, depth=4) at regcomp.c​:13332
#10 0x00005555556bcb3a in S_regpiece (pRExC_state=0x7fffffffd440, flagp=0x7fffffffcc20, depth=3) at regcomp.c​:12457
#11 0x00005555556bc43a in S_regbranch (pRExC_state=0x7fffffffd440, flagp=0x7fffffffccc8, first=1, depth=2) at regcomp.c​:12377
#12 0x00005555556b9c5c in S_reg (pRExC_state=0x7fffffffd440, paren=0, flagp=0x7fffffffd178, depth=1) at regcomp.c​:12088
#13 0x000055555569d2f2 in Perl_re_op_compile (patternp=0x0, pat_count=1, expr=0x555555b7e2c8, eng=0x555555b46d20 <PL_core_reg_engine>, old_re=0x0,
  is_bare_re=0x0, orig_rx_flags=0, pm_flags=0) at regcomp.c​:7705
#14 0x00005555555ba19a in Perl_pmruntime (o=0x555555b7e308, expr=0x555555b7e2c8, repl=0x0, flags=1, floor=0) at op.c​:7130
#15 0x0000555555670244 in Perl_yyparse (gramtype=258) at perly.y​:1234
#16 0x0000555555839ec6 in S_doeval_compile (gimme=1 '\001', outside=0x555555b538f0, seq=4294967246, hh=0x0) at pp_ctl.c​:3502
#17 0x0000555555841c61 in Perl_pp_entereval () at pp_ctl.c​:4478
#18 0x000055555570c635 in Perl_runops_debug () at dump.c​:2537
#19 0x00005555555ed63d in S_run_body (oldscope=1) at perl.c​:2716
#20 0x00005555555ecbbb in perl_run (my_perl=0x555555b51260) at perl.c​:2639
#21 0x00005555555a1181 in main (argc=2, argv=0x7fffffffe1e8, env=0x7fffffffe200) at perlmain.c​:134

[p5pRT]

From @jkeenan

On Fri, 26 Apr 2019 17​:01​:14 GMT, randir wrote​:

I was finally able to bisect this to

d62e749 is the first bad commit

Sergey, could you post how you bisected this?

Did you use Porting/bisect.pl? If so, could you post the full command you used to get the result?

If you didn't use Porting/bisect.pl, could you post what you did use?

If there's some special magic here, I think we should add it to our documentation.

Thank you very much.

--
James E Keenan (jkeenan@​cpan.org)

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @khwilliamson

On 4/23/19 8​:33 AM, Sergey Aleynikov (via RT) wrote​:

# New Ticket Created by Sergey Aleynikov
# Please include the string​: [perl #134059]
# in the subject line of all future correspondence about this issue.
# <URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=134059 >

This is a bug report for perl from sergey.aleynikov@​gmail.com,
generated with the help of perlbug 1.41 running under perl 5.29.9.

-----------------------------------------------------------------
[Please describe your issue here]

While fuzzing perl v5.29.9-63-g2496d8f3f7 built with afl and run
under libdislocator, I found the following program

/0000000000000000[\N{U+0.00}0000/

to emit 'panic​: regcomp.c​: 13346​: negative offset​: -68719492257 trying
to output message for pattern ?​:\N{U+0.00}|[\N{U+0.00}0000)'
diagnostics.
This seems to be a regression in blead, but I wasn't able to bisect it.

This is fixed by the attached patch

[p5pRT]

From @khwilliamson

0001-PATCH-perl-134059-panic-outputting-a-warning.patch

From 7f08dd4d0989eee2b2ed9af3e525206ba5632414 Mon Sep 17 00:00:00 2001
From: Karl Williamson <khw@cpan.org>
Date: Mon, 29 Apr 2019 15:24:18 -0600
Subject: [PATCH] PATCH: [perl #134059]  panic outputting a warning

This was due to a logic error on my part.  We need to save and restore a
value.  Instead, it was getting restored to the wrong value.

This particular instance of the bug was outputting a fatal error
message, so that the only harm is not giving the user the correct info,
and creating unnecessary work for them and us when it gets reported.
But this bug could manifest itself when trying to output just a warning
that the program otherwise would carry on from.
---
 regcomp.c       | 12 ++++++++++--
 t/re/reg_mesg.t |  1 +
 2 files changed, 11 insertions(+), 2 deletions(-)

diff --git a/regcomp.c b/regcomp.c
index fbd5c1809a..5029fba48e 100644
--- a/regcomp.c
+++ b/regcomp.c
@@ -131,6 +131,8 @@ struct RExC_state_t {
     char	*parse;			/* Input-scan pointer. */
     char        *copy_start;            /* start of copy of input within
                                            constructed parse string */
+    char        *save_copy_start;       /* Provides one level of saving
+                                           and restoring 'copy_start' */
     char        *copy_start_in_input;   /* Position in input string
                                            corresponding to copy_start */
     SSize_t	whilem_seen;		/* number of WHILEM in this expr */
@@ -229,6 +231,7 @@ struct RExC_state_t {
 #define RExC_precomp	(pRExC_state->precomp)
 #define RExC_copy_start_in_input (pRExC_state->copy_start_in_input)
 #define RExC_copy_start_in_constructed  (pRExC_state->copy_start)
+#define RExC_save_copy_start_in_constructed  (pRExC_state->save_copy_start)
 #define RExC_precomp_end (pRExC_state->precomp_end)
 #define RExC_rx_sv	(pRExC_state->rx_sv)
 #define RExC_rx		(pRExC_state->rx)
@@ -821,8 +824,13 @@ static const scan_data_t zero_scan_data = {
 } STMT_END
 
 /* Setting this to NULL is a signal to not output warnings */
-#define TURN_OFF_WARNINGS_IN_SUBSTITUTE_PARSE RExC_copy_start_in_constructed = NULL
-#define RESTORE_WARNINGS RExC_copy_start_in_constructed = RExC_precomp
+#define TURN_OFF_WARNINGS_IN_SUBSTITUTE_PARSE                               \
+    STMT_START {                                                            \
+      RExC_save_copy_start_in_constructed  = RExC_copy_start_in_constructed;\
+      RExC_copy_start_in_constructed = NULL;                                \
+    } STMT_END
+#define RESTORE_WARNINGS                                                    \
+    RExC_copy_start_in_constructed = RExC_save_copy_start_in_constructed 
 
 /* Since a warning can be generated multiple times as the input is reparsed, we
  * output it the first time we come to that point in the parse, but suppress it
diff --git a/t/re/reg_mesg.t b/t/re/reg_mesg.t
index c5c79f0323..d10fa2c09a 100644
--- a/t/re/reg_mesg.t
+++ b/t/re/reg_mesg.t
@@ -318,6 +318,7 @@ my @death =
  '/\p{Is_Other_Alphabetic=F}/ ' => 'Can\'t find Unicode property definition "Is_Other_Alphabetic=F" {#} m/\p{Is_Other_Alphabetic=F}{#}/',
  '/\x{100}(?(/' => 'Unknown switch condition (?(...)) {#} m/\\x{100}(?({#}/', # [perl #133896]
  '/(?[\N{KEYCAP DIGIT NINE}/' => '\N{} in inverted character class or as a range end-point is restricted to one character {#} m/(?[\\N{U+39.FE0F.20E3{#}}/', # [perl #133988]
+ '/0000000000000000[\N{U+0.00}0000/' => 'Unmatched [ {#} m/0000000000000000[{#}\N{U+0.00}0000/', # [perl #134059]
 );
 
 # These are messages that are death under 'use re "strict"', and may or may
-- 
2.17.1

[p5pRT]

From @dur-randir

On Mon, 29 Apr 2019 15​:45​:44 -0700, jkeenan wrote​:

Sergey, could you post how you bisected this?

Difficulties that I encounter with bisecting come mostly from how I generate all those test cases - the binary fuzzed is actually a modified perl. I've started with light modifications, but now they're more and more deep, so when I have a failing test case, it can "resist" bisecting in a number of ways, including​:

- it just doesn't fail on non-modified perl, which can either be an error in modifications, or those modifications add something that is required for the reproduction (for ex., a text eval block)
- it fails both before and after bisect point, but with different messages/outcomes, and trying to set failure mode based on the message fails, so, again, I just add more workarounds

So, there's no magic, just knowledge of how are those test cases generated and what can affect them.

[p5pRT]

From @khwilliamson

Fixed by

commit cc16d26
Author​: Karl Williamson <khw@​cpan.org>
Date​: Mon Apr 29 15​:24​:18 2019 -0600

  PATCH​: [perl #134059] panic outputting a warning
 
  This was due to a logic error on my part. We need to save and restore a
  value. Instead, it was getting restored to the wrong value.
 
  This particular instance of the bug was outputting a fatal error
  message, so that the only harm is not giving the user the correct info,
  and creating unnecessary work for them and us when it gets reported.
  But this bug could manifest itself when trying to output just a warning
  that the program otherwise would carry on from.
--
Karl Williamson

[p5pRT]

@khwilliamson - Status changed from 'open' to 'pending release'