COREDUMP: lexer does not restore its state correctly

[p5pRT]

Migrated from rt.perl.org#6874 (status was 'resolved')

Searchable as RT6874$

[p5pRT]

From @tobez

Hi,

The following coredumps 5.005_03, 5.6.0, and 5.6.1 on FreeBSD (perl on
FreeBSD uses system malloc)​:

  perl -e '{s//${}/; //}'

Here is the stacktrace with -DDEBUGGING (I modified 5.6.1 so that
conditionally compiled run-time realloc check coredumps instead of
croaking; that's what normal (non-debugging) perl does, anyway.

#0 0x80a437c in Perl_saferealloc (where=0x8142680, size=4294962233) at util.c​:127
127 unsigned *panic = 0; *panic=0xdeadbeef;
(gdb) bt
#0 0x80a437c in Perl_saferealloc (where=0x8142680, size=4294962233) at util.c​:127
#1 0x80c2454 in Perl_sv_grow (sv=0x814d46c, newlen=4294962233) at sv.c​:1249
#2 0x807b3e8 in S_scan_str (start=0x814200a "//}\n", keep_quoted=0, keep_delims=0) at toke.c​:6689
#3 0x8079e42 in S_scan_pat (start=0x814200a "//}\n", type=31) at toke.c​:6152
#4 0x806ee37 in Perl_yylex () at toke.c​:3589
#5 0x807d702 in Perl_yyparse () at perly.c​:1432
#6 0x805c461 in S_parse_body (env=0x0, xsinit=0x805926c <xs_init>) at perl.c​:1317
#7 0x805b8dc in perl_parse (my_perl=0x8140030, xsinit=0x805926c <xs_init>, argc=3, argv=0xbfbff4f4, env=0x0) at perl.c​:895
#8 0x805920c in main (argc=3, argv=0xbfbff4f4, env=0xbfbff504) at perlmain.c​:50
#9 0x80590d5 in _start ()
(gdb)

\Anton.

Perl Info

Flags:
    category=core
    severity=high

Site configuration information for perl v5.6.0:

Configured by markm at Mon Apr 24 11:11:27 SAST 2000.

Summary of my perl5 (revision 5.0 version 6 subversion 0) configuration:
  Platform:
    osname=freebsd, osvers=5.0-current, archname=i386-freebsd
    uname='freebsd freefall.FreeBSD.org 5.0-current freebsd 5.0-current #0: $Date$ '
    config_args='-Dprefix=/usr/local -Darchlib=/usr/libdata/perl/5.6.0/mach -Dprivlib=/usr/libdata/perl/5.6.0 -Dsitearch=/usr/local/lib/perl5/site_perl/5.6.0/mach -Dsitelib=/usr/local/lib/perl5/site_perl/5.6.0 -Dinstallsitearch=/usr/local/lib/perl5/site_perl/5.6.0/mach -Dinstallsitelib=/usr/local/lib/perl5/site_perl/5.6.0 -Dinstallbin=/usr/local/bin -Dinstallsitebin=/usr/local/bin -Dinstallscript=/usr/local/bin -Dman1dir=/usr/local/man/man1 -Dman3dir=/usr/local/lib/perl5/5.6.0/man/man3 -Duseshrplib=true -Ulocincpth= -Uloclibpth= -Dpager=/usr/bin/more -des'
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=undef use5005threads=undef useithreads=undef usemultiplicity=undef
    useperlio=undef d_sfio=undef uselargefiles=define 
    use64bitint=undef use64bitall=undef uselongdouble=undef usesocks=undef
  Compiler:
    cc='cc', optimize='undef', gccversion=2.95.2 19991024 (release)
    cppflags='-fno-strict-aliasing -DAPPLLIB_EXP="/usr/libdata/perl/BSDPAN"'
    ccflags ='-fno-strict-aliasing -DAPPLLIB_EXP="/usr/libdata/perl/BSDPAN"'
    stdchar='char', d_stdstdio=undef, usevfork=true
    intsize=4, longsize=4, ptrsize=4, doublesize=8
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=4, usemymalloc=n, prototype=define
  Linker and Libraries:
    ld='cc', ldflags ='-Wl,-E -lperl -lm '
    libpth=/usr/lib
    libs=-lm -lc -lcrypt
    libc=, so=so, useshrplib=true, libperl=libperl.so.4
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags=' -Wl,-R/usr/lib'
    cccdlflags='-DPIC -fpic', lddlflags='-Wl,-E -shared -lperl -lm '

Locally applied patches:
    SUIDMAIL - fixes for suidperl security


@INC for perl v5.6.0:
    /home/tobez/src/Prima
    /home/tobez/src/IPA
    /usr/libdata/perl/BSDPAN
    /usr/libdata/perl/5.6.0/mach
    /usr/libdata/perl/5.6.0
    /usr/local/lib/perl5/site_perl/5.6.0/mach
    /usr/local/lib/perl5/site_perl/5.6.0
    /usr/local/lib/perl5/site_perl/5.6.0
    .


Environment for perl v5.6.0:
    HOME=/home/tobez
    LANG=ru_RU.KOI8-R
    LANGUAGE (unset)
    LC_MESSAGES=ASCII
    LC_MONETARY=ASCII
    LC_NUMERIC=ASCII
    LC_TIME=ASCII
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/home/tobez/bin:/usr/local/site/bin:/usr/local/pgsql/bin:/usr/X11R6/bin
    PERL5LIB=/home/tobez/src/Prima:/home/tobez/src/IPA
    PERL_BADLANG (unset)
    SHELL=/usr/local/bin/zsh

[p5pRT]

From @floatingatoll

Linux perl (@​10209, 5.6.0)​:

syntax error at -e line 1, near "${"
syntax error at -e line 1, near "//}"
Execution of -e aborted due to compilation errors.

Linux, solaris perl (5.5.3)​:

syntax error at -e line 1, near "${"
Out of memory during ridiculously large request at -e line 1.

I don't have any FreeBSD boxen to test this on. Could someone give a try under a recent devel version? It appears to be similar to
the solaris 5.5.3 Out of memory error.

R.

[p5pRT]

From @nwc10

I don't have any FreeBSD boxen to test this on. Could someone give a try under a recent devel version? It appears to be similar to
the solaris 5.5.3 Out of memory error.

Program received signal SIGSEGV, Segmentation fault.
0x281c07d5 in isatty () from /usr/lib/libc.so.4
(gdb) where
#0 0x281c07d5 in isatty () from /usr/lib/libc.so.4
#1 0x281c0aed in isatty () from /usr/lib/libc.so.4
#2 0x281c0c80 in isatty () from /usr/lib/libc.so.4
#3 0x281c1401 in realloc () from /usr/lib/libc.so.4
#4 0x809071e in Perl_safesysrealloc ()
#5 0x80a517b in Perl_sv_grow ()
#6 0x80778f0 in S_scan_str ()
#7 0x8076913 in S_scan_pat ()
#8 0x806d762 in Perl_yylex ()
#9 0x80798ad in Perl_yyparse ()
#10 0x805f5ef in S_parse_body ()
#11 0x805ec30 in perl_parse ()
#12 0x805d1d6 in main ()
#13 0x805d0dd in _start ()

That's DEVEL10155 on FreeBSD 4.3-STABLE
I don't have a -g perl handy to go hunting this one.

Nicholas Clark

[p5pRT]

From @jhi

From libc realloc() to isatty()...recursively? I think the heap is
hosed in some grandioso way.

#3 0x281c1401 in realloc () from /usr/lib/libc.so.4
#4 0x809071e in Perl_safesysrealloc ()
#5 0x80a517b in Perl_sv_grow ()
#6 0x80778f0 in S_scan_str ()
#7 0x8076913 in S_scan_pat ()
#8 0x806d762 in Perl_yylex ()
#9 0x80798ad in Perl_yyparse ()
#10 0x805f5ef in S_parse_body ()
#11 0x805ec30 in perl_parse ()
#12 0x805d1d6 in main ()
#13 0x805d0dd in _start ()

That's DEVEL10155 on FreeBSD 4.3-STABLE
I don't have a -g perl handy to go hunting this one.

Nicholas Clark

[p5pRT]

From @nwc10

Probably. I guess I'm not awake enough to be thinking too well. Here's
a -g DEVEL10209

(gdb) where
#0 0x281fc7d5 in isatty () from /usr/lib/libc.so.4
#1 0x281fcaed in isatty () from /usr/lib/libc.so.4
#2 0x281fcc80 in isatty () from /usr/lib/libc.so.4
#3 0x281fd401 in realloc () from /usr/lib/libc.so.4
#4 0x80a5064 in Perl_safesysrealloc (where=0x815e680, size=4294961977)
  at util.c​:131
#5 0x80c2851 in Perl_sv_grow (sv=0x81691b0, newlen=4294961977) at sv.c​:1254
#6 0x807f80d in S_scan_str (start=0x815e00a "//}\n", keep_quoted=0,
  keep_delims=0) at toke.c​:6738
#7 0x807e3d2 in S_scan_pat (start=0x815e00a "//}\n", type=31) at toke.c​:6203
#8 0x8072b0e in Perl_yylex () at toke.c​:3650
#9 0x8081ed6 in Perl_yyparse () at perly.c​:1507
#10 0x80601b1 in S_parse_body (env=0x0, xsinit=0x805d3a8 <xs_init>)
  at perl.c​:1343
#11 0x805f738 in perl_parse (my_perl=0x815c030, xsinit=0x805d3a8 <xs_init>,
  argc=3, argv=0xbfbffc48, env=0x0) at perl.c​:917
#12 0x805d348 in main (argc=3, argv=0xbfbffc48, env=0xbfbffc58)
  at perlmain.c​:55
#13 0x805d23d in _start ()

4294961977 is 0xffffeb39

I need to breakpoint before the call into Perl_sv_grow, else the stack is
too hosed. Here's sv at line 6737 of toke.c

(gdb) call Perl_sv_dump(sv)
SV = PVIV(0x815d83c) at 0x81691b0
  REFCNT = 1
  FLAGS = (POK,pPOK)
  IV = 47
  PV = 0x815e680 ""\0
  CUR = 0
  LEN = 80

Not sure if this is helpful. My excuse for stopping is that it's past
midnight, and the machine I'm running the gdb on is at the wrong end of
a dialup.

Is going SEGV for wacko input a reportable bug in BSD's realloc()?

Nicholas Clark

[p5pRT]

From [Unknown Contact. See original ticket]

Nick> Is going SEGV for wacko input a reportable bug in BSD's realloc()?

Yes. However, the following patch contains a belt & braces avoidance of
the problem. Basically, the C<"${}"> construct (masquerading in the rhs
of s///) causes a syntax error which leaves a "}" for the lexer and
parser to scan once the parser gets through the error recovery on
finding '$' and HASHBRACK adjacent. Scanning that "}" causes premature
end-of-block processing, which contains a call to leave_scope(). That
blows away the value of PL_lex_state, which would have restored
PL_bufend. It restores the rest of the buffer variables, though, so (in
the case in the subect line), we resume with "; //}". The scan_str()
call which happens for the // there wants to be sure there's enough
space for the string it's going to scan, so it does an SvGROW using the
stale PL_bufend. That causes a request for somewhere around 4700
characters on my system, and the immense 'negative' request Nick traced
down on FreeBSD.

The patch adds saving PL_bufend to sublex_push(), so that future ways of
tickling premature end-of-block processing won't leave it stale. It
also changes the guessing for '{' so that "${}" and friends will be
reported as syntax errors but will have the right tokens so as not to
propagate bogus extra errors. (The second syntax error reported for the
expression in the subject line is because of the premature end-of-block
handling--the 'second' (correct) end-of-block is mis-detected as an
error since there's no longer an open block.)

  --spider

Inline Patch

--- toke.c.DIST	Sat May 26 22:42:45 2001
+++ toke.c	Mon May 28 06:14:12 2001
@@ -1043,6 +1043,7 @@
     SAVEI32(PL_lex_inwhat);
     SAVECOPLINE(PL_curcop);
     SAVEPPTR(PL_bufptr);
+    SAVEPPTR(PL_bufend);
     SAVEPPTR(PL_oldbufptr);
     SAVEPPTR(PL_oldoldbufptr);
     SAVEPPTR(PL_last_lop);
@@ -3227,8 +3228,16 @@
 		else
 		    PL_lex_brackstack[PL_lex_brackets++] = XOPERATOR;
 		s = skipspace(s);
-		if (*s == '}')
+		if (*s == '}') {
+		    if (PL_expect == XREF && PL_lex_state == LEX_INTERPNORMAL) {
+			PL_expect = XTERM;
+			/* This hack is to get the ${} in the message. */
+			PL_bufptr = s+1;
+			yyerror("syntax error");
+			break;
+		    }
 		    OPERATOR(HASHBRACK);
+		}
 		/* This hack serves to disambiguate a pair of curlies
 		 * as being a block or an anon hash.  Normally, expectation
 		 * determines that, but in cases where we're not in a

[p5pRT]

From @Tux

Sounds very familiar. Your approach sounds more generaic than I used in patch
6340, which I cannot trace on ActiveState, so I cannot reproduce. (And I realy
like that patch for my YAPC​::Europe-2.0.01 talk)