free(): invalid pointer traceback, possibly in sourcing (.)

[rocky]

Description of problem:

When I use kshdb, I can get an invalid free() traceback and core dump.

Ksh version:

version sh (AT&T Research) 93v-971-g4784bab

Steps to reproduce:

1. Checkout kshdb:

$ git clone https://github.com/rocky/kshdb.git

2. Configure it

$ cd kshdb
$ ksh autogen.sh

3. Create a debugger startup profile:

echo "print hi there" > $HOME/.kshdbrc

4. Run the debugger on an integration test

$ ./kshdb -L . test/example/dbg-test1.sh 
ksh debugger, kshdb, release 0.07dev

Copyright 2008-2011, 2018 Rocky Bernstein
This is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.

./kshdb[88]: .[28]: source: syntax error at line 295: `<<' unexpected
(/src/external-vcs/github/rocky/kshdb/test/example/dbg-test1.sh:22):
x=22
Evaluate unrecognized commands is on.
*** Error in `/usr/local/bin/ksh': free(): invalid pointer: 0x0000000000a9e330 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7fe1f6b757e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8037a)[0x7fe1f6b7e37a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7fe1f6b8253c]
/usr/local/bin/ksh(ast_free+0x22)[0x532828]
/usr/local/bin/ksh(b_dot_cmd+0x7dc)[0x41be5c]
/usr/local/bin/ksh(sh_exec+0x200f)[0x4bd1c9]
/usr/local/bin/ksh(sh_exec+0x4ed6)[0x4c0090]
/usr/local/bin/ksh(sh_exec+0x63b2)[0x4c156c]
/usr/local/bin/ksh(sh_exec+0x4e97)[0x4c0051]
/usr/local/bin/ksh(b_dot_cmd+0x6c1)[0x41bd41]
/usr/local/bin/ksh(sh_funct+0x2a2)[0x4c42e5]
/usr/local/bin/ksh(sh_exec+0x2b0a)[0x4bdcc4]
/usr/local/bin/ksh(sh_exec+0x4e97)[0x4c0051]
/usr/local/bin/ksh(b_dot_cmd+0x6c1)[0x41bd41]
/usr/local/bin/ksh(sh_funct+0x2a2)[0x4c42e5]
/usr/local/bin/ksh(sh_exec+0x2b0a)[0x4bdcc4]
/usr/local/bin/ksh(sh_exec+0x4e97)[0x4c0051]
/usr/local/bin/ksh(sh_exec+0x637a)[0x4c1534]
/usr/local/bin/ksh(sh_exec+0x4ed6)[0x4c0090]
/usr/local/bin/ksh(sh_exec+0x637a)[0x4c1534]
/usr/local/bin/ksh(sh_exec+0x4e97)[0x4c0051]
/usr/local/bin/ksh(sh_exec+0x5e23)[0x4c0fdd]
/usr/local/bin/ksh(sh_exec+0x4e97)[0x4c0051]
/usr/local/bin/ksh(b_dot_cmd+0x6c1)[0x41bd41]
/usr/local/bin/ksh(sh_funct+0x2a2)[0x4c42e5]
/usr/local/bin/ksh(sh_exec+0x2b0a)[0x4bdcc4]
/usr/local/bin/ksh(sh_exec+0x4e97)[0x4c0051]
/usr/local/bin/ksh(sh_exec+0x5e23)[0x4c0fdd]
/usr/local/bin/ksh(sh_exec+0x4e97)[0x4c0051]
/usr/local/bin/ksh(sh_exec+0x5e23)[0x4c0fdd]
/usr/local/bin/ksh(sh_exec+0x4e97)[0x4c0051]
/usr/local/bin/ksh(sh_funscope_20120720+0xa03)[0x4c5724]
/usr/local/bin/ksh(sh_funct+0x2fd)[0x4c4340]
/usr/local/bin/ksh(sh_exec+0x2b0a)[0x4bdcc4]
/usr/local/bin/ksh(sh_exec+0x4e97)[0x4c0051]
/usr/local/bin/ksh(b_dot_cmd+0x6c1)[0x41bd41]
/usr/local/bin/ksh(sh_funct+0x2a2)[0x4c42e5]
/usr/local/bin/ksh(sh_exec+0x2b0a)[0x4bdcc4]
/usr/local/bin/ksh(sh_exec+0x4e97)[0x4c0051]
/usr/local/bin/ksh(sh_exec+0x637a)[0x4c1534]
/usr/local/bin/ksh(sh_exec+0x4e97)[0x4c0051]
/usr/local/bin/ksh(sh_funscope_20120720+0xa03)[0x4c5724]
/usr/local/bin/ksh(sh_funct+0x2fd)[0x4c4340]
/usr/local/bin/ksh(sh_exec+0x2b0a)[0x4bdcc4]
/usr/local/bin/ksh(sh_eval_20120720+0x2f4)[0x4c5e4e]
/usr/local/bin/ksh(sh_trap_20120720+0x301)[0x4572f1]
/usr/local/bin/ksh(sh_debug+0x514)[0x4b9c1b]
/usr/local/bin/ksh(sh_setlist+0x20a5)[0x484104]
/usr/local/bin/ksh(sh_exec+0x10ee)[0x4bc2a8]
/usr/local/bin/ksh(sh_exec+0x4e97)[0x4c0051]
/usr/local/bin/ksh(sh_eval_20120720+0x2f4)[0x4c5e4e]
/usr/local/bin/ksh(b_dot_cmd+0x75f)[0x41bddf]
/usr/local/bin/ksh(sh_exec+0x200f)[0x4bd1c9]
/usr/local/bin/ksh(sh_exec+0x4ed6)[0x4c0090]
/usr/local/bin/ksh(sh_eval_20120720+0x2f4)[0x4c5e4e]
/usr/local/bin/ksh(b_eval+0x145)[0x41b677]
/usr/local/bin/ksh(sh_exec+0x200f)[0x4bd1c9]
/usr/local/bin/ksh(sh_exec+0x4e97)[0x4c0051]
/usr/local/bin/ksh(sh_exec+0x5e23)[0x4c0fdd]
/usr/local/bin/ksh(exfile+0xe58)[0x4810e6]
/usr/local/bin/ksh(sh_main+0x1035)[0x480257]
/usr/local/bin/ksh(main+0x25)[0x4ac23f]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7fe1f6b1e830]
======= Memory map: ========
00400000-005c4000 r-xp 00000000 08:15 22460                              /usr/local/bin/ksh
007c3000-007d1000 r--p 001c3000 08:15 22460                              /usr/local/bin/ksh
007d1000-007d7000 rw-p 001d1000 08:15 22460                              /usr/local/bin/ksh
007d7000-007df000 rw-p 00000000 00:00 0 
00979000-00b1a000 rw-p 00000000 00:00 0                                  [heap]
7fe1f0000000-7fe1f0021000 rw-p 00000000 00:00 0 
7fe1f0021000-7fe1f4000000 ---p 00000000 00:00 0 
7fe1f5f29000-7fe1f5f3f000 r-xp 00000000 08:15 504101                     /lib/x86_64-linux-gnu/libgcc_s.so.1
7fe1f5f3f000-7fe1f613e000 ---p 00016000 08:15 504101                     /lib/x86_64-linux-gnu/libgcc_s.so.1
7fe1f613e000-7fe1f613f000 rw-p 00015000 08:15 504101                     /lib/x86_64-linux-gnu/libgcc_s.so.1
7fe1f613f000-7fe1f6afe000 r--p 00000000 08:15 7371                       /usr/lib/locale/locale-archive
7fe1f6afe000-7fe1f6cbe000 r-xp 00000000 08:15 504064                     /lib/x86_64-linux-gnu/libc-2.23.so
7fe1f6cbe000-7fe1f6ebe000 ---p 001c0000 08:15 504064                     /lib/x86_64-linux-gnu/libc-2.23.so
7fe1f6ebe000-7fe1f6ec2000 r--p 001c0000 08:15 504064                     /lib/x86_64-linux-gnu/libc-2.23.so
7fe1f6ec2000-7fe1f6ec4000 rw-p 001c4000 08:15 504064                     /lib/x86_64-linux-gnu/libc-2.23.so
7fe1f6ec4000-7fe1f6ec8000 rw-p 00000000 00:00 0 
7fe1f6ec8000-7fe1f6ecb000 r-xp 00000000 08:15 504087                     /lib/x86_64-linux-gnu/libdl-2.23.so
7fe1f6ecb000-7fe1f70ca000 ---p 00003000 08:15 504087                     /lib/x86_64-linux-gnu/libdl-2.23.so
7fe1f70ca000-7fe1f70cb000 r--p 00002000 08:15 504087                     /lib/x86_64-linux-gnu/libdl-2.23.so
7fe1f70cb000-7fe1f70cc000 rw-p 00003000 08:15 504087                     /lib/x86_64-linux-gnu/libdl-2.23.so
7fe1f70cc000-7fe1f71d4000 r-xp 00000000 08:15 504132                     /lib/x86_64-linux-gnu/libm-2.23.so
7fe1f71d4000-7fe1f73d3000 ---p 00108000 08:15 504132                     /lib/x86_64-linux-gnu/libm-2.23.so
7fe1f73d3000-7fe1f73d4000 r--p 00107000 08:15 504132                     /lib/x86_64-linux-gnu/libm-2.23.so
7fe1f73d4000-7fe1f73d5000 rw-p 00108000 08:15 504132                     /lib/x86_64-linux-gnu/libm-2.23.so
7fe1f73d5000-7fe1f73fb000 r-xp 00000000 08:15 504037                     /lib/x86_64-linux-gnu/ld-2.23.so
7fe1f75c4000-7fe1f75c8000 rw-p 00000000 00:00 0 
7fe1f75f9000-7fe1f75fa000 rw-p 00000000 00:00 0 
7fe1f75fa000-7fe1f75fb000 r--p 00025000 08:15 504037                     /lib/x86_64-linux-gnu/ld-2.23.so
7fe1f75fb000-7fe1f75fc000 rw-p 00026000 08:15 504037                     /lib/x86_64-linux-gnu/ld-2.23.so
7fe1f75fc000-7fe1f75fd000 rw-p 00000000 00:00 0 
7ffc331ec000-7ffc3320d000 rw-p 00000000 00:00 0                          [stack]
7ffc332c3000-7ffc332c6000 r--p 00000000 00:00 0                          [vvar]
7ffc332c6000-7ffc332c8000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Aborted (core dumped)

Actual results:

Expected results:

Additional info:

[siteshwar]

I tried several times to reproduce it, but could not reproduce. Can you provide us a coredump ?

[rocky]

Here is a google drive link with the core dump and the ksh binary that was run https://drive.google.com/drive/folders/1Pjw94cFMVmf6CWSrHkHk-0lTQiRyrnfh?usp=sharing

[rocky]

@siteshwar I don't know how I remembered this, but the reason you are not getting a core dump is because the bug is triggered when you source the debugger's startup file. So create in $HOME/.kshdbrc something like this:

set autoeval on
print "kshdb profile loaded"

Alternatively, you could uncomment out https://github.com/rocky/kshdb/blob/master/command/source.sh#L30-L31 and you will have a command inside the debugger called "source" which will read in debugger commands. That too will cause the core dump.

I suspect something in https://github.com/rocky/kshdb/blob/master/lib/processor.sh#L84-L116 is causing the problem.

[siteshwar]

@rocky When ksh crashes, ${_Dbg_fd[_Dbg_fd_last]} contains a very large fd number that is invalid. ksh does not detect this invalid fd and crashes in sh_redirect function. I will open a pull request to show an error on such invalid fds, but you need to fix it in your code too.

[siteshwar]

@rocky Please try my changes from #592 and let me know if I missed anything.

[rocky]

Thanks for looking into and potentially fixing.

I will try this when I get a chance, but it might be a couple of days from now. Also I will fix my buggy ksh code now that I have some inkling of what might be wrong. So thanks for pointing that out. Again that too be in a couple of days and definitely after trying the fix.

[rocky]

Yeah, I know I'm late to reply but I did try 88c4906 and that not only doesn't crash but also give useful information to find and fix my bug. Thanks!

BTW, kshdb is an excellent program for finding serious ksh problems like this one:

• it is a large ksh program which is testable,
• by necessity it does all sorts of manipulation of things that many other ksh programs don't dare to touch.
• it does something that ksh users might find helpful
• historically has turned up a number of ksh bugs
• and is buggy ;-)

(Yes, I make lots of mistakes, but then that's why I work on debuggers.)

So please before a major release of ksh, consider running ksh as a stress test program. Even better at it to one of the longer run CI tests.

[siteshwar]

@rocky Thanks for confirming the fix. I agree that we should test tools that are written in ksh. I will open an issue to follow up on it.