retdec-bin2llvmir segfaults due to an infinite recursion in retdec::rtti_finder::parseGccRtti()

[pmckeon]

alCloning.zip

Getting the following error when trying to disassemble a powerpc binary:

##### Checking if file is a Mach-O Universal static library...

##### Checking if file is an archive...
RUN: /home/peter/projects/retdec-install/bin/retdec-ar-extractor /home/peter/projects/retdec-install/bin/alCloning --arch-magic
Not an archive, going to the next step.

##### Gathering file information...
RUN: /home/peter/projects/retdec-install/bin/retdec-fileinfo -c /home/peter/projects/retdec-install/bin/alCloning.c.json --similarity /home/peter/projects/retdec-install/bin/alCloning --no-hashes=all --crypto /home/peter/projects/retdec-install/bin/../share/retdec/support/generic/yara_patterns/signsrch/signsrch.yara --max-memory-half-ram
Input file               : /home/peter/projects/retdec-install/bin/alCloning
File format              : ELF
File class               : 32-bit
File type                : Executable file
Architecture             : PowerPC
Endianness               : Big endian
Entry point address      : 0x10003f10
Entry point offset       : 0x3f10
Entry point section name : .text
Entry point section index: 11
Bytes on entry point     : 7c290b7854210036380000009421fff07c0803a6900100003d00100685a8da88480590509421ffe07c0802a6429f000593c1
Detected tool            : GCC (4.3.2) (compiler), DWARF heuristic
Detected tool            : GCC (4.5.x) (compiler), 100 from 147 significant nibbles (68.0272%)
Original language        : C, C++
Warning: Unknown note type found.

##### Trying to unpack /home/peter/projects/retdec-install/bin/alCloning into /home/peter/projects/retdec-install/bin/alCloning-unpacked.tmp by using generic unpacker...
RUN: /home/peter/projects/retdec-install/bin/retdec-unpacker /home/peter/projects/retdec-install/bin/alCloning -o /home/peter/projects/retdec-install/bin/alCloning-unpacked.tmp --max-memory-half-ram
No matching plugins found for 'GCC 4.3.2'
No matching plugins found for 'GCC 4.5.x'
##### Unpacking by using generic unpacker: nothing to do

##### Trying to unpack /home/peter/projects/retdec-install/bin/alCloning into /home/peter/projects/retdec-install/bin/alCloning-unpacked.tmp by using UPX...
RUN: upx -d /home/peter/projects/retdec-install/bin/alCloning -o /home/peter/projects/retdec-install/bin/alCloning-unpacked.tmp
upx: /home/peter/projects/retdec-install/bin/alCloning: NotPackedException: not packed by UPX
##### Unpacking by using UPX: nothing to do

##### Decompiling /home/peter/projects/retdec-install/bin/alCloning into /home/peter/projects/retdec-install/bin/alCloning.c.backend.bc...
RUN: /home/peter/projects/retdec-install/bin/retdec-bin2llvmir -provider-init -decoder -verify -x87-fpu -main-detection -idioms-libgcc -inst-opt -cond-branch-opt -syscalls -stack -constants -param-return -local-vars -inst-opt -simple-types -generate-dsm -remove-asm-instrs -class-hierarchy -select-fncs -unreachable-funcs -inst-opt -value-protect -instcombine -tbaa -targetlibinfo -basicaa -domtree -simplifycfg -domtree -early-cse -lower-expect -targetlibinfo -tbaa -basicaa -globalopt -mem2reg -instcombine -simplifycfg -basiccg -domtree -early-cse -lazy-value-info -jump-threading -correlated-propagation -simplifycfg -instcombine -simplifycfg -reassociate -domtree -loops -loop-simplify -lcssa -loop-rotate -licm -lcssa -instcombine -scalar-evolution -loop-simplifycfg -loop-simplify -aa -loop-accesses -loop-load-elim -lcssa -indvars -loop-idiom -loop-deletion -memdep -gvn -memdep -sccp -instcombine -lazy-value-info -jump-threading -correlated-propagation -domtree -memdep -dse -dce -bdce -adce -die -simplifycfg -instcombine -strip-dead-prototypes -globaldce -constmerge -constprop -instnamer -domtree -instcombine -instcombine -tbaa -targetlibinfo -basicaa -domtree -simplifycfg -domtree -early-cse -lower-expect -targetlibinfo -tbaa -basicaa -globalopt -mem2reg -instcombine -simplifycfg -basiccg -domtree -early-cse -lazy-value-info -jump-threading -correlated-propagation -simplifycfg -instcombine -simplifycfg -reassociate -domtree -loops -loop-simplify -lcssa -loop-rotate -licm -lcssa -instcombine -scalar-evolution -loop-simplifycfg -loop-simplify -aa -loop-accesses -loop-load-elim -lcssa -indvars -loop-idiom -loop-deletion -memdep -gvn -memdep -sccp -instcombine -lazy-value-info -jump-threading -correlated-propagation -domtree -memdep -dse -dce -bdce -adce -die -simplifycfg -instcombine -strip-dead-prototypes -globaldce -constmerge -constprop -instnamer -domtree -instcombine -simple-types -stack-ptr-op-remove -inst-opt -idioms -global-to-local -dead-global-assign -instcombine -phi2seq -value-protect -disable-inlining -disable-simplify-libcalls -config-path /home/peter/projects/retdec-install/bin/alCloning.c.json -max-memory-half-ram -o /home/peter/projects/retdec-install/bin/alCloning.c.backend.bc
Running phase: Initialization ( 0.00s )
Running phase: LLVM ( 0.01s )
Running phase: Providers initialization ( 0.01s )
Error: Decompilation to LLVM IR failed

Then the following if I run through gdb:

Program received signal SIGSEGV, Segmentation fault.
0x0000555555a28f52 in retdec::loader::SegmentDataSource::loadData(unsigned long, unsigned long, std::vector<unsigned char, std::allocator<unsigned char> >&) const ()

[s3rvac]

Thank you for the report. I was able to reproduce the issue. RetDec fails inside retdec-bin2llvmir because of a stack overflow due to infinite recursion in retdec::rtti_finder::parseGccRtti() in src/rtti-finder/rtti/rtti_gcc_parser.cpp. Line 126 in the following piece of code is executed over and over, leading to stack-space exhaustion:

25 std::shared_ptr<retdec::rtti_finder::ClassTypeInfo> retdec::rtti_finder::parseGccRtti(
...
123     if (baseAddr.isDefined() && img->isPointer(addrOfBaseAddr)
124             && baseAddr != rttiAddr)
125     {
126         baseRtti = parseGccRtti(img, rttis, baseAddr);

@PeterMatula, can you take a look at this?

[PeterMatula]

C++ related structures in this binary are a bit different than our parser (rtti-finder library) expected them to be. What happened is that we were parsing one possible RTTI entry, which referenced other possible RTTI entry as its base class, which then referenced the first entry again - infinite recursion. This should not be possible in C++ classes. I added a safeguard against this, so it does not crash.

However, there is something going on that we clearly can not handle. I'm not familiar enough with all the structures compiler generates to support C++, but this is worth a look, because right now we ignore most of it and it could be a good source of info. I created another issues (#410) based on the sample from here.