ICE: `Double free or corruption (out)`

[straight-shoota]

I broke the compiler with this program:

module Syscall
  ::Syscall.def_syscall mmap, Pointer(Void), addr : Void,
end
Syscall.mmap nil, 1

Compiler output:

$ crystal build .test/double-free.cr --stats
Parse:                             00:00:00.000102989 (   1.02MB)
Semantic (top level):              00:00:00.505966427 (  91.32MB)
Semantic (new):                    00:00:00.001037006 (  91.32MB)
Semantic (type declarations):      00:00:00.019574009 (  91.32MB)
Semantic (abstract def check):     00:00:00.011003474 ( 107.32MB)
Semantic (restrictions augmenter): 00:00:00.008279029 ( 107.32MB)
Semantic (ivars initializers):     00:00:00.019763257 ( 107.32MB)
Semantic (cvars initializers):     00:00:00.104154542 ( 139.32MB)
Semantic (main):                   00:00:00.277608783 ( 139.38MB)
Semantic (cleanup):                00:00:00.000291553 ( 139.38MB)
Semantic (recursive struct check): 00:00:00.000741830 ( 139.38MB)
Codegen (crystal):                 00:00:00.247681350 ( 139.38MB)
double free or corruption (out)
Codegen (bc+obj):                  00:00:00.041336891 ( 139.38MB)
/home/linuxbrew/.linuxbrew/bin/ld: cannot find S-yscall.o: No such file or directory
/home/linuxbrew/.linuxbrew/bin/ld: cannot find C-rystal5858O-nceS-tate.o: No such file or directory
/home/linuxbrew/.linuxbrew/bin/ld: cannot find A-rray40P-ointer40B-ool4141.o: No such file or directory
collect2: error: ld returned 1 exit status
Error: execution of command failed with exit status 1: cc "${@}" -o /home/johannes/src/crystal-lang/crystal/double-free  -rdynamic -L/home/linuxbrew/.linuxbrew/Cellar/crystal/1.10.1/bin/../../../../lib -lpcre2-8 -lgc -lpthread -L/home/linuxbrew/.linuxbrew/Cellar/libevent/2.1.12_1/lib -levent -lrt -lpthread -ldl

There are a couple of weird things:

• First of all, there's apparently a broken pointer, looks like somewhere in the LLVM code
• The compiler doesn't seem to recognize this and continues trying to link object files that weren't created due to the error.
• Sometimes the error message is actually free(): invalid pointer, but I figure that's probably the same error just manifesting in a different way...

An error is of course on my side using Void that way. The parameter type should actually be Pointer(Void) of course. But the compiler shoud not crash like that.

[HertzDevil]

The raw assembly:

ret = uninitialized Pointer(Void)

asm("syscall" : "={rax}"(ret)
              : "{rax}"(9), "{rdi}"(nil)
              : "rcx", "r11", "memory"
              : "volatile")

A shorter one:

asm("nop" :: "r"(nil))

Some other argument expressions produce module validation errors. Crystal is quite unrestricted here, we should probably restrict the arguments to some primitive numbers and pointers.

The following LLVM IR alone also crashes:

; llvm-as test.ll & llc test.bc

%Nil = type {}

define void @foo() {
entry:
  call void asm "nop", "r"(%Nil zeroinitializer)
  ret void
}

This crash is tracked in llvm/llvm-project#36972.