Validating LLVM with random programs

The llvm-stress utility is a tool included with LLVM that is compiled as part of the normal build. It generates random LLVM Intermediate Representation programs, using information about the target. These programs won't run correctly and can't be checked for correct code generation, but can catch crashes or unsupported IR constructs.

1. Add build directory to path

    export PATH=.../NyuziToolchain/build/bin:$PATH
   

2. Generate random stream

    llvm-stress -size 1000 -o stress.ll
   

3. Run test

    llc -o - stress.ll
   

If this fails, isolate the failure using instructions in Debugging Backend Crashes with bugpoint

The following script will run this in a loop:

#!/bin/bash

while :
do
  SEED=$((65536*$RANDOM+$RANDOM))
  echo "$SEED"
  PROGLEN=$(($RANDOM % 2000))
  llvm-stress -seed=$SEED -o stress.ll -size=$PROGLEN
  if [ $? -ne 0 ]; then
    echo "llvm-stress failed"
    exit 1
  fi

  llc stress.ll
  if [ $? -eq 0 ]; then
      echo "PASS"
  else
      cp stress.ll fail$SEED.ll
  fi
done

The bugpoint program can be used in tandem with llvm-stress to find crashes by trying lots of different optimization options. Unfortunately, it currently returns an error because of a missing target library:

bugpoint -find-bugs -llc-safe stress.ll 
/Users/jeffbush/src/NyuziToolchain/build/bin/ld.lld: error: unable to find library -lm

There are a number of newer utilities for fuzzing parts of LLVM: http://llvm.org/docs/FuzzingLLVM.html