How to run tests

[certik]

Here is how I build Enzyme. How do I execute tests?

$ cd enzyme
$ mkdir build
$ cd build
$ cmake .. 
-- The C compiler identification is GNU 7.5.0
-- The CXX compiler identification is GNU 7.5.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
LLVM_SHLIBEXT=.so
found llvm dir /home/ondrej/repos/Enzyme/enzyme/build
found llvm lit /home/ondrej/repos/Enzyme/enzyme/build
CMAKE_PREFIX_PATH /home/ondrej/repos/Enzyme/enzyme/build
-- Linker detection: GNU ld
found llvm include directory here: /usr/lib/llvm-6.0/include
found llvm definitions -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS
found llvm version 6
first llvm include directory/usr/lib/llvm-6.0/include
found enzyme sources ActiveVariable.cppEnzyme.cppEnzymeLogic.cppFunctionUtils.cppGradientUtils.cppTypeAnalysis.cppUtils.cppSCEV/ScalarEvolutionExpander.cpp
-- Configuring done
-- Generating done
-- Build files have been written to: /home/ondrej/repos/Enzyme/enzyme/build
$ make -j4
Scanning dependencies of target intrinsics_gen
[  0%] Built target intrinsics_gen
Scanning dependencies of target LLVMEnzyme-6
[ 11%] Building CXX object Enzyme/CMakeFiles/LLVMEnzyme-6.dir/ActiveVariable.cpp.o
[ 22%] Building CXX object Enzyme/CMakeFiles/LLVMEnzyme-6.dir/Enzyme.cpp.o
[ 33%] Building CXX object Enzyme/CMakeFiles/LLVMEnzyme-6.dir/EnzymeLogic.cpp.o
[ 44%] Building CXX object Enzyme/CMakeFiles/LLVMEnzyme-6.dir/FunctionUtils.cpp.o
/home/ondrej/repos/Enzyme/enzyme/Enzyme/ActiveVariable.cpp: In function ‘void addCallRemovingCycle(std::vector<llvm::CallInst*>&, llvm::CallInst*)’:
/home/ondrej/repos/Enzyme/enzyme/Enzyme/ActiveVariable.cpp:186:41: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
                 if (newtrace.size()-1-j == i) break;
                     ~~~~~~~~~~~~~~~~~~~~^~~~
[ 55%] Building CXX object Enzyme/CMakeFiles/LLVMEnzyme-6.dir/GradientUtils.cpp.o
[ 66%] Building CXX object Enzyme/CMakeFiles/LLVMEnzyme-6.dir/TypeAnalysis.cpp.o
/home/ondrej/repos/Enzyme/enzyme/Enzyme/TypeAnalysis.cpp: In member function ‘ValueData ValueData::KeepForCast(const llvm::DataLayout&, llvm::Type*, llvm::Type*) const’:
/home/ondrej/repos/Enzyme/enzyme/Enzyme/TypeAnalysis.cpp:75:50: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
         if (pair.first[0] != -1 && pair.first[0] < tosize) {
[ 77%] Building CXX object Enzyme/CMakeFiles/LLVMEnzyme-6.dir/Utils.cpp.o
[ 88%] Building CXX object Enzyme/CMakeFiles/LLVMEnzyme-6.dir/SCEV/ScalarEvolutionExpander.cpp.o
[100%] Linking CXX shared module LLVMEnzyme-6.so
[100%] Built target LLVMEnzyme-6
$ ctest 
*********************************
No test configuration file found!
*********************************
Usage

  ctest [options]

$ make check-enzyme-integration
[  0%] Built target intrinsics_gen
[ 90%] Built target LLVMEnzyme-6
Scanning dependencies of target check-enzyme-integration
[100%] Running enzyme integration tests
/bin/sh: 1: ../../: Permission denied
test/Integration/CMakeFiles/check-enzyme-integration.dir/build.make:57: recipe for target 'test/Integration/CMakeFiles/check-enzyme-integration' failed
make[3]: *** [test/Integration/CMakeFiles/check-enzyme-integration] Error 126
CMakeFiles/Makefile2:296: recipe for target 'test/Integration/CMakeFiles/check-enzyme-integration.dir/all' failed
make[2]: *** [test/Integration/CMakeFiles/check-enzyme-integration.dir/all] Error 2
CMakeFiles/Makefile2:303: recipe for target 'test/Integration/CMakeFiles/check-enzyme-integration.dir/rule' failed
make[1]: *** [test/Integration/CMakeFiles/check-enzyme-integration.dir/rule] Error 2
Makefile:203: recipe for target 'check-enzyme-integration' failed
make: *** [check-enzyme-integration] Error 2

[wsmoses]

make check-enzyme should run tests (I'll make a quick readme).

[certik]

This is what I get:

$ make check-enzyme
[  0%] Built target intrinsics_gen
[ 90%] Built target LLVMEnzyme-6
Scanning dependencies of target check-enzyme
[100%] Running enzyme regression tests
/bin/sh: 1: ../../: Permission denied
test/Enzyme/CMakeFiles/check-enzyme.dir/build.make:57: recipe for target 'test/Enzyme/CMakeFiles/check-enzyme' failed
make[3]: *** [test/Enzyme/CMakeFiles/check-enzyme] Error 126
CMakeFiles/Makefile2:269: recipe for target 'test/Enzyme/CMakeFiles/check-enzyme.dir/all' failed
make[2]: *** [test/Enzyme/CMakeFiles/check-enzyme.dir/all] Error 2
CMakeFiles/Makefile2:276: recipe for target 'test/Enzyme/CMakeFiles/check-enzyme.dir/rule' failed
make[1]: *** [test/Enzyme/CMakeFiles/check-enzyme.dir/rule] Error 2
Makefile:190: recipe for target 'check-enzyme' failed
make: *** [check-enzyme] Error 2

[certik]

But if I run things manually then it works:

ondrej@pn1707483:~/.../test/Enzyme(tx-cc)$ opt-6.0 < sin.ll -load ../../build/Enzyme/LLVMEnzyme-6.so  -enzyme -enzyme_preopt=false -O3 -S 
; ModuleID = '<stdin>'
source_filename = "<stdin>"

; Function Attrs: nounwind readnone
define double @tester(double %x) local_unnamed_addr #0 {
entry:
  %0 = tail call fast double @llvm.sin.f64(double %x)
  ret double %0
}

; Function Attrs: nounwind readnone
define double @test_derivative(double %x) local_unnamed_addr #0 {
entry:
  %0 = tail call fast double @llvm.cos.f64(double %x) #2
  ret double %0
}

; Function Attrs: nounwind readnone speculatable
declare double @llvm.cos.f64(double) #1

; Function Attrs: nounwind readnone speculatable
declare double @llvm.sin.f64(double) #1

; Function Attrs: nounwind readnone
define double @preprocess_tester(double %x) local_unnamed_addr #0 {
entry:
  %0 = tail call fast double @llvm.sin.f64(double %x)
  ret double %0
}

attributes #0 = { nounwind readnone }
attributes #1 = { nounwind readnone speculatable }
attributes #2 = { nounwind }

[wsmoses]

Can you post the output of make check-enzyme VERBOSE=1, I'm wondering what exact command is being run. Offhand, my first guess is there's a file path with a space that isn't being quoted properly.

[certik]

Sure:

$ make check-enzyme VERBOSE=1
/home/ondrej/ext/cmake-3.16.2-Linux-x86_64/bin/cmake -S/home/ondrej/repos/Enzyme/enzyme -B/home/ondrej/repos/Enzyme/enzyme/build --check-build-system CMakeFiles/Makefile.cmake 0
make -f CMakeFiles/Makefile2 check-enzyme
make[1]: Entering directory '/home/ondrej/repos/Enzyme/enzyme/build'
/home/ondrej/ext/cmake-3.16.2-Linux-x86_64/bin/cmake -S/home/ondrej/repos/Enzyme/enzyme -B/home/ondrej/repos/Enzyme/enzyme/build --check-build-system CMakeFiles/Makefile.cmake 0
/home/ondrej/ext/cmake-3.16.2-Linux-x86_64/bin/cmake -E cmake_progress_start /home/ondrej/repos/Enzyme/enzyme/build/CMakeFiles 10
make -f CMakeFiles/Makefile2 test/Enzyme/CMakeFiles/check-enzyme.dir/all
make[2]: Entering directory '/home/ondrej/repos/Enzyme/enzyme/build'
make -f CMakeFiles/intrinsics_gen.dir/build.make CMakeFiles/intrinsics_gen.dir/depend
make[3]: Entering directory '/home/ondrej/repos/Enzyme/enzyme/build'
cd /home/ondrej/repos/Enzyme/enzyme/build && /home/ondrej/ext/cmake-3.16.2-Linux-x86_64/bin/cmake -E cmake_depends "Unix Makefiles" /home/ondrej/repos/Enzyme/enzyme /home/ondrej/repos/Enzyme/enzyme /home/ondrej/repos/Enzyme/enzyme/build /home/ondrej/repos/Enzyme/enzyme/build /home/ondrej/repos/Enzyme/enzyme/build/CMakeFiles/intrinsics_gen.dir/DependInfo.cmake --color=
make[3]: Leaving directory '/home/ondrej/repos/Enzyme/enzyme/build'
make -f CMakeFiles/intrinsics_gen.dir/build.make CMakeFiles/intrinsics_gen.dir/build
make[3]: Entering directory '/home/ondrej/repos/Enzyme/enzyme/build'
make[3]: Nothing to be done for 'CMakeFiles/intrinsics_gen.dir/build'.
make[3]: Leaving directory '/home/ondrej/repos/Enzyme/enzyme/build'
[  0%] Built target intrinsics_gen
make -f Enzyme/CMakeFiles/LLVMEnzyme-6.dir/build.make Enzyme/CMakeFiles/LLVMEnzyme-6.dir/depend
make[3]: Entering directory '/home/ondrej/repos/Enzyme/enzyme/build'
cd /home/ondrej/repos/Enzyme/enzyme/build && /home/ondrej/ext/cmake-3.16.2-Linux-x86_64/bin/cmake -E cmake_depends "Unix Makefiles" /home/ondrej/repos/Enzyme/enzyme /home/ondrej/repos/Enzyme/enzyme/Enzyme /home/ondrej/repos/Enzyme/enzyme/build /home/ondrej/repos/Enzyme/enzyme/build/Enzyme /home/ondrej/repos/Enzyme/enzyme/build/Enzyme/CMakeFiles/LLVMEnzyme-6.dir/DependInfo.cmake --color=
make[3]: Leaving directory '/home/ondrej/repos/Enzyme/enzyme/build'
make -f Enzyme/CMakeFiles/LLVMEnzyme-6.dir/build.make Enzyme/CMakeFiles/LLVMEnzyme-6.dir/build
make[3]: Entering directory '/home/ondrej/repos/Enzyme/enzyme/build'
make[3]: Nothing to be done for 'Enzyme/CMakeFiles/LLVMEnzyme-6.dir/build'.
make[3]: Leaving directory '/home/ondrej/repos/Enzyme/enzyme/build'
[ 90%] Built target LLVMEnzyme-6
make -f test/Enzyme/CMakeFiles/check-enzyme.dir/build.make test/Enzyme/CMakeFiles/check-enzyme.dir/depend
make[3]: Entering directory '/home/ondrej/repos/Enzyme/enzyme/build'
cd /home/ondrej/repos/Enzyme/enzyme/build && /home/ondrej/ext/cmake-3.16.2-Linux-x86_64/bin/cmake -E cmake_depends "Unix Makefiles" /home/ondrej/repos/Enzyme/enzyme /home/ondrej/repos/Enzyme/enzyme/test/Enzyme /home/ondrej/repos/Enzyme/enzyme/build /home/ondrej/repos/Enzyme/enzyme/build/test/Enzyme /home/ondrej/repos/Enzyme/enzyme/build/test/Enzyme/CMakeFiles/check-enzyme.dir/DependInfo.cmake --color=
make[3]: Leaving directory '/home/ondrej/repos/Enzyme/enzyme/build'
make -f test/Enzyme/CMakeFiles/check-enzyme.dir/build.make test/Enzyme/CMakeFiles/check-enzyme.dir/build
make[3]: Entering directory '/home/ondrej/repos/Enzyme/enzyme/build'
[100%] Running enzyme regression tests
cd /home/ondrej/repos/Enzyme/enzyme/build/test/Enzyme && ../../ -v /home/ondrej/repos/Enzyme/enzyme/build/test/Enzyme
/bin/sh: 1: ../../: Permission denied
test/Enzyme/CMakeFiles/check-enzyme.dir/build.make:57: recipe for target 'test/Enzyme/CMakeFiles/check-enzyme' failed
make[3]: *** [test/Enzyme/CMakeFiles/check-enzyme] Error 126
make[3]: Leaving directory '/home/ondrej/repos/Enzyme/enzyme/build'
CMakeFiles/Makefile2:269: recipe for target 'test/Enzyme/CMakeFiles/check-enzyme.dir/all' failed
make[2]: *** [test/Enzyme/CMakeFiles/check-enzyme.dir/all] Error 2
make[2]: Leaving directory '/home/ondrej/repos/Enzyme/enzyme/build'
CMakeFiles/Makefile2:276: recipe for target 'test/Enzyme/CMakeFiles/check-enzyme.dir/rule' failed
make[1]: *** [test/Enzyme/CMakeFiles/check-enzyme.dir/rule] Error 2
make[1]: Leaving directory '/home/ondrej/repos/Enzyme/enzyme/build'
Makefile:190: recipe for target 'check-enzyme' failed
make: *** [check-enzyme] Error 2

[wsmoses]

Ah, it wasn't able to find llvm-lit.

Do you see a file at the following location or somewhere similar on your machine: /usr/lib/llvm-6/build/utils/lit/lit.py

If so, add this to your cmake (with the appropriate path):

-DLLVM_EXTERNAL_LIT=/usr/lib/llvm-6/build/utils/lit/lit.py

[certik]

Unfortunately I don't have any utils subdirectory nor any lit.py. I can see it is some kind of a testing tool: https://github.com/llvm-mirror/llvm/tree/2c4ca6832fa6b306ee6a7010bfb80a3f2596f824/utils/lit

[wsmoses]

Assuming you're on ubuntu, can you try installing llvm-6-tools and llvm-6-dev ?

[certik]

That worked. The packages are called llvm-6.0-tools and llvm-6.0-dev. I use Ubuntu 18.04. Here is the cmake command:

cmake -DLLVM_EXTERNAL_LIT=/usr/lib/llvm-6.0/build/utils/lit/lit.py ..

and I uploaded the test results here: https://gist.github.com/certik/f3202b6bc907481e460084cc19ff9905.

Some pass and some fail.

[wsmoses]

That makes sense, we stopped testing against LLVM 6 ("officially supporting" / testing versions 7-9) because it wasn't possible to write some tests in a way that was compatible for LLVM 6 and later versions (since there was a change to memcpy where it took an extra argument and a few other things). That said, functionally Enzyme should still work on LLVM 6 (since it was just hard to write tests that were compatible with all versions of LLVM) and skimming through your output that seems to be the case.

[certik]

I can upgrade, I only used LLVM 6 since it was already installed. I installed llvm-9-tools and llvm-9-dev and reconfigured:

$ cmake -DLLVM_EXTERNAL_LIT=/usr/lib/llvm-9/build/utils/lit/lit.py ..
-- The C compiler identification is GNU 7.5.0
-- The CXX compiler identification is GNU 7.5.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
LLVM_SHLIBEXT=.so
found llvm dir /home/ondrej/repos/Enzyme/enzyme/build
found llvm lit /usr/lib/llvm-9/build/utils/lit/lit.py
CMAKE_PREFIX_PATH /home/ondrej/repos/Enzyme/enzyme/build
CMake Error at /usr/lib/llvm-9/lib/cmake/llvm/LLVMExports.cmake:1323 (message):
  The imported target "yaml-bench" references the file

     "/usr/lib/llvm-9/bin/yaml-bench"

  but this file does not exist.  Possible reasons include:

  * The file was deleted, renamed, or moved to another location.

  * An install or uninstall procedure did not complete successfully.

  * The installation package was faulty and contained

     "/usr/lib/llvm-9/lib/cmake/llvm/LLVMExports.cmake"

  but not all the files it references.

Call Stack (most recent call first):
  /usr/lib/llvm-9/cmake/LLVMConfig.cmake:245 (include)
  CMakeLists.txt:42 (find_package)


-- Configuring incomplete, errors occurred!
See also "/home/ondrej/repos/Enzyme/enzyme/build/CMakeFiles/CMakeOutput.log".

It could be that LLVM 9 is broken in Ubuntu. So I installed LLVM 8 and reconfigured with:

cmake -DLLVM_DIR=/usr/lib/llvm-8 -DLLVM_EXTERNAL_LIT=/usr/lib/llvm-8/build/utils/lit/lit.py ..

and that seems to work. Looks like there is only one failure now: https://gist.github.com/certik/3c0482cf7886b92d293342ee41b2e82c

Update: that failure was caused by myself, I modified one of the tests to play with it. When I removed my modifications, everything passes:

Testing Time: 1.71s
  Expected Passes    : 130
  Expected Failures  : 8

[wsmoses]

Ah excellent, yeah that failure seemed odd to me, but glad to see it was resolved!

[certik]

Thanks!