compiles with llvm-16

[stepasite]

• port to llvm-16 (not backward-compatible with older llvms), WIP

• wip, but probably not correct on may places

• rule out old PM on llvm 16 and above

• #if LLVM_VERSION_MAJOR >= 16, WIP

• #if LLVM_VERSION_MAJOR >= 16, WIP

• #if LLVM_VERSION_MAJOR >= 16, WIP

• #if LLVM_VERSION_MAJOR >= 16, WIP

• #if LLVM_VERSION_MAJOR >= 16, WIP

• #if LLVM_VERSION_MAJOR >= 16

[wsmoses]

Forward mode has new pass manager tests (in addition to old pm ones). Inside the llvm lit config, at or above LLVM 16 can we get away with setting the loadEnzyme equivalent pieces to use the new pm (we do this for clangEnzyme tests)?

If not, we can version lock all the test on old pm / new pm (e.g. if [%llvmver < 16 ] oldpm lit config;

[stepasite]

Forward mode has new pass manager tests (in addition to old pm ones). Inside the llvm lit config, at or above LLVM 16 can we get away with setting the loadEnzyme equivalent pieces to use the new pm (we do this for clangEnzyme tests)?

If not, we can version lock all the test on old pm / new pm (e.g. if [%llvmver < 16 ] oldpm lit config;

unable to grasp your idea/question, sorry

[wsmoses]

Forward mode has new pass manager tests (in addition to old pm ones). Inside the llvm lit config, at or above LLVM 16 can we get away with setting the loadEnzyme equivalent pieces to use the new pm (we do this for clangEnzyme tests)?
If not, we can version lock all the test on old pm / new pm (e.g. if [%llvmver < 16 ] oldpm lit config;

unable to grasp your idea/question, sorry

We can potentially version gate the old/new pass manager pipelines:

Enzyme/enzyme/test/Integration/ReverseMode/omp.c

Line 2 in 2571534

// RUN: if [ %llvmver -ge 9 ]; then %clang -fopenmp -std=c11 %O0TBAA -fno-vectorize -fno-unroll-loops %s -S -emit-llvm -o - | %opt - %loadEnzyme -enzyme -S | %clang -fopenmp -x ir - -o %s.out && %s.out; fi

Addtionally, forward mode LLVM tests already has both new and old PMs. https://github.com/EnzymeAD/Enzyme/blob/25715346011f7fb8953eb38d3bf1307a2088b256/enzyme/test/Enzyme/ForwardMode/add.ll

The reverse mode LLVM tests will require a bit more work to set up, though. But getting the LLVM tests running can be a separate PR.

After #1104 lands can we try to enable C++ integration test CI (which should already be set up for the new pm infrastructure).

[stepasite]

OK, thanks. Hopefully got the first idea to disable old PM tests with llvm 16.

Here:

https://github.com/EnzymeAD/Enzyme/blob/25715346011f7fb8953eb38d3bf1307a2088b256/enzyme/test/Enzyme/ForwardMode/add.ll

I found this ; RUN: %opt < %s %newLoadEnzyme -passes="enzyme" -enzyme-preopt=false -S | FileCheck %s but not sure about %newLoadEnzyme. Using -load=LLVMEnzyme-16.so simply does not work and I am getting error: opt: unknown pass name 'enzyme'

[wsmoses]

See

Enzyme/enzyme/test/lit.site.cfg.py.in

Line 55 in 0edca62

+ ' -load-pass-plugin=@ENZYME_BINARY_DIR@/Enzyme/LLVMEnzyme-' + config.llvm_ver + config.llvm_shlib_ext

[stepasite]

See

Enzyme/enzyme/test/lit.site.cfg.py.in

Line 55 in 0edca62

+ ' -load-pass-plugin=@ENZYME_BINARY_DIR@/Enzyme/LLVMEnzyme-' + config.llvm_ver + config.llvm_shlib_ext

Thanks, I will give it a try.

Anyway, I conditioned all the tests where an alternative test invoked via new pm exists.

If you are ok with it, I can add the same condition also to tests where only old pm is used.

[wsmoses]

That would work. However, if it's not too much, we really should add a new pass manager line to the other tests.

[wsmoses]

Also we should add LLVM16 to the CI matrix here https://github.com/EnzymeAD/Enzyme/blob/main/.github/workflows/ccpp.yml and https://github.com/EnzymeAD/Enzyme/blob/main/.github/workflows/enzyme-ci.yml

[stepasite]

That would work. However, if it's not too much, we really should add a new pass manager line to the other tests.

OK, I see. But before doing this, I would like to find out why I get zeros instead of getting proper derivatives. No issues with llvm-15, but problems arise with llvm-16 build; there is something wrong within my changes.

[stepasite]

@wsmoses Before pushing it into this PR, could you please check the PR into my main?

stepasite#11

I am enabling llvm-16 in gha. Also trying to format code with the following:

docker run -it --rm --workdir /Enzyme -v $(pwd):/Enzyme clang-format-lint --clang-format-executable /clang-format/clang-format11 -r --exclude ./Enzyme/SCEV --style llvm --inplace true ./Enzyme

The result of formatting step is strange as there are reformatted parts in Enzyme.cpp that I haven't touched at all. Any idea what is going on?

[wsmoses]

Something odd is still hapenning int he PR since per CI above, the earlier version tests are failing.

[stepasite]

Something odd is still hapenning int he PR since per CI above, the earlier version tests are failing.

Yes, there is something wrong. Hopefully, I will have enough time to go through the changes through the weekend.

[stepasite]

@wsmoses I am getting unknown function pass 'simplifycfg' and unknown function pass 'inline' with the new pm when passing those phases into function() option. Could you please advise?

[wsmoses]

Inlining is a module level pass so put it on its own outside the function.

[wsmoses]

Apparently simplifycfg was spelled "simplifycfg" in the new PM on LLVM 11 +, and "simplify-cfg" before.

Perhaps for ease, we add that pass name as a custom replacement rule in here

Enzyme/enzyme/test/lit.site.cfg.py.in

Line 53 in a72f89e

config.substitutions.append(('%newLoadEnzyme', ''

?

[stepasite]

/usr/lib/llvm-15/bin/opt: unknown function pass 'early-cse-memssa'
/usr/lib/llvm-15/bin/opt: unknown function pass 'inline'

I am wondering what is going on, both passes should exist, any idea @wsmoses ?

[wsmoses]

@jdoerfert @tgymnich can you assist @stepasite ?

[wsmoses]

Finished this up and merged in #1207, thanks for your help!!

[stepasite]

Finished this up and merged in #1207, thanks for your help!!

Great to see llvm 16 is supported. Anyway, looking at all the changes you had to do, the cake was too big for my first bite. Thanks.