Hitting new segfaults in DynamicExpressions.jl

[MilesCranmer]

I have a integration test of Enzyme in DynamicExpressions.jl that has been working fine for the past few months but recently started hitting segfaults (without changes to DynamicExpressions itself as far as I can tell).

Here's the test:

using Pkg
pkg"add Test Preferences Enzyme DynamicExpressions"
using Test
using Preferences
using Enzyme
set_preferences!("DynamicExpressions", "instability_check" => "error")
# ^ Seems to be necessary to trigger error

using DynamicExpressions

operators = OperatorEnum(; binary_operators=(+, -, *, /), unary_operators=(cos, sin))
const eval_options = (turbo=Val(false),)
x1, x2 = (i -> Node(Float64; feature=i)).(1:2)

tree = Node(1, x1, Node(1, x2))

X = randn(3, 100);
dX = zero(X)

function f(tree, X, operators, output)
    output[] = sum(eval_tree_array(tree, X, operators; eval_options...)[1])
    return nothing
end

output = [0.0]
doutput = [1.0]

autodiff(
    Reverse,
    f,
    Const(tree),
    Duplicated(X, dX),
    Const(operators),
    Duplicated(output, doutput),
)

true_dX = cat(ones(100), -sin.(X[2, :]), zeros(100); dims=2)'
@test true_dX ≈ dX

I noticed today that I started getting segfaults:

[33909] signal (11.2): Segmentation fault: 11
in expression starting at REPL[18]:1
typekeyvalue_hash at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/jltypes.c:1631 [inlined]
lookup_typevalue at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/jltypes.c:1059
jl_inst_arg_tuple_type at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/jltypes.c:2224
arg_type_tuple at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/gf.c:2232 [inlined]
jl_lookup_generic_ at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/gf.c:3021 [inlined]
ijl_apply_generic at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/gf.c:3073
macro expansion at /Users/mcranmer/.julia/packages/DynamicExpressions/IF10i/src/Evaluate.jl:123 [inlined]
_eval_tree_array at /Users/mcranmer/.julia/packages/DispatchDoctor/VhnHE/src/stabilization.jl:306
macro expansion at /Users/mcranmer/.julia/packages/DynamicExpressions/IF10i/src/Evaluate.jl:197 [inlined]
dispatch_deg2_eval at /Users/mcranmer/.julia/packages/DynamicExpressions/IF10i/src/Evaluate.jl:162
macro expansion at /Users/mcranmer/.julia/packages/DynamicExpressions/IF10i/src/Evaluate.jl:128 [inlined]
_eval_tree_array at /Users/mcranmer/.julia/packages/DispatchDoctor/VhnHE/src/stabilization.jl:306
macro expansion at /Users/mcranmer/.julia/packages/DynamicExpressions/IF10i/src/Evaluate.jl:86 [inlined]
#eval_tree_array#2 at /Users/mcranmer/.julia/packages/DispatchDoctor/VhnHE/src/stabilization.jl:306 [inlined]
eval_tree_array at /Users/mcranmer/.julia/packages/DispatchDoctor/VhnHE/src/stabilization.jl:301 [inlined]
f at ./REPL[15]:2 [inlined]
diffejulia_f_2832wrap at ./REPL[15]:0
macro expansion at /Users/mcranmer/.julia/packages/Enzyme/Pljwm/src/compiler.jl:6630 [inlined]
enzyme_call at /Users/mcranmer/.julia/packages/Enzyme/Pljwm/src/compiler.jl:6231 [inlined]
CombinedAdjointThunk at /Users/mcranmer/.julia/packages/Enzyme/Pljwm/src/compiler.jl:6108 [inlined]
autodiff at /Users/mcranmer/.julia/packages/Enzyme/Pljwm/src/Enzyme.jl:314 [inlined]
autodiff at /Users/mcranmer/.julia/packages/Enzyme/Pljwm/src/Enzyme.jl:338 [inlined]
autodiff at /Users/mcranmer/.julia/packages/Enzyme/Pljwm/src/Enzyme.jl:323
unknown function (ip: 0x2ccc8807b)
_jl_invoke at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/gf.c:0 [inlined]
ijl_apply_generic at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/gf.c:3077
jl_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/./julia.h:1982 [inlined]
do_call at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/interpreter.c:126
eval_body at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/interpreter.c:0
jl_interpret_toplevel_thunk at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/interpreter.c:775
jl_toplevel_eval_flex at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/toplevel.c:934
jl_toplevel_eval_flex at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/toplevel.c:877
jl_toplevel_eval_flex at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/toplevel.c:877
ijl_toplevel_eval at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/toplevel.c:943 [inlined]
ijl_toplevel_eval_in at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/toplevel.c:985
eval at ./boot.jl:385 [inlined]
eval_user_input at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/REPL/src/REPL.jl:150
repl_backend_loop at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/REPL/src/REPL.jl:246
#start_repl_backend#46 at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/REPL/src/REPL.jl:231
start_repl_backend at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/REPL/src/REPL.jl:228
_jl_invoke at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/gf.c:0 [inlined]
ijl_apply_generic at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/gf.c:3077
#run_repl#59 at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/REPL/src/REPL.jl:389
run_repl at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/REPL/src/REPL.jl:375
jfptr_run_repl_91865.3 at /Users/mcranmer/.julia/juliaup/julia-1.10.4+0.aarch64.apple.darwin14/lib/julia/sys.dylib (unknown line)
_jl_invoke at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/gf.c:0 [inlined]
ijl_apply_generic at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/gf.c:3077
#1013 at ./client.jl:432
jfptr_YY.1013_82831.3 at /Users/mcranmer/.julia/juliaup/julia-1.10.4+0.aarch64.apple.darwin14/lib/julia/sys.dylib (unknown line)
_jl_invoke at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/gf.c:0 [inlined]
ijl_apply_generic at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/gf.c:3077
jl_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/./julia.h:1982 [inlined]
jl_f__call_latest at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/builtins.c:812
#invokelatest#2 at ./essentials.jl:892 [inlined]
invokelatest at ./essentials.jl:889 [inlined]
run_main_repl at ./client.jl:416
exec_options at ./client.jl:333
_start at ./client.jl:552
jfptr__start_82857.3 at /Users/mcranmer/.julia/juliaup/julia-1.10.4+0.aarch64.apple.darwin14/lib/julia/sys.dylib (unknown line)
_jl_invoke at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/gf.c:0 [inlined]
ijl_apply_generic at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/gf.c:3077
jl_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/./julia.h:1982 [inlined]
true_main at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/jlapi.c:582
jl_repl_entrypoint at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/jlapi.c:731
Allocations: 51110531 (Pool: 51038715; Big: 71816); GC: 53

Version info:

julia> versioninfo()
Julia Version 1.10.4
Commit 48d4fd48430 (2024-06-04 10:41 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: macOS (arm64-apple-darwin22.4.0)
  CPU: 8 × Apple M1 Pro
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, apple-m1)
Threads: 6 default, 0 interactive, 3 GC (on 6 virtual cores)

One explanation for the change would be DispatchDoctor's new version. However, I tried lowering that version and it still seems to segfault.

I also tried lowering Enzyme to v0.12.23 and v0.12.22 but those seem to segfault as well. Anything I could check to help debug this? Does the error message ring any bells?

[wsmoses]

Not offhand, but if you'd be able to reduce the complexity here a bit [ie reduce external dependencies] that would make it faster to see what's happening and fix

[wsmoses]

Also check 0.12.24 -- that does include a segfault fix.

[wsmoses]

It could also be worth checking if the Julia GC segfault fix resolves this [it is waiting on a release in https://github.com/JuliaLang/julia/pull/54851 fix from https://github.com/JuliaLang/julia/pull/54837 ]

[MilesCranmer]

Tried all of these, no luck:

• I can't seem to reproduce it with a smaller MWE from the code in Extremely long compilation time on recursive, branching functions (DynamicExpressions.jl) #1156; it seems to be from something else.
• I also tried the backports branch for 1.10.5. Unfortunately it doesn't solve the segfault.
• I also tried 0.12.24, unfortunately it doesn't solve it either.

[MilesCranmer]

It's weird, I can reproduce it with older versions of Enzyme too. Is there some key dependency that updated recently within the compat bounds, maybe GPUCompiler.jl?

[wsmoses]

Hm, maybe? Either way happy to try to debug [perhaps over a pair debugging call] -- but just as a forewarning things are way harder to figure out with more code/dependencies, so anything you can do to repro it in a simpler form will make it easier to fix

[wsmoses]

@MilesCranmer so I ran this locally [linux x86, 1.10] and it ran without issue.

[MilesCranmer]

Hm, weird. My GitHub actions are Linux based and run into an issue on it: https://github.com/SymbolicML/DynamicExpressions.jl/actions/runs/10035991958/job/27732912305. It’s not a segfault though:

Incorrect number of arguments passed to called function!
  %16 = call { {} addrspace(10)*, i8 } @julia____eval_tree_array_simulator_600_6574_inner.1([1 x {} addrspace(10)*]* noundef nonnull align 8 dereferenceable(40) %15, {} addrspace(10)* noundef nonnull align 16 dereferenceable(40) %2, {} addrspace(10)* noundef nonnull align 16 dereferenceable(40) %3) #115, !dbg !13425
; Function Attrs: mustprogress willreturn
define internal fastcc void @preprocess_julia__eval_tree_array_6687({ {} addrspace(10)*, i8 }* noalias nocapture nofree noundef nonnull writeonly sret({ {} addrspace(10)*, i8 }) align 8 dereferenceable(16) "enzyme_type"="{[-1]:Pointer, [-1,0]:Pointer, [-1,0,0]:Pointer, [-1,0,0,-1]:Float@double, [-1,0,8]:Integer, [-1,0,9]:Integer, [-1,0,10]:Integer, [-1,0,11]:Integer, [-1,0,12]:Integer, [-1,0,13]:Integer, [-1,0,14]:Integer, [-1,0,15]:Integer, [-1,0,16]:Integer, [-1,0,17]:Integer, [-1,0,18]:Integer, [-1,0,19]:Integer, [-1,0,20]:Integer, [-1,0,21]:Integer, [-1,0,22]:Integer, [-1,0,23]:Integer, [-1,0,24]:Integer, [-1,0,25]:Integer, [-1,0,26]:Integer, [-1,0,27]:Integer, [-1,0,28]:Integer, [-1,0,29]:Integer, [-1,0,30]:Integer, [-1,0,31]:Integer, [-1,0,32]:Integer, [-1,0,33]:Integer, [-1,0,34]:Integer, [-1,0,35]:Integer, [-1,0,36]:Integer, [-1,0,37]:Integer, [-1,0,38]:Integer, [-1,0,39]:Integer, [-1,8]:Integer}" %0, [1 x {} addrspace(10)*]* noalias nocapture nofree noundef nonnull writeonly align 8 dereferenceable(8) "enzyme_inactive" "enzyme_type"="{[-1]:Pointer}" "enzymejl_returnRoots" %1, {} addrspace(10)* noundef nonnull align 8 dereferenceable(40) "enzyme_type"="{[-1]:Pointer, [-1,0]:Integer, [-1,1]:Integer, [-1,8]:Float@double, [-1,16]:Integer, [-1,17]:Integer, [-1,18]:Integer, [-1,24]:Pointer, [-1,32]:Pointer}" "enzymejl_parmtype"="140045387451552" "enzymejl_parmtype_ref"="2" %2, {} addrspace(10)* noundef nonnull align 16 dereferenceable(40) "enzyme_type"="{[-1]:Pointer, [-1,0]:Pointer, [-1,0,-1]:Float@double, [-1,8]:Integer, [-1,9]:Integer, [-1,10]:Integer, [-1,11]:Integer, [-1,12]:Integer, [-1,13]:Integer, [-1,14]:Integer, [-1,15]:Integer, [-1,16]:Integer, [-1,17]:Integer, [-1,18]:Integer, [-1,19]:Integer, [-1,20]:Integer, [-1,21]:Integer, [-1,22]:Integer, [-1,23]:Integer, [-1,24]:Integer, [-1,25]:Integer, [-1,26]:Integer, [-1,27]:Integer, [-1,28]:Integer, [-1,29]:Integer, [-1,30]:Integer, [-1,31]:Integer, [-1,32]:Integer, [-1,33]:Integer, [-1,34]:Integer, [-1,35]:Integer, [-1,36]:Integer, [-1,37]:Integer, [-1,38]:Integer, [-1,39]:Integer}" "enzymejl_parmtype"="140045519677072" "enzymejl_parmtype_ref"="2" %3) unnamed_addr #114 !dbg !13423 {
top:
  %4 = call {}*** @julia.get_pgcstack()
  %5 = bitcast {}*** %4 to {}**
  %6 = getelementptr inbounds {}*, {}** %5, i64 -14
  %7 = getelementptr inbounds {}*, {}** %6, i64 16
  %8 = bitcast {}** %7 to i8**
  %9 = load i8*, i8** %8, align 8
  %10 = call noalias nonnull dereferenceable(8) dereferenceable_or_null(8) {} addrspace(10)* @julia.gc_alloc_obj({}** %6, i64 8, {} addrspace(10)* addrspacecast ({}* inttoptr (i64 140045377162128 to {}*) to {} addrspace(10)*)), !enzyme_fromstack !104
  call void @zeroType.58({} addrspace(10)* %10, i8 0, i64 8), !enzyme_zerostack !76
  %11 = bitcast {} addrspace(10)* %10 to [1 x {} addrspace(10)*] addrspace(10)*, !enzyme_caststack !76
  %12 = call {}*** @julia.get_pgcstack() #115
  %ptls_field3 = getelementptr inbounds {}**, {}*** %12, i64 2
  %13 = bitcast {}*** %ptls_field3 to i64***
  %ptls_load45 = load i64**, i64*** %13, align 8, !tbaa !77
  %14 = getelementptr inbounds i64*, i64** %ptls_load45, i64 2
  %safepoint = load i64*, i64** %14, align 8, !tbaa !81
  fence syncscope("singlethread") seq_cst
  call void @julia.safepoint(i64* %safepoint) #115, !dbg !13424
  fence syncscope("singlethread") seq_cst
  call fastcc void @julia__promote_op_6691() #116, !dbg !13424
  %15 = addrspacecast [1 x {} addrspace(10)*] addrspace(10)* %11 to [1 x {} addrspace(10)*]*, !dbg !13425
  %16 = call { {} addrspace(10)*, i8 } @julia____eval_tree_array_simulator_600_6574_inner.1([1 x {} addrspace(10)*]* noundef nonnull align 8 dereferenceable(40) %15, {} addrspace(10)* noundef nonnull align 16 dereferenceable(40) %2, {} addrspace(10)* noundef nonnull align 16 dereferenceable(40) %3) #115, !dbg !13425
  %.fca.0.extract = extractvalue { {} addrspace(10)*, i8 } %16, 0, !dbg !13425
  %.fca.1.extract = extractvalue { {} addrspace(10)*, i8 } %16, 1, !dbg !13425
  %17 = getelementptr inbounds [1 x {} addrspace(10)*], [1 x {} addrspace(10)*]* %1, i64 0, i64 0, !dbg !13425
  store {} addrspace(10)* %.fca.0.extract, {} addrspace(10)** %17, align 8, !dbg !13425, !noalias !13426
  %.sroa.0.0..sroa_idx = getelementptr inbounds { {} addrspace(10)*, i8 }, { {} addrspace(10)*, i8 }* %0, i64 0, i32 0, !dbg !13425
  store {} addrspace(10)* %.fca.0.extract, {} addrspace(10)** %.sroa.0.0..sroa_idx, align 8, !dbg !13425, !noalias !13426
  %.sroa.3.0..sroa_idx = getelementptr inbounds { {} addrspace(10)*, i8 }, { {} addrspace(10)*, i8 }* %0, i64 0, i32 1, !dbg !13425
  store i8 %.fca.1.extract, i8* %.sroa.3.0..sroa_idx, align 8, !dbg !13425, !noalias !13426
  ret void, !dbg !13425
}

[MilesCranmer]

Oh yeah I can also reproduce this same error on my mac, but it requires a modification to the code example above:

using Pkg: activate, add, instantiate
activate(; temp=true)
add(["Preferences", "DynamicExpressions", "Enzyme"])
instantiate()

using Preferences
set_preferences!("DynamicExpressions", "instability_check" => "error", "instability_check_codegen_level" => "min")
# ^ Seems to be necessary to trigger error

using DynamicExpressions, Enzyme

operators = OperatorEnum(; binary_operators=(+, -, *, /), unary_operators=(cos, sin))
const eval_options = (turbo=Val(false),)
x1, x2 = (i -> Node(Float64; feature=i)).(1:2)

tree = Node(1, x1, Node(1, x2))

X = randn(3, 100);
dX = zero(X)

function f(tree, X, operators, output)
    output[] = sum(eval_tree_array(tree, X, operators; eval_options...)[1])
    return nothing
end

output = [0.0]
doutput = [1.0]

autodiff(
    Reverse,
    f,
    Const(tree),
    Duplicated(X, dX),
    Const(operators),
    Duplicated(output, doutput),
)

true_dX = cat(ones(100), -sin.(X[2, :]), zeros(100); dims=2)'
@show true_dX ≈ dX

which gives me

ERROR: LoadError: LLVM error: function failed verification (1)

[wsmoses]

Okay ive figured out how to repro the first gc error.

That said the amt of code is huge so it'll take a lot of effort to diagnose, if you can minimize it -- including some of the other ones you found -- to the simplest possible

[wsmoses]

Okay I think i found the cause [and hopefully fixed] your original GC error

[wsmoses]

Fixed by #1667