uretprobe latest aya version cannot pass anymore ctx.ret().unwrap to bpf_probe_read

[jornfranke]

Rust: 1.84.0-nightly
Aya-ebpf: main#240291ab8121939515275b0b41df0439761e1a3c
Context: https://github.com/ZuInnoTe/rust-ebpf-localnet-kernel-filter-study/blob/main/uprobe-libcall-filter/uprobe-libcall-filter-ebpf/src/main.rs#L110

I try essentially to do the following:

          bpf_probe_read_user(
                                output_buf.buf.as_mut_ptr() as *mut core::ffi::c_void,
                               ctx.ret().unwrap(),
                                *src_buffer_ptr,
                            ); 

This fails with

Error: the BPF_PROG_LOAD syscall failed. Verifier output: 0: R1=ctx() R10=fp0
0: (bf) r6 = r1                       ; R1=ctx() R6_w=ctx()
1: (85) call bpf_get_current_pid_tgid#14      ; R0_w=scalar()
2: (7b) *(u64 *)(r10 -16) = r0        ; R0_w=scalar(id=1) R10=fp0 fp-16_w=scalar(id=1)
3: (7b) *(u64 *)(r10 -24) = r6        ; R6_w=ctx() R10=fp0 fp-24_w=ctx()
4: (79) r8 = *(u64 *)(r6 +80)         ; R6_w=ctx() R8_w=scalar()
5: (bf) r1 = r8                       ; R1_w=scalar(id=2) R8_w=scalar(id=2)
6: (67) r1 <<= 32                     ; R1_w=scalar(smax=0x7fffffff00000000,umax=0xffffffff00000000,smin32=0,smax32=umax32=0,var_off=(0x0; 0xffffffff00000000))
7: (bf) r2 = r1                       ; R1_w=scalar(id=3,smax=0x7fffffff00000000,umax=0xffffffff00000000,smin32=0,smax32=umax32=0,var_off=(0x0; 0xffffffff00000000)) R2_w=scalar(id=3,smax=0x7fffffff00000000,umax=0xffffffff00000000,smin32=0,smax32=umax32=0,var_off=(0x0; 0xffffffff00000000))
8: (c7) r2 s>>= 32                    ; R2_w=scalar(smin=0xffffffff80000000,smax=0x7fffffff)
9: (65) if r2 s> 0x0 goto pc+7 17: R0_w=scalar(id=1) R1_w=scalar(id=3,smax=0x7fffffff00000000,umax=0xffffffff00000000,smin32=0,smax32=umax32=0,var_off=(0x0; 0xffffffff00000000)) R2_w=scalar(smin=umin=smin32=umin32=1,smax=umax=umax32=0x7fffffff,var_off=(0x0; 0x7fffffff)) R6_w=ctx() R8_w=scalar(id=2) R10=fp0 fp-16_w=scalar(id=1) fp-24_w=ctx()
17: (77) r1 >>= 32                    ; R1_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff))
18: (25) if r1 > 0x4000 goto pc+31    ; R1_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=16384,var_off=(0x0; 0x7fff))
19: (bf) r2 = r10                     ; R2_w=fp0 R10=fp0
20: (07) r2 += -16                    ; R2_w=fp-16
21: (18) r1 = 0xffff9cdbe9801000      ; R1_w=map_ptr(map=SSLREADARGSMAP,ks=8,vs=8)
23: (85) call bpf_map_lookup_elem#1   ; R0=map_value_or_null(id=4,map=SSLREADARGSMAP,ks=8,vs=8)
24: (bf) r7 = r0                      ; R0=map_value_or_null(id=4,map=SSLREADARGSMAP,ks=8,vs=8) R7_w=map_value_or_null(id=4,map=SSLREADARGSMAP,ks=8,vs=8)
25: (15) if r7 == 0x0 goto pc-16      ; R7_w=map_value(map=SSLREADARGSMAP,ks=8,vs=8)
26: (b7) r1 = 0                       ; R1_w=0
27: (63) *(u32 *)(r10 -4) = r1        ; R1_w=0 R10=fp0 fp-8=0000????
28: (bf) r2 = r10                     ; R2_w=fp0 R10=fp0
29: (07) r2 += -4                     ; R2_w=fp-4
30: (18) r1 = 0xffff9cd9ea8a0f00      ; R1_w=map_ptr(map=SSLREADDATABUF,ks=4,vs=16384)
32: (85) call bpf_map_lookup_elem#1   ; R0=map_value_or_null(id=5,map=SSLREADDATABUF,ks=4,vs=16384)
33: (15) if r0 == 0x0 goto pc-24      ; R0=map_value(map=SSLREADDATABUF,ks=4,vs=16384)
34: (79) r3 = *(u64 *)(r7 +0)         ; R3_w=scalar() R7=map_value(map=SSLREADARGSMAP,ks=8,vs=8)
35: (79) r7 = *(u64 *)(r10 -24)       ; R7_w=ctx() R10=fp0 fp-24=ctx()
36: (79) r2 = *(u64 *)(r7 +80)        ; R2_w=scalar() R7_w=ctx()
37: (bf) r1 = r0                      ; R0=map_value(map=SSLREADDATABUF,ks=4,vs=16384) R1_w=map_value(map=SSLREADDATABUF,ks=4,vs=16384)
38: (bf) r6 = r0                      ; R0=map_value(map=SSLREADDATABUF,ks=4,vs=16384) R6_w=map_value(map=SSLREADDATABUF,ks=4,vs=16384)
39: (85) call bpf_probe_read_user#112
R2 min value is negative, either use unsigned or 'var &= const'
verification time 342 usec
stack depth 24+0+0+0
processed 37 insns (limit 1000000) max_states_per_insn 0 total_states 3 peak_states 3 mark_read 2

This used to work before, but I am not sure if it is due to a Rust nightly update or an Aya update.

Interestingly enough if I hardcode the number that is in ctx.ret().unwrap() (I verified that indeed it is the same number), e.g.

          bpf_probe_read_user(
                                output_buf.buf.as_mut_ptr() as *mut core::ffi::c_void,
                               341,
                                *src_buffer_ptr,
                            ); 

then the program runs without issues.
Any ideas what could be the reason?

[tamird]

Your panic handler is a lie:
https://github.com/ZuInnoTe/rust-ebpf-localnet-kernel-filter-study/blob/3d33b2e7237786c494db6c30fd0307bcaf10a054/uprobe-libcall-filter/uprobe-libcall-filter-ebpf/src/main.rs#L210
Try making it an infinite loop so the verifier gives you a more actionable error. We've made that change in the template recently and long ago in the main repo (in e621a09).

My guess is that your ctx.ret().unwrap() is unsafe and the verifier complains about the possible nonsense value (since your panic handler is no-op).

[jornfranke]

Thanks for the hint on the panic handler (I indeed took it long time ago from a example in the repository and did not think about it much). I will update it to get something more meaningful!
I checked the content of ctx.ret().unwrap() by logging it via warn!(..) (maybe not the best approach) and the value is correct and makes sense. As I said before if I hardcode the very same value then there is no issue.
And it used to work before (I admit though - I changed to many parts to be sure about the root cause, but this is just a self-study thing and nothing production grade).
However, I speculate now, will investigate with the panic handler and let you know.

[jornfranke]

Hmm when changing the panic handler to an infinite loop none of the probes works (even those that were working before)

#[panic_handler]
fn panic(_info: &core::panic::PanicInfo) -> ! {
    loop {}
}

2024-10-19T18:03:50Z INFO  uprobe_libcall_filter_app] Configuring operation filter
[2024-10-19T18:03:50Z INFO  uprobe_libcall_filter_app] Configuring application: openssl3.1
[2024-10-19T18:03:50Z INFO  uprobe_libcall_filter_app] Configuring openssl_lib: /lib64/libssl.so.3.1.4
Error: the BPF_PROG_LOAD syscall failed. Verifier output: last insn is not an exit or jmp
verification time 4 usec
stack depth 0+0+0+0+0
processed 0 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0


Caused by:
    Invalid argument (os error 22)

I took the panic handler btw. from https://github.com/aya-rs/book/blob/main/examples/kprobetcp/kprobetcp-ebpf/src/main.rs#L68

[jornfranke]

Ok by using some debug logging I can confirm this is really related to

 bpf_probe_read(
                                output_buf.buf.as_mut_ptr() as *mut core::ffi::c_void,
                                (&ret_value_len).clone().try_into().unwrap(),
                                *src_buffer_ptr,
                            );

if I do not provide a variable, but a hardcoded value it works

  bpf_probe_read_user(
                                output_buf.buf.as_mut_ptr() as *mut core::ffi::c_void,
                               341,
                                *src_buffer_ptr,
                            ); 

I am really not sure why this happens. Any idea would be great. Will try to update aya again (just did it 2 days ago) - based on the latest in main.

[tamird]

just don't unwrap.

[tamird]

I sent aya-rs/book#173 so other people don't use nonsense panic handlers.

[jornfranke]

Thanks. I tried

    // get return value (is the length of data read)
    let ret_value_len: i32 = match ctx.ret() {
        Some(ret) => ret,
        None => return 0
    };

Still the same error. However, I may lack some knowledge here to understand the issue. It used to work on a previous aya version. It works with local variables which is strange ret_value_len is a local variable. I can print it out via warn!() and it contains the correct value.
Only when handing it over to bpf_probe_read_user I get this strange error.
I tried also now a couple of older rust nightly versions no success.
Will look further .

Kernel version is 6.11.3-1

[alessandrod]

No unwrapping/panicing in ebpf code. You must handle all errors.