guestagent: ticker: watch sys_exit_bind with eBPF

[AkihiroSuda]

The event watcher is now triggered immediately on sys_exit_bind, not waiting for the next 3-second tick.

This commit resolves the long-standing TODO since the initial commit:

lima/cmd/lima-guestagent/daemon_linux.go

Lines 57 to 63 in 7459f45

	newTicker := func() (<-chan time.Time, func()) {
	// TODO: use an equivalent of `bpftrace -e 'tracepoint:syscalls:sys_*_bind { printf("tick\n"); }')`,
	// without depending on `bpftrace` binary.
	// The agent binary will need CAP_BPF file cap.
	ticker := time.NewTicker(tick)
	return ticker.C, ticker.Stop
	}

Close #3067
Close #3766
Close #4021

[balajiv113]

Does this work for UDP as well ??

[jandubois]

I don't have time to test right now, but I think this will not trigger when the port is closed, right?

One common problem with our polling is that test suites create a container with exposed ports, and after stopping the container create another one again in the next test that exposes the same ports. Due to the 3 second polling the old forwards may not have been removed yet, blocking the new container from exposing the same ports again.

Maybe there is another trace point to also kick the ticker, like sys_exit_close or tcp:tcp_close, if that works reliably?

[AkihiroSuda]

Does this work for UDP as well ??

Yes

sys_exit_close

Seems too frequent

$ sudo bpftrace -e 'tracepoint:syscalls:sys_exit_close { printf("%d %s %s\n", elapsed / 1000 / 1000 / 1000, probe, comm); }'
Attaching 1 probe...                                   
0 tracepoint:syscalls:sys_exit_close tmux: server      
0 tracepoint:syscalls:sys_exit_close tmux: server      
0 tracepoint:syscalls:sys_exit_close tmux: server      
1 tracepoint:syscalls:sys_exit_close tmux: server      
1 tracepoint:syscalls:sys_exit_close tmux: server      
2 tracepoint:syscalls:sys_exit_close tmux: server      
2 tracepoint:syscalls:sys_exit_close tmux: server      
2 tracepoint:syscalls:sys_exit_close containerd        
2 tracepoint:syscalls:sys_exit_close containerd        
2 tracepoint:syscalls:sys_exit_close containerd        
2 tracepoint:syscalls:sys_exit_close containerd 
...

tcp:tcp_close

Doesn't seem to exist in /sys/kernel/debug/tracing/available_events

[AkihiroSuda]

One common problem with our polling is that test suites create a container with exposed ports, and after stopping the container create another one again in the next test that exposes the same ports. Due to the 3 second polling the old forwards may not have been removed yet, blocking the new container from exposing the same ports again.

How does this happen? The guest kernel isn't aware of the usage of the host port, so it doesn't block bind() in the guest.
After bind() in the guest succeeds, the ticker event is triggered, and the hostagent updates the host port status accordingly.

[balajiv113]

LGTM 👍

Looks like i overcomplicated by thinking of getting a real event with open ports.

[jandubois]

How does this happen? The guest kernel isn't aware of the usage of the host port, so it doesn't block bind() in the guest.
After bind() in the guest succeeds, the ticker event is triggered, and the hostagent updates the host port status accordingly.

I guess the issue used to be that the test would connect to the dangling port on the host. So maybe just looking for bind events is enough; will need to write a test to verify.

[jandubois]

I haven't reviewed the code (@balajiv113 seems to have done so already), but I wrote a test (#4077) to verify it works properly, which is seems to do.

This seems like the most straightforward fix to the issue!

[norio-nomura]

After this is merged, the CPU utilization rate of lima-guestagent is always close to 90%.

I confirmed that it will be resolved with reverted.