Netstacklat: Evaluation porting Simon's netstacklat to ebpf_exporter

[netoptimizer]

Created this PR so @simosund can see the changes and evaluation numbers from prod experiment.

Based on code from PR xdp-project/bpf-examples#129

[simosund]

Changing the filter_cgroup() to use the cgroup_id_map type hash.
Surprisingly this type seems to be faster than type cgrp_storage.

name netstacklat_tcp_recv_timestamp

• run_time_ns 10705407914 run_cnt 41576592 = 257.48 ns
• diff: 325.95 - 257.48 = 68.47 ns better

name netstacklat_skb_consume_udp

• run_time_ns 3716653454 run_cnt 8499677 = 437.27 ns
• diff: 496.10 - 437.27 = 58.83 ns better

On this AMD CPU with SRSO enabled, we have extra overheads on BPF helper calls.
The filter_cgroup() for type cgrp_storage has two extra helper calls,
bpf_get_current_task_btf() and bpf_cgrp_storage_get(), but we eliminated the
bpf_get_current_cgroup_id() helper call. Still this doesn't fully account for
diff. That said, if a BPF-prog already have the struct_task available then the
bpf_get_current_task_btf() can also be eliminated, so type cgrp_storage might
still be useful in that case

Hmm, that the cgroup storage was slower was a bit unexpected.

And shouldn't it even be the same number of helper calls? You essentially trade bpf_get_current_cgroup_id() for bpf_get_current_task() and bpf_map_lookup_elem() for bpf_cgrp_storage_get().

[simosund]

One thought here. In the example cgroup_id_map_cgrp_storage.bpf.c that PR cloudflare#565 added, they implement their lookup/check slightly differently from you.

Instead of just checking if the cgrp storage exists for the given cgroup, in the example they zero-initialize it if it doesn't exist, and then check if the content is non-zero. To me that seems like it should be more work than just checking for existence as you do. However, if a cgrp lookup miss takes significantly longer than a lookup hit for some reason, perhaps the way they do it in the example makes sense. Might be worth trying at least.

[netoptimizer]

I tried it again and benchmarked this on production workload.

1. First line is: Type: hash = 183 ns
2. Second line is: Type: cgrp_storage = 240 ns with zero as flags.
3. Last line is: Type: cgrp_storage = 248 ns with BPF_LOCAL_STORAGE_GET_F_CREATE (and comparing value 0 vs 1)

Conclusion is clear: the cgrp_storage type is slower.

[simosund]

That cgrp storage was not only slower, but that much slower (~60ns), is a bit surprising (and disappointing). But thanks for doing the tests, then we at least know!

See that you have it running with some dashboard at least =)