new(ci): scheduled bpfcompat distro-kernel compatibility matrix for modern_bpf

[ErenAri]

What type of PR is this?

/kind feature

Any specific area of the project related to this PR?

/area automation

Does this PR require a change in the driver versions?

No.

What this PR does / why we need it:

Adds a non-blocking, scheduled (weekly + manually dispatchable) workflow that builds bpf_probe.o from this tree and load/attach-validates it inside disposable QEMU/KVM VMs running unmodified vendor distro cloud images, using bpfcompat (Apache-2.0). The job summary shows a per-kernel pass/fail matrix with structured failure reasons — including which program variant each kernel selects (recvmmsg_x/bpf_loop vs recvmmsg_old_x, iterator support) — plus a JSON/Markdown report artifact.

The manifest in test/bpfcompat/ mirrors the libpman loader contract and stays owned by maintainers here:

• runtime-sized maps (maps.c/ringbuffer.c)
• bpf_loop helper-gated program variants (exit_event_progs_table)
• trial-probed BPF iterators with the statically-referenced prog-array dispatch programs kept autoloaded (support_probing.c)

Green end-to-end run on a stock ubuntu-latest runner, 3m18s including the libs build, image downloads, and 3 VM validations: https://github.com/ErenAri/libs/actions/runs/27374069013

Scope: complements drivers_ci / bpfvalidator / kernel-testing — load/attach evidence per kernel, not event-capture testing.

Which issue(s) this PR fixes:

Fixes #3023

Special notes for your reviewer:

• Opened as a draft to serve as a discussion starting point, as suggested by @FedeDP on cncf-falco-dev. Happy to change the matrix, schedule, file locations, or approach — or close if existing coverage is felt sufficient.
• The scheduled trigger only takes effect once merged to the default branch; until then the linked fork run is the reference. workflow_dispatch allows on-demand runs.
• The matrix is intentionally small (Ubuntu 22.04, Debian 12, Ubuntu 24.04, ~600 MB cloud image each, downloaded per run). It can grow — the catalog also covers Amazon Linux 2/2023, RHEL, Oracle UEK, openSUSE, and lockdown profiles (image pipeline: vendor images only, sha256-recorded/pinnable).
• Supply chain: the action is pinned by tag (can pin by SHA if preferred) and downloads checksum-verified, cosign-signed release binaries; validation runs entirely inside throwaway VMs.

Does this PR introduce a user-facing change?:

NONE

[poiana]

Welcome @ErenAri! It looks like this is your first PR to falcosecurity/libs 🎉

[github-actions]

Perf diff from master - unit tests

    20.76%     +1.01%  [.] std::__shared_ptr<sinsp_threadinfo, (__gnu_cxx::_Lock_policy)2>::__shared_ptr(std::__weak_ptr<sinsp_threadinfo, (__gnu_cxx::_Lock_policy)2> const&, std::nothrow_t)
    11.26%     -0.94%  [.] sinsp_threadinfo::get_main_thread()
     4.18%     +0.48%  [.] thread_group_info::get_first_thread() const
     9.24%     +0.36%  [.] std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()
     3.51%     -0.18%  [.] sinsp_threadinfo::get_fd_table()
    19.99%     -0.14%  [.] sinsp_thread_manager::create_thread_dependencies(std::shared_ptr<sinsp_threadinfo> const&)
     8.84%     -0.11%  [.] std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_add_ref_lock_nothrow()
     0.12%     -0.10%  [.] sinsp_threadinfo::~sinsp_threadinfo()
     0.13%     -0.08%  [.] sinsp_evt::load_params()
     0.17%     -0.08%  [.] void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char const*>(char const*, char const*, std::forward_iterator_tag)

Heap diff from master - unit tests

peak heap memory consumption: 0B
peak RSS (including heaptrack overhead): 0B
total memory leaked: 0B

Heap diff from master - scap file

peak heap memory consumption: 0B
peak RSS (including heaptrack overhead): 0B
total memory leaked: 0B

Benchmarks diff from master

Comparing gbench_data.json to /root/actions-runner/_work/libs/libs/build/gbench_data.json
Benchmark                                                                               Time             CPU      Time Old      Time New       CPU Old       CPU New
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
BM_sinsp_split_mean                                                                  +0.0613         +0.0615           274           291           274           291
BM_sinsp_split_median                                                                +0.0598         +0.0600           274           290           274           290
BM_sinsp_split_stddev                                                                -0.1028         -0.0734             2             2             2             2
BM_sinsp_split_cv                                                                    -0.1546         -0.1271             0             0             0             0
BM_sinsp_concatenate_paths_relative_path_mean                                        -0.0219         -0.0219            88            86            88            86
BM_sinsp_concatenate_paths_relative_path_median                                      -0.0220         -0.0220            88            86            88            86
BM_sinsp_concatenate_paths_relative_path_stddev                                      -0.5386         -0.5464             0             0             0             0
BM_sinsp_concatenate_paths_relative_path_cv                                          -0.5283         -0.5363             0             0             0             0
BM_sinsp_concatenate_paths_empty_path_mean                                           +0.0405         +0.0406            47            49            47            49
BM_sinsp_concatenate_paths_empty_path_median                                         +0.0735         +0.0736            46            49            46            49
BM_sinsp_concatenate_paths_empty_path_stddev                                         -0.8033         -0.8020             3             1             3             1
BM_sinsp_concatenate_paths_empty_path_cv                                             -0.8109         -0.8097             0             0             0             0
BM_sinsp_concatenate_paths_absolute_path_mean                                        +0.0225         +0.0225            86            88            86            88
BM_sinsp_concatenate_paths_absolute_path_median                                      +0.0110         +0.0111            86            87            86            87
BM_sinsp_concatenate_paths_absolute_path_stddev                                      +6.0396         +6.0837             0             2             0             2
BM_sinsp_concatenate_paths_absolute_path_cv                                          +5.8849         +5.9278             0             0             0             0
BM_sinsp_sanitize_string_fast_path_ascii_short_mean                                  -0.0081         -0.0079            19            19            19            19
BM_sinsp_sanitize_string_fast_path_ascii_short_median                                +0.0012         +0.0014            19            19            19            19
BM_sinsp_sanitize_string_fast_path_ascii_short_stddev                                -0.9712         -0.9716             1             0             1             0
BM_sinsp_sanitize_string_fast_path_ascii_short_cv                                    -0.9710         -0.9713             0             0             0             0
BM_sinsp_sanitize_string_fast_path_ascii_long_mean                                   +0.0090         +0.0093           165           166           165           166
BM_sinsp_sanitize_string_fast_path_ascii_long_median                                 +0.0028         +0.0032           165           165           165           165
BM_sinsp_sanitize_string_fast_path_ascii_long_stddev                                +23.4261        +24.1256             0             5             0             5
BM_sinsp_sanitize_string_fast_path_ascii_long_cv                                    +23.2090        +23.8944             0             0             0             0
BM_sinsp_sanitize_string_fast_path_multibyte_short_mean                              +0.0014         +0.0016            14            14            14            14
BM_sinsp_sanitize_string_fast_path_multibyte_short_median                            -0.0015         -0.0012            14            14            14            14
BM_sinsp_sanitize_string_fast_path_multibyte_short_stddev                            -0.9561         -0.9554             0             0             0             0
BM_sinsp_sanitize_string_fast_path_multibyte_short_cv                                -0.9562         -0.9554             0             0             0             0
BM_sinsp_sanitize_string_fast_path_multibyte_long_mean                               -0.0160         -0.0160          4943          4864          4941          4862
BM_sinsp_sanitize_string_fast_path_multibyte_long_median                             -0.0130         -0.0131          4928          4864          4926          4862
BM_sinsp_sanitize_string_fast_path_multibyte_long_stddev                             -0.9718         -0.9716            60             2            60             2
BM_sinsp_sanitize_string_fast_path_multibyte_long_cv                                 -0.9714         -0.9712             0             0             0             0
BM_sinsp_sanitize_string_fast_path_mixed_long_mean                                   -0.1105         -0.1105          2366          2105          2366          2104
BM_sinsp_sanitize_string_fast_path_mixed_long_median                                 -0.1102         -0.1101          2365          2104          2364          2104
BM_sinsp_sanitize_string_fast_path_mixed_long_stddev                                 -0.6465         -0.6555             4             2             4             1
BM_sinsp_sanitize_string_fast_path_mixed_long_cv                                     -0.6025         -0.6127             0             0             0             0
BM_sinsp_sanitize_string_slow_path_c1_controls_long_alloc_mean                       +0.0339         +0.0340          5939          6140          5936          6138
BM_sinsp_sanitize_string_slow_path_c1_controls_long_alloc_median                     +0.0009         +0.0008          6135          6141          6134          6139
BM_sinsp_sanitize_string_slow_path_c1_controls_long_alloc_stddev                     -0.9885         -0.9888           281             3           282             3
BM_sinsp_sanitize_string_slow_path_c1_controls_long_alloc_cv                         -0.9889         -0.9891             0             0             0             0
BM_sinsp_sanitize_string_slow_path_c1_controls_long_noalloc_mean                     +0.0018         +0.0021          6009          6020          6005          6018
BM_sinsp_sanitize_string_slow_path_c1_controls_long_noalloc_median                   -0.0035         -0.0031          6041          6020          6036          6018
BM_sinsp_sanitize_string_slow_path_c1_controls_long_noalloc_stddev                   -0.9917         -0.9921           196             2           197             2
BM_sinsp_sanitize_string_slow_path_c1_controls_long_noalloc_cv                       -0.9918         -0.9922             0             0             0             0
BM_sinsp_sanitize_string_slow_path_sparse_invalid_long_alloc_mean                    +0.0029         +0.0031           325           326           325           326
BM_sinsp_sanitize_string_slow_path_sparse_invalid_long_alloc_median                  +0.0041         +0.0043           325           326           325           326
BM_sinsp_sanitize_string_slow_path_sparse_invalid_long_alloc_stddev                  -0.2733         -0.2585             2             1             2             1
BM_sinsp_sanitize_string_slow_path_sparse_invalid_long_alloc_cv                      -0.2754         -0.2608             0             0             0             0
BM_sinsp_sanitize_string_slow_path_sparse_invalid_long_noalloc_mean                  +0.0122         +0.0123           218           221           218           221
BM_sinsp_sanitize_string_slow_path_sparse_invalid_long_noalloc_median                +0.0071         +0.0070           218           220           218           219
BM_sinsp_sanitize_string_slow_path_sparse_invalid_long_noalloc_stddev                +4.1614         +4.1764             1             4             1             4
BM_sinsp_sanitize_string_slow_path_sparse_invalid_long_noalloc_cv                    +4.0990         +4.1135             0             0             0             0
BM_sinsp_sanitize_string_slow_path_all_invalid_long_alloc_mean                       +0.0002         +0.0002         12443         12445         12439         12441
BM_sinsp_sanitize_string_slow_path_all_invalid_long_alloc_median                     +0.0002         +0.0001         12441         12443         12437         12438
BM_sinsp_sanitize_string_slow_path_all_invalid_long_alloc_stddev                     -0.0341         +0.0441             6             6             6             7
BM_sinsp_sanitize_string_slow_path_all_invalid_long_alloc_cv                         -0.0342         +0.0439             0             0             0             0
BM_sinsp_sanitize_string_slow_path_all_invalid_long_noalloc_mean                     +0.0788         +0.0788         11358         12253         11354         12250
BM_sinsp_sanitize_string_slow_path_all_invalid_long_noalloc_median                   +0.1037         +0.1038         11099         12250         11094         12246
BM_sinsp_sanitize_string_slow_path_all_invalid_long_noalloc_stddev                   -0.9814         -0.9818           473             9           473             9
BM_sinsp_sanitize_string_slow_path_all_invalid_long_noalloc_cv                       -0.9828         -0.9831             0             0             0             0

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 75.74%. Comparing base (01b5cf7) to head (ed3bcc1).
⚠️ Report is 59 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3024      +/-   ##
==========================================
+ Coverage   75.58%   75.74%   +0.16%     
==========================================
  Files         299      299              
  Lines       33123    33251     +128     
  Branches     5138     5139       +1     
==========================================
+ Hits        25037    25187     +150     
+ Misses       8086     8064      -22     

Flag	Coverage Δ
libsinsp	75.74% <ø> (+0.16%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ErenAri]

Gentle ping on this 🙂 — it's intentionally non-blocking (a scheduled lane / discussion-starter per @FedeDP in #3023), so no rush at all, but I'd really value a maintainer's eyes when there's a moment.

@Andreagit97 — given it's deliberately complementary to bpfvalidator (load + attach + per-kernel program-variant selection, not event capture), your read would be especially helpful, and I'm happy to align the manifest with how libpman evolves.

Small update since I opened it: I've also validated the enterprise/backported tier — RHEL 8/9/10 (via the free ABI rebuilds), Oracle UEK 7/8, Amazon Linux 2 incl. the no-BTF 4.14, and openSUSE — 14/14 load+attach, which is the band where "kernel version ≠ feature support" bites hardest.

Glad to trim scope, adjust the matrix/schedule/file layout, or walk through it at a community call — whatever's easiest for the maintainers. Thanks!

[ErenAri]

Friendly ping for review — @ekoops, Andrea Terzolo kindly pointed me your way as the drivers maintainer.

Quick recap of this PR: a non-blocking, scheduled CI lane that builds bpf_probe.o from the tree and load/attach-validates it across vendor distro cloud images in disposable QEMU/KVM VMs, recording the per-kernel program-variant selection (e.g. 5.15 → recvmmsg_old_x / no dump_task; 6.1+ → the bpf_loop variants + both iterators). It's deliberately complementary to bpfvalidator — load/attach + variant evidence, not event capture — and it mirrors libpman's loader contract (runtime-sized maps, helper-gated variants, trial-probed iterators) via a manifest under test/bpfcompat/, so maintainers own it. The PoC workflow is green on a stock ubuntu-latest runner.

Because it's scheduled and non-blocking it can't break merges. Happy to adjust scope/matrix/file layout, or walk through it on a community call — would value your read whenever you have a moment.

[Andreagit97]

I finally took some time to look into this, here are my 2 cents:

First of all thanks for the contribution!

I understand the rationale for validating the .o directly: it avoids requiring users to build and maintain a custom loader binary for different configurations.
That said, as a personal preference, I still lean toward validating through a binary, since this also tests the userspace loader path and avoids keeping a separate config in sync with loader changes over time. Moreover using a binary it would be also possible to run unit tests on different machines.
Do you plan to support a binary command as an alternative to the .o flow?

In any case, we don’t change the loader logic very often in this project, so this approach can still make sense as long as the maintenance cost stays low.

[Andreagit97]

i would probably use the SHA instead of the tag, it is safer. Same in all the other actions of this workflow

[terror96]

I agree, it sometimes pops up as a suggestion in security scanning.

[ErenAri]

Thanks, that makes sense. I’ll pin bpfcompat to a commit SHA instead of the version tag, and I’ll check the other actions in this workflow as well.

On the binary flow: yes, I agree with the point. The current .o-based validation is meant to keep the initial integration simple and low-maintenance, but supporting a binary command mode is a good direction since it would also validate the userspace loader path. I’ll add that as an alternative flow so projects can choose between direct artifact validation and full loader-path validation.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[ErenAri]

Thanks @Andreagit97 — both points addressed.

Pinning by SHA — done. actions/checkout, actions/upload-artifact, and the Kernel-Guard/bpfcompat action are now pinned to full commit SHAs (with the version tag kept in a trailing comment) throughout the workflow. Thanks @terror96 for the second on this.

Validating through a binary — I agree this is the more valuable signal: it exercises the userspace loader path, and there's no separate config to keep in sync with loader changes over time. bpfcompat now supports exactly that — a command/binary mode where, instead of loading the .o with the bundled validator, it runs your loader inside each matrix-kernel VM and the per-kernel verdict is the command's exit code:

bpfcompat test-command --bin ./your-loader --cmd '$BPFCOMPAT_BIN --self-test' --matrix ...

For this PR that maps cleanly onto scap-open: build the modern_bpf engine, ship scap-open into each guest, run it bounded (a fixed event count or a short timeout), and treat a clean startup+exit as "libpman brought the probe up on this kernel." That validates the real loader path you described, not just the kernel's acceptance of the object — and it also means unit/loader tests could run per-kernel the same way.

I'm happy to add that as a second job here (alongside, or in place of, the .o flow) if you'd like the PR to demonstrate the binary path — or keep the .o flow as the low-maintenance default and document command mode as the opt-in. Either way; just let me know which you'd prefer and I'll push it.

[Andreagit97]

Thanks! if i had to choose one approach i would just use the one with the binary (scap-open). It seems more maintainable over time and allow us to test also the loader. This is just my personal taste, but since I'm no more an approver of the repo i would leave the final decision to the maintainers @falcosecurity/libs-maintainers

[ekoops]

Hey, thank you for your effort. As @Andreagit97 , I strongly prefer the binary-based approach. What's the advantage of bpfcompat over bpfvalidator we are already using in CI? It seems that bpfcompat is able to test the binary over a specific combination of distro/kernel, as I inferred from the names your specified in "profiles". Is it correct?

[ErenAri]

Yes, that’s correct. bpfcompat’s main advantage over a local validator is that it validates the compiled eBPF artifact against real distro/kernel combinations. It boots each target kernel in a VM, runs the load/attach check there, and reports a per-kernel compatibility result.
For Falco, this can work as a release/CI gate for modern_bpf, especially to catch regressions on older supported kernels. It can also run Falco’s own loader through command mode, so the result can match Falco’s real loading path instead of relying only on the bundled validator.

[ErenAri]

@ekoops @Andreagit97 — done: the PR is now binary-based, as you both preferred. ed3bcc1 reworks the workflow to build scap-open from this tree (statically linked, modern_bpf skeleton embedded) and run it inside each matrix kernel VM; its exit code is the per-kernel verdict (scap-open --modern_bpf --num_events 10 — scap_open() exercises libpman's full load path, then captures a bounded number of events). The .o+manifest flow and test/bpfcompat/modern-bpf-manifest.yaml are removed — there's no config to keep in sync with the loader anymore; the loader is the contract.

To @ekoops' question, the concrete difference from bpfvalidator in one real run — this exact static build, executed via the workflow's command on real vendor images (QEMU/KVM):

Kernel	Verdict	Detail
ubuntu-20.04 (5.4)	❌ exit 95	libpman: ring buffer map type is not supported (errno: 22) — correct rejection
almalinux-8 (4.18)	✅ exit 0	RHEL ring-buffer backport: probe loads, attaches, and captures events on a 4.18 kernel
ubuntu-22.04 (5.15)	✅ exit 0	*_old_x variant band

That AlmaLinux row is the case a version check can't answer: a lower-numbered enterprise kernel passing while a higher-numbered upstream one fails — caught by running the real loader on the real vendor kernel.

One portability note baked into the workflow: scap-open is built with MUSL_OPTIMIZED_BUILD=ON (-static), because the same binary has to run unchanged on guests whose glibc is older than the runner's (RHEL-family). PR description updated to match. Happy to adjust the matrix, event count, or schedule.

[Andreagit97]

LGTM, thanks!

[poiana]

LGTM label has been added.

Details

Git tree hash: f09b6aff2696676d9a1a2199b45406747f7a7726

[poiana]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ekoops, ErenAri, leogr

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [ekoops,leogr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ErenAri]

Heads-up before the first scheduled run (Monday 06:00 UTC): I dress-rehearsed the merged workflow on a fork and it needs two extra apt packages on the runner (the SHA-pinned action builds its validator from source, which needs libbpf headers). One-line fix with a fully green rehearsal run linked: #3055 — would be great to get it in before Monday so the lane's first run is green.