falco kernel module doesn't build on beta (2983) but works on stable (2905)

[jepio]

Description

Falco uses a docker container to bind mount kernel module paths from the host and then builds a kernel module.
This build runs inside a debian 10 container, with access to kernel modules dirs.

Impact

Flatcar doesn't work with an upstream project, needs downstream changes to workflows.

Environment and steps to reproduce

1. Action:

docker run -i -t --privileged -v /root/.falco:/root/.falco -v /proc:/host/proc:ro -v /boot:/host/boot:ro -v /lib/modules:/host/lib/modules:ro -v /usr:/host/usr:ro -v /etc:/host/etc:ro --name build  falcosecurity/falco:latest

2. Error: on beta this fails with: scripts/basic/fixdep: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by scripts/basic/fixdep)

Expected behavior

Expected build to work.

Additional information

The supported way to build kernel modules on Flatcar is through the developer container, but that is a custom solution that doesn't integrate with other projects. The docker container used by falco is a fragile approach, and we're hitting two issues with it. The build system tries to be compatible with old kernels so builds with gcc-5 first, but has a fallback to gcc-8. This is not quite the same gcc that current Flatcar kernels are built with (gcc-9) but could work. The second issue is that it depends on the kernel build directory being prepared for kernel module builds, ours is partially compatible (since it worked before, but i believe this is more of an accident than intentional). Beta channel binaries are built with glibc-2.33 which is newer than what the container provides so they can't be executed.

We also have a flaw in that arm64 images contain kernel module binaries from the build system (amd64). We could improve the situation:

• build statically linked kernel module build system binaries into images
• provide enough kernel build system files so that make modules_prepare works (but this requires ongoing maintenance)
• find another way to improve falco build support (bind mount /lib64?)

[jepio]

The first bullet point would require something like this in one of the ebuilds:
make modules_prepare HOSTLDFLAGS=-static

[jepio]

Tried to go down the static linking path, that failed. sysdig-agent has a better build system which requires bind mounting /usr and relocates the binaries to find the correct libraries/linker. Falco's build system doesn't support this.

The best that I could come up with is this gist: https://gist.github.com/jepio/ee301a76b437c80f5533588172bc790e. It creates a wrapper script and a bunch of symlinks that use the host dynamic linker and glibc, and that works for building falco. Should we ship that configuration at build time? @pothos @tormath1, what do you think.

[jepio]

So with this falcosecurity/falco#2043 merged, this should work out-of-the-box when Falco 0.33.0 is released. You can already test this with the master tag of the falco container images:

docker run --rm -i -t
  --privileged
  -v /root/.falco:/root/.falco
  -v /proc:/host/proc:ro
  -v /boot:/host/boot:ro
  -v /lib/modules:/host/lib/modules:ro
  -v /usr:/host/usr:ro
  -v /etc:/host/etc:ro
  falcosecurity/falco-driver-loader:master

Pre-built kernel modules/bpf probes would require adding to falco test-infra, I wonder if there are enough users of falco+flatcar that this would be justified.

[dongsupark]

Tested on Stable 3227.2.4, and it works out of the box.