Run Maestro KBuild Job using TuxMake container environment

[pawiecz]

Is your feature request related to a problem? Please describe.
TuxMake and Maestro maintain very similar container images for kernel builds. This task should focus on verifying if images from one framework can be reused in the other.

Describe the solution you'd like
Initially a successful run of a Maestro KBuild build.sh (example) will be enough.
Long-term it would be nice to maintain only a single set of container images: Maestro ones also almost pass environment check:

$ tuxmake --runtime=docker --image=kernelci/gcc-12:x86-kselftest-kernelci --check-environment | grep FAIL
E: Environment check failed
FAIL - bzip2 in PATH
FAIL - socat in PATH
FAIL - zstd in PATH

(cc @nuclearcat @bhcopeland @terceiro)

[pawiecz]

Initial attempt assumed recreating kbuild-gcc-12-x86-allnoconfig Job on kernelci/staging-stable Tree.

Following steps needed to be made to get a successful build:

# use similar execution environment
$ docker run --rm -it tuxmake/x86_64_gcc-12 bash

# (within container environment)
# collect build config fragments and set up artifacts directories
wget -P /tmp/kci/artifacts/fragments https://files-staging.kernelci.org/kbuild-gcc-12-x86-allnoconfig-689310a4498bf8a0e6f66507/fragments/0.config https://files-staging.kernelci.org/kbuild-gcc-12-x86-allnoconfig-689310a4498bf8a0e6f66507/fragments/1.config
# collect pre-cached data
wget -O /data/linux-firmware.tar.gz https://storage.kernelci.org/linux-firmware.tar.gz
# create remaining directories that build script expects to be present
mkdir -p /tmp/kci/linux
# run the build script
wget https://files-staging.kernelci.org/kbuild-gcc-12-x86-allnoconfig-689310a4498bf8a0e6f66507/build.sh
chmod +x build.sh 
./build.sh

These steps resulted in almost matching Job artifacts:

# find /tmp/kci/artifacts/
/tmp/kci/artifacts/
/tmp/kci/artifacts/build_kselftest.log
/tmp/kci/artifacts/.config
/tmp/kci/artifacts/build_kimage_stderr.log
/tmp/kci/artifacts/build_kselftest_stderr.log
/tmp/kci/artifacts/bzImage
/tmp/kci/artifacts/state.txt
/tmp/kci/artifacts/fragments
/tmp/kci/artifacts/fragments/1.config
/tmp/kci/artifacts/fragments/0.config
/tmp/kci/artifacts/kselftest.tar.gz
/tmp/kci/artifacts/build_kimage.log
/tmp/kci/artifacts/build.log
/tmp/kci/artifacts/vmlinux # .xz in Maestro

(Maestro also keeps a metadata file for KBuild Jobs)

[terceiro]

cool!

I forgot to mention this earlier, but this is no coincidence: when I started the tuxmake images, they were based on the package list of kernelci at the time.

[pawiecz]

Thanks for letting me know!

There are two more actions to complete this task:

1. Run a similar experiment to my initial one but with a kernel that could be used to boot an actual device (not just a build test)
2. Select a set of TuxMake images, replace the ones in config/jobs.yaml (see below) and verify that they work as expected on staging:

kernelci-pipeline$ grep -E "image:.*(gcc|clang)" config/jobs.yaml | sort -u
    image: ghcr.io/kernelci/{image_prefix}clang-17:arm64-kselftest-kernelci
    image: ghcr.io/kernelci/{image_prefix}clang-17:arm-kselftest-kernelci
    image: ghcr.io/kernelci/{image_prefix}clang-17:riscv64-kselftest-kernelci
    image: ghcr.io/kernelci/{image_prefix}clang-17:x86-kselftest-kernelci
    image: ghcr.io/kernelci/{image_prefix}gcc-12:arc-kselftest-kernelci
    image: ghcr.io/kernelci/{image_prefix}gcc-12:arm64-kselftest-kernelci
    image: ghcr.io/kernelci/{image_prefix}gcc-12:arm-kselftest-kernelci
    image: ghcr.io/kernelci/{image_prefix}gcc-12:mips-kselftest-kernelci
    image: ghcr.io/kernelci/{image_prefix}gcc-12:riscv64-kselftest-kernelci
    image: ghcr.io/kernelci/{image_prefix}gcc-12:x86-kselftest-kernelci
    image: ghcr.io/kernelci/{image_prefix}gcc-12:x86-kunit-kernelci

I could complete (1) on my own but I'll need support to complete (2).