Add Makefile for executing user space apps on QEMU guest shell

[StevenChen1997]

Observation during testing

• In current QEMU guest shell environment, we have no C dynamic linker/loader for executing user application userspace_ioctl and cat_nonblock if we use the following gcc command to build.

gcc -g -o cat_nonblock cat_nonblock.c
gcc -g -o userspace_ioctl userspace_ioctl.c

• In QEMU guest shell, we will get the following execution result:

lkmpg:/mnt/lkmpg/examples/other# ./cat_nonblock
/bin/sh: ./cat_nonblock: not found
lkmpg:/mnt/lkmpg/examples/other# ./userspace_ioctl
/bin/sh: ./userspace_ioctl: not found

Root cause

• Use command file to figure out dependencies, found that C dynamic linker/loader /lib64/ld-linux-x86-64.so.2 is required.

$ file cat_nonblock | fold -s -w $(tput cols)
cat_nonblock: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, 
interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=4accdc9b913e30abf0d472ddb34d0b0cc661ca2f, for 
GNU/Linux 3.2.0, with debug_info, not stripped

$ file userspace_ioctl | fold -s -w $(tput cols)
userspace_ioctl: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, 
interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=40b8a83882abfed398bbeb72722668a363c9e86e, for 
GNU/Linux 3.2.0, with debug_info, not stripped

$ ls /lib64/ld-linux-* -al
lrwxrwxrwx 1 root root 44 Jan 30 16:27 /lib64/ld-linux-x86-64.so.2 -> ../lib/x86_64-linux-gnu/ld-linux2

• Found that QEMU guest shell has no C dynamic linker/loader (/lib64/ld-linux-x86-64.so.2) deployed.

lkmpg:~# ls /lib64
ls: /lib64: No such file or directory
lkmpg:~# ls
bin      etc      linuxrc  proc     sbin     tmp
dev      init     mnt      root     sys      usr
lkmpg:~# 

Proposed solution in this PR

An alternative solution is attaching -static flag in gcc input arguments. This flag makes gcc staticly link and load C runtime libraries into final executable file. The disadvantage is bigger executable file size and redundant C runtime library scattered in different executable files. (Same code but loaded into multiple processes)

gcc -static -g -o cat_nonblock cat_nonblock.c
gcc -static -g -o userspace_ioctl userspace_ioctl.c

In QEMU guest shell, we will get the following execution result:

lkmpg:/mnt/lkmpg/examples/other# ./cat_nonblock          # Able to execute now
Usage: ./cat_nonblock <filename>
Reads the content of a file, but doesn't wait for input
lkmpg:/mnt/lkmpg/examples/other# ./userspace_ioctl       # Able to execute now
Can't open device file: /dev/char_dev, error:-1

About the Makefile

• Consider that not all environments has no C dynamic linker/loader, using flag -static for compilation depends on your execution environment. Hence, I implemented a Makefile which lets user select if attaching the -static flag is required. I also attached the make command into build-modules.sh to automate the build flow for userspace apps running on QEMU guest shell.
• If your environment has already installed with /lib64/ld-linux-x86-64.so.2 or alternative C dynamic linker/loader, you can use make directly for compiling.

Trade off and further discussion

• For the executable with static linked library, its size is much bigger than the executable with dynamic linked library. In addition, the common used library for both programs should be aggregated to single file and load it dynamically to avoid redundant loading on same library.
• Maybe adding C dynamic linker/loader into QEMU guest shell is a better choice. Please feel free to discuss further under this Pull Request.
• Reference: comparing size of executable files

$ gcc -g -o cat_nonblock_dyn cat_nonblock.c
$ gcc -g -o userspace_ioctl_dyn userspace_ioctl.c

$ gcc -static -g -o cat_nonblock_st cat_nonblock.c
$ gcc -static -g -o userspace_ioctl_st userspace_ioctl.c

lkmpg/examples/other $ ls
Makefile  cat_nonblock.c  cat_nonblock_dyn  cat_nonblock_st  userspace_ioctl.c  userspace_ioctl_dyn  userspace_ioctl_st

lkmpg/examples/other $ file cat_nonblock_dyn cat_nonblock_st userspace_ioctl_dyn userspace_ioctl_st | fold -s -w $(tput cols)
cat_nonblock_dyn:    ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter 
/lib64/ld-linux-x86-64.so.2, BuildID[sha1]=4accdc9b913e30abf0d472ddb34d0b0cc661ca2f, for GNU/Linux 3.2.0, with debug_info, not stripped
cat_nonblock_st:     ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, 
BuildID[sha1]=79331f608a4398ddccfacf7cfa1cceee190ba8fd, for GNU/Linux 3.2.0, with debug_info, not stripped
userspace_ioctl_dyn: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter 
/lib64/ld-linux-x86-64.so.2, BuildID[sha1]=40b8a83882abfed398bbeb72722668a363c9e86e, for GNU/Linux 3.2.0, with debug_info, not stripped
userspace_ioctl_st:  ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, 
BuildID[sha1]=39cd2d7e3d64d6dd1659e952d87f10c8a2c2938c, for GNU/Linux 3.2.0, with debug_info, not stripped

lkmpg/examples/other $ du -sh cat_nonblock_dyn cat_nonblock_st userspace_ioctl_dyn userspace_ioctl_st
20K     cat_nonblock_dyn
772K    cat_nonblock_st
20K     userspace_ioctl_dyn
772K    userspace_ioctl_st

---

Summary by cubic

Add a Makefile to build user-space apps with optional static linking and wire it into the build so cat_nonblock and userspace_ioctl run on the QEMU guest shell without a dynamic linker. Tracks header deps and rebuilds when QEMU_ENV changes.

• New Features
  • Added examples/other/Makefile: builds all .c; adds -static when QEMU_ENV=1; auto-generates .d; writes .qemu-env to detect env changes.
  • Updated devtools/build-modules.sh: uses make -C and builds with QEMU_ENV=1; tightened .gitignore to examples/other for binaries, .d, and .qemu-env.

^{Written for commit a091b90. Summary will update on new commits.}

[cubic-dev-ai]

All reported issues were addressed across 3 files (changes from recent commits).

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>