Strange application behaviour difference between host and unikernel

[grobe0ba]

Application

The application I'm attempting to run under OSv is a small toy I threw together that uses https://github.com/cesanta/mongoose to serve a handful of embedded files over HTTP, and can be found at https://github.com/grobe0ba/ssl-config-gen-osv.

I have placed the application in apps/ssl-config-gen, and successfully built a kernel.elf/usr.img combo which successfully runs under qemu.

The Problem

When run on the host (linking without -shared using the debug target of the application Makefile), the executable runs correctly, serves the embedded files as so forth without issue.

When run on OSv, mongoose fails to recognize the embedded assets, and simply returns errors to that effect over HTTP.

Mongoose does not have runtime detection of these assets, it is a compile-time configuration. I have confirmed using strings|grep that the assets are in fact still present in the stripped shared object that OSv is using.

Help me, Obi-wan Kenobi(s)

Any ideas that y'all could throw my way would be appreciated, as I really have no idea how this behaviour is changing like this. I know I could just embed the assets using OSv itself, and in general that would be a better option, but I was taking an opportunity to play with mongoose as well.

This may not be the correct forum for this issue, but since the application builds and runs correctly both on RHEL and OpenBSD, I'm inclined to think there aren't any issues with mongoose causing this, although I could be wrong.

Even with this interesting little conundrum, y'all have done a lot of good work, and I appreciate it. Thanks!

Build Host

I'm using a local build of OSv, built on RHEL 8.5 (Linux lindev.pulpie.xyz 4.18.0-348.7.1.el8_5.x86_64 #1 SMP Wed Dec 8 21:51:17 EST 2021 x86_64 x86_64 x86_64 GNU/Linux) using x86_64-linux-musl-gcc (x86_64-linux-musl-gcc (GCC) 9.4.0) built using richfelker/musl-cross-make@0f22991 and boostorg/boost@9d3f9bc (tag: boost-1.77.0).

Environment during build

export CROSS_PREFIX=x86_64-linux-musl-
export HOST_CXX=c++

Image build step

$ ./scripts/build fs=rofs image=ssl-config-gen
Building into build/release.x64
  GEN gen/include/osv/version.h
No such image configuration: ssl-config-gen. Assuming list of modules.
Importing /home/grobe0ba/Projects/osv/apps/ssl-config-gen/module.py
Modules:
  ssl-config-gen.*
cc -I. -DMG_ENABLE_PACKED_FS=1 -DMG_ENABLE_LINES=1 -DMG_ENABLE_LOG=1 -fPIC -c -o ssl-config-gen.o ssl-config-gen.c
cc -I. -DMG_ENABLE_PACKED_FS=1 -DMG_ENABLE_LINES=1 -DMG_ENABLE_LOG=1 -c -o pack.o pack.c
cc  -o pack pack.o
./pack `find assets -type f | xargs` > packed_fs.c
cc -I. -DMG_ENABLE_PACKED_FS=1 -DMG_ENABLE_LINES=1 -DMG_ENABLE_LOG=1 -fPIC -c -o packed_fs.o packed_fs.c
cc -I. -DMG_ENABLE_PACKED_FS=1 -DMG_ENABLE_LINES=1 -DMG_ENABLE_LOG=1 -fPIC -c -o mongoose.o mongoose.c
cc  -fPIC -shared -o ssl-config-gen.so ssl-config-gen.o \
        packed_fs.o mongoose.o
Preparing usr.manifest
Appending /home/grobe0ba/Projects/osv/apps/ssl-config-gen/usr.manifest to usr.manifest
Preparing bootfs.manifest
Saving command line to /home/grobe0ba/Projects/osv/build/release.x64/cmdline
Building into build/release.x64
  GEN gen/include/osv/version.h
Writing image
Adding /libenviron.so
Adding /libvdso.so
Adding /tools/mount-fs.so
Adding /tools/umount.so
Adding /usr/lib/libgcc_s.so.1
Adding /etc/hosts
Link /etc/mnttab to /proc/mounts
Adding /etc/fstab
Adding /ssl-config-gen.so
First block: 11783, blocks count: 2
Directory entries count 17
Symlinks count 1
Inodes count 18

[wkozaczuk]

Hi,

I am not very familiar with the app. But after I ran OSv with the trace enabled to see how it interacts with the filesystem I could see this:

./scripts/run.py --forward="tcp::8080-:8080" --trace=vfs*

...
0xffff8000014e4040 >init            0         0.114085104 vfs_lstat            pathname=/ssl-config-gen.so, stat=0x00002000000fd710
0xffff8000014e4040 >init            0         0.114087489 vfs_lstat_ret        
0xffff8000014e4040 >init            0         0.114087789 vfs_open             "/ssl-config-gen.so" 0x0 00
0xffff8000014e4040 >init            0         0.114088979 vfs_open_ret         3
0xffff8000014e4040 >init            0         0.114089090 vfs_close            3
0xffff8000014e4040 >init            0         0.114089197 vfs_close_ret        
0xffff8000018c9040 /ssl-config-gen  0         0.117238028 vfs_fcntl            3 3 0x901f
0xffff8000018c9040 /ssl-config-gen  0         0.117238792 vfs_fcntl_ret        "32770"
0xffff8000018c9040 /ssl-config-gen  0         0.117239031 vfs_fcntl            3 4 0x8802
0xffff8000018c9040 /ssl-config-gen  0         0.117240725 vfs_fcntl_ret        "0"
0xffff8000018c9040 /ssl-config-gen  0         0.117240874 vfs_fcntl            3 2 0x1
0xffff8000018c9040 /ssl-config-gen  0         0.117240991 vfs_fcntl_ret        "0"
0xffff8000018c9040 /ssl-config-gen  0         0.117268009 vfs_pwritev          1 0x0000200000200bf0 0x2 0x-1
0xffff8000018c9040 /ssl-config-gen  0         0.118601989 vfs_pwritev_ret      0x52
0xffff8000018c9040 /ssl-config-gen  0         4.177907884 vfs_fcntl            4 4 0x0
0xffff8000018c9040 /ssl-config-gen  0         4.177908619 vfs_fcntl_ret        "0"
0xffff8000018c9040 /ssl-config-gen  0         4.177914301 vfs_fcntl            4 3 0x0
0xffff8000018c9040 /ssl-config-gen  0         4.177914420 vfs_fcntl_ret        "32770"
0xffff8000018c9040 /ssl-config-gen  0         4.177914504 vfs_fcntl            4 4 0x8802
0xffff8000018c9040 /ssl-config-gen  0         4.177914688 vfs_fcntl_ret        "0"
0xffff8000018c9040 /ssl-config-gen  0         4.177914771 vfs_fcntl            4 2 0x1
0xffff8000018c9040 /ssl-config-gen  0         4.177914895 vfs_fcntl_ret        "0"
0xffff8000018c9040 /ssl-config-gen  0         4.177954961 vfs_pwritev          1 0x00002000001ff4a0 0x2 0x-1
0xffff8000018c9040 /ssl-config-gen  0         4.179079968 vfs_pwritev_ret      0x3b
0xffff8000018c9040 /ssl-config-gen  0         4.179090387 vfs_stat             "assets" 0x00002000001fe5c0
0xffff8000018c9040 /ssl-config-gen  0         4.179101487 vfs_stat_err         2
0xffff8000018c9040 /ssl-config-gen  0         4.179339118 vfs_close            4
0xffff8000018c9040 /ssl-config-gen  0         4.179342836 vfs_close_ret        

So the app clearly tried to look for assets file/directory and it was not present on OSv filesystem.

The solution was quite simple - add assets to the filesystem. The patch to the usr.manifest is this:

diff --git a/usr.manifest b/usr.manifest
index 74a9d28..7ad820f 100644
--- a/usr.manifest
+++ b/usr.manifest
@@ -1 +1,2 @@
 /ssl-config-gen.so: ${MODULE_DIR}/ssl-config-gen.so
+/assets/**: ${MODULE_DIR}/assets/**

And the app seems to function just fine.

[grobe0ba]

Yeah, that's the solution for a proper deployment. The current design is supposed to access assets stored in the binary itself, and it looks like I didn't actually do that correctly.

No bugs to be found here, just errors on my part.

Thanks again, y'all.