Run semu inside web browser via WebAssembly translation

[ChinYikMing]

WebAssembly is capable to port rv32emu to browser. Then, I would like to test the WebAssembly translation of the semu codebase to see if kernels (like Linux or xv6) can be booted in a browser in order to facilitate the further integration of semu to rv32emu.

I would like to start with the smaller codebase and boot xv6 first. A few technical points are as follows:

1. It is necessary to simulate a new shell that is operating in a web frontend (Xterm.js is a good solution), after which the command is forwarded to the kernel's running shell. Some of the scope issues with the variables generated by Emscripten are confusing to me, so that I have implemented a hardcoded ls command to confirm the kernel is booted successfully. The demo below shows Emscripten's capability to boot a kernel.
2. The sh shell's dollar sign ($) of xv6 does not flush after exec, possibly some source kernel code needs to be changed.

Demo: xv6

You shall see output of ls command like below:

$ .              1 1 1024
..             1 1 1024
README         2 2 2059
cat            2 3 25016
echo           2 4 23800
forktest       2 5 13688
grep           2 6 28568
init           2 7 24896
kill           2 8 23752
ln             2 9 23704
ls             2 10 27320
mkdir          2 11 23856
rm             2 12 23840
sh             2 13 43824
stressfs       2 14 24864
usertests      2 15 159064
grind          2 16 39424
wc             2 17 26208
zombie         2 18 23224
console        3 19 0

Next, command bridging from web frontend shell(Xterm.js) to real shell will be solved, and then boot the Linux kernel.

[ChinYikMing]

Command bridging from web frontend shell(Xterm.js) to real shell has been solved. Now, xv6 can be booted and use it normally.

Before entering any command, please wait and see this kernel message:

init: starting sh

Otherwise, kernel might panic.

Porting source: link

[ChinYikMing]

semu uses mmap for fixed memory mapping but fixed memory mapping is not currently supported by emscripten's libc. Thus, the following patch shall be applied:

+    char buf[4096];
+    int offset = 0;
+    int ret;
+    while((ret = read(fd, buf, 4096)) > 0) {
+	    memcpy(*ram_loc + offset, buf, ret);
+	    offset += ret;
+    }

-    *ram_loc = mmap(*ram_loc, st.st_size, PROT_READ | PROT_WRITE,
-                    MAP_FIXED | MAP_PRIVATE, fd, 0);
-    if (*ram_loc == MAP_FAILED) {
-        perror("mmap");
-	printf("errno: %d\n", errno);
-	printf("EINVAL: %d\n", EINVAL);
-	printf("size: %lld\n", st.st_size);
-	printf("name: %s\n", name);
-        close(fd);
-        exit(2);
-    }
...

After applying this patch, Linux can be boot and show dmesg messages.

[    0.000000] Linux version 6.1.0-dirty (chinyikming@chinyikming-ROG-Strix-G533ZW-G533ZW) (riscv32-buildroot-linux-gnu-gcc.br_real (Buildroot -ge07402af) 12.3.0, GNU ld (GNU Binutils) 2.39) #26 Tue Jan 23 02:47:20 CST 2024
[    0.000000] Machine model: semu
[    0.000000] earlycon: ns16550 at MMIO 0xf4000000 (options '')
[    0.000000] printk: bootconsole [ns16550] enabled
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000000000000-0x000000001fffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
...

But, it will stuck at some point. The difficult part to determine the stuck point is that it does not generate error although initcall_debug log_level=8 are passed as bootargs to kernel. After tracing the kernel, console_on_rootfs calls filp_open. Thus, I added some debug messages to these two functions to determine kernel is stuck at console_on_rootfs since the debug message in filp_open is not printed out.

console_on_rootfs

void __init console_on_rootfs(void)
{
+	pr_debug("before filp_open /dev/console");
	struct file *file = filp_open("/dev/console", O_RDWR, 0);
	...
}

filp_open

void __init console_on_rootfs(void)
{
+	pr_debug("before getname_kernel\n");
	struct filename *name = getname_kernel(filename);
	...
}

To reproduce it, follow the steps in linux-wasm branch. You will see something like below:

...
[    3.223146] calling  tcp_congestion_default+0x0/0x34 @ 1
[    3.223476] initcall tcp_congestion_default+0x0/0x34 returned 0 after 4 usecs
[    3.223867] calling  ip_auto_config+0x0/0xfb8 @ 1
[    3.224271] initcall ip_auto_config+0x0/0xfb8 returned 0 after 79 usecs
[    3.224663] calling  clk_disable_unused+0x0/0xf8 @ 1
[    3.225002] initcall clk_disable_unused+0x0/0xf8 returned 0 after 4 usecs
[    3.225403] calling  of_platform_sync_state_init+0x0/0x2c @ 1
[    3.225725] initcall of_platform_sync_state_init+0x0/0x2c returned 0 after 3 usecs
[    3.226112] before filp_open /dev/console

According to the stuck point, I believe the bug is highly relevent to console/tty/UART. Any ideas or experiences on this?

[jserv]

I believe the bug is highly relevent to console/tty/UART. Any ideas or experiences on this?

You have the option to disable the standard UART path and retain earlycon solely for debugging purposes.

[ChinYikMing]

But, it will stuck at some point. The difficult part to determine the stuck point is that it does not generate error although initcall_debug log_level=8 are passed as bootargs to kernel. After tracing the kernel, console_on_rootfs calls filp_open. Thus, I added some debug messages to these two functions to determine kernel is stuck at console_on_rootfs since the debug message in filp_open is not printed out.

After tracing the kernel call stack more deeply, the real stuck point is calling spin_unlock_irqrestore inside serial8250_do_set_termios . This synchronization problem might be complex since it does not exist in C runtime but only in Node runtime. Thus, I simply disable serial8250_do_set_termios, and the kernel runs normally in Node runtime(so does in C runtime).

Note: disable serial8250_do_set_termios might affect the baud rate, start/stop bit, etc.

Kernel call stack:

spin_unlock_irqrestore
    ^
    |
    | next call
    |
[    4.391383] [<c034ffb0>] serial8250_do_set_termios+0x590/0x610
[    4.391748] [<c03500c4>] serial8250_set_termios+0x94/0x98
[    4.392096] [<c034f1c8>] uart_change_speed+0xbc/0x174
[    4.392430] [<c034f738>] uart_startup+0x2bc/0x31c
[    4.392754] [<c034f850>] uart_port_activate+0xb8/0xf4
[    4.393088] [<c034f048>] tty_port_open+0x114/0x1ac
[    4.393414] [<c034f448>] uart_open+0x4c/0x80
[    4.393718] [<c034eaf8>] tty_open+0x4a0/0x5b4
[    4.394027] [<c034d410>] chrdev_open+0x2d4/0x308
[    4.394340] [<c034cf40>] do_dentry_open+0x424/0x44c
[    4.394663] [<c034cfb8>] vfs_open+0x50/0x68
[    4.394960] [<c034e038>] path_openat+0xbf4/0xcb0
[    4.395274] [<c034e164>] do_filp_open+0x70/0x114
[    4.395586] [<c034d064>] file_open_name+0x94/0xac
[    4.395902] [<c034d0fc>] filp_open+0x80/0xc0
[    4.396201] [<c0359d4c>] console_on_rootfs+0x34/0x84
[    4.396539] [<c0359f70>] kernel_init_freeable+0x1d4/0x230
[    4.396896] [<c0351ff0>] kernel_init+0x24/0x118

Although the kernel boots, but one more problem(very weird) comes out. The kernel hangs when single line of output printed out during running the "/etc/ini.d/rcS" script (before login shell), see demo, press Enter when Run /init as init process kernel message is shown. As you can see, the console prints "Starting syslogd:" and hangs. I tried comment out all echo/printf command , then the rest of service are created successfully. Thus, I have confirmed that printing single line to console hangs the kernel.

[jserv]

Can you confirm if the above appears for latest linux-6.1.y ?

[jserv]

I found an interesting WebAssembly-based RISC-V system emulator capable of running Linux kernel with MMU. See RISC-V Emulator with workable WASM!
Check build_and_run.yaml for building and running.

rv32iasu_emulator

[ChinYikMing]

Can you confirm if the above appears for latest linux-6.1.y ?

I have tried v5.17, v5.19, v6.1, v6.7 and the stuck point still the same.

For simplicity, comment out serial8250_do_set_termios temporarily can make the kernel boot successfully but hangs when a line is printed out to shell or /dev/console. Tracing how the line copy to kernel space and how the corresponding /dev/console driver handle it might help to figure out what is going on.

The tracing should be done on host machine. The first tool I can think about is ftrace. Do you recommend ftrace or other tracing tools (ebpf) ?

[jserv]

The tracing should be done on host machine. The first tool I can think about is ftrace. Do you recommend ftrace or other tracing tools ( ebpf ) ?

ftrace might be a bit heavy for current system emulation since it expects huge available system resources.

ply might be useful for some cases.

[LekKit]

Could the terminal emulation efforts be shared and reused in the RVVM project?

WebAssembly is already working there: https://lekkit.github.io/test/index.html

I am working on a VT emulator locally but I don't think i am able to write a VT100 renderer in JS

[jserv]

Could the terminal emulation efforts be shared and reused in the RVVM project?
WebAssembly is already working there: https://lekkit.github.io/test/index.html
I am working on a VT emulator locally but I don't think i am able to write a VT100 renderer in JS

Hi @LekKit,

Thank you for your recent messages regarding RISC-V system emulation. I wanted to share some background on my work in this area and discuss potential collaboration opportunities.

My motivation for initiating the semu project stemmed from a need in my research on dynamic compilers and operating systems. I found there was no small, simple, open-source RISC-V system emulator capable of running the Linux kernel in less than 3,000 lines of code. This compact codebase was crucial for my research goals.

The initial public release of semu achieved this, containing less than 2,000 lines of C code while including minimal VirtIO support. This streamlined design made it easier for my students to understand and contribute to the project.

Recently, my team and I have made further progress in this field:

1. We submitted a paper titled "Accelerate RISC-V Instruction Set Simulation by Tiered JIT Compilation" to the VMIL'24 conference. This work explores methods to enhance RISC-V instruction set simulation performance.
2. We developed rv32emu, another open-source project I initiated for teaching and research purposes.

I want to acknowledge that RVVM is an excellent open-source project. In fact, I have had my students study its codebase to deepen their understanding of RISC-V instruction set simulation and system emulation.

Moving forward, instead of reworking the somewhat hacky semu code, I would like to propose a collaborative approach. If you are amenable, I would encourage my students (who are contributors to the semu project) to contribute WebAssembly components or VirtIO-specific implementations to RVVM. This could be a mutually beneficial way to advance RISC-V emulation efforts.

[LekKit]

Wow thank you for such a kind and thoughtful response! I really appreciate it

Yes, I don't want to sound too selfish, but I believe that if we want something competitive with QEMU and unique the efforts should be directed and not fragmened. But I really think that your project is very cool too.

Additionally, RVVM is also sorta oriented at being "small", but mostly in terms of native code footprint. Trying to write readable / well-documented / portable / error-handling code tends to be long anyways, but we can disable a lot of features like JIT, RV64 CPU and different devices/networking via useflags.

I am also wondering if anything on my side can be further reinforced and improved. Sorry if some of my contributors are bothering you lately, I hope they figure their stuff out :D (I don't have time to look at virtio stuff yet and wanna focus on CPU side, network stack and image format)