Dynamic Loading

[rennergade]

General issue thread for implementing/discussing dynamic loading

[rennergade]

I came across this thread in Zulip where dynamic loading options were discussed: https://bytecodealliance.zulipchat.com/#narrow/stream/206238-general/topic/Is.20it.20possible.20to.20instantiate.20two.20modules.20into.20one.20instance.3F

[rennergade]

Emscripten seems to do dynamic loading with wasm: https://emscripten.org/docs/compiling/Dynamic-Linking.html

[rennergade]

It seems like RLBox is only configured for statically linked libraries but what theyre doing here is still interesting: https://rlbox.dev/chapters/wasm-sandbox.html

[qianxichen233]

I finished a proof of concept implementation for dynamic loading. I am currently able to load a simple library wasm file with the constant data correctly mapped and indirect function table correctly placed. The main process is able to use the string and function pointers in library, and the library is also able to use the string and function pointers from main process.

Current issue is about thread local storage interacting with -pie flag. When I tried to compile a program that uses lind_syscall, all the PIE relative address of constant data in library are misaligned for exactly 4 bytes. It turns out this was because errno used in lind_syscall.c, since once I delete the errno in lind_syscall.c, everything works again. I suspect this is related to TLS handling with -pie flag since errno is just defined as a thread-local variable. Now I am pretty sure this is caused by __thread keyword on errno since once I delete the __thread keyword, it could also work

[qianxichen233]

I did some more experiments to figure out what is going on.

For the above issue, I found that if I intentionally subtract the 4 bytes when assigning memory base address of library, all the strings can be correctly printed. So it seems like the entire data section is misaligned by the 4 bytes. Then I tried to create some more thread local variables to see what is going to happen, and I found out that the data section is always misaligned by exact TLS size bytes. In the previous example, there is only one thread variable which is int type (so the TLS size is 4 bytes), so the data is misaligned by 4 bytes. If I create more thread variables for diferent size of TLS and its always misaligned for exactly TLS bytes. So if I intentionally minus TLS size bytes when assigning memory base, the constant data can be correctly pointed.

However, when I tried to replace write with printf in library function, I still get issues even if I intentionally subtract TLS bytes to memory base. I looked into the issue, and found out that even all the constant strings are correctly aligned, there are a few globally defined struct which becomes misaligned. And what's more interesting is that if I do not subtract TLS bytes to memory base, then these globally defined struct will be correctly aligned but the constant strings will become misaligned. I did some more investigation and found out that these globally defined struct has a little bit difference access mode than constant strings. Unlike directly using the pointers adding memory base to access like constant string, it is defined as a wasm global (GOT.data.internal.name) and utilized intermediate pointer (data section) to access.

There are three data sections when compiling c to wasm with TLS support (without -pie):

1. tdata section: essentially TLS data
2. rodata section: places where global data (like global string) live
3. data section: Holds pointer to the global data in rodata section.

When -pie option is added to wasm-ld, all of these three sections will be (sequentially) merged into one section.

Since rodata section is the place where the string lives, in the merged data section, some of this (originally rodata) section is misaligned by TLS size, but the (originally data) section isn't

I did two more experiments to verify this:

1. I tried to manually modify the wasm binary to subtract TLS size to the constant string pointer used, and it works: all the string, no matter from main process or library is correctly printed, and printf on both main process and library is working normally (which means the globally defined struct used by printf is also correctly aligned).
2. I tried not to modify the wasm binary, but slightly changed how I copy the constant data to memory:
   Instead of

ptr::copy_nonoverlapping(src.as_ptr(), addr as *mut u8, src.len());

where src is the library's constant data, and addr is the destination address in main process's linear memory, allocated by rawposix mmap.
I experimennted with

let end = 30000;
ptr::copy_nonoverlapping(src.as_ptr(), (addr + 1312) as *mut u8, end); // TLS size = 1312
ptr::copy_nonoverlapping(src.as_ptr().add(end), (addr + end as u64) as *mut u8, src.len() - end);

which intentionally copied most of the rodata section with TLS size off, which counteracts the misalignment in string accessing, and remains the data section not affected since they are already correctly aligned. This will lost the 1312 bytes of data in the middle, but this is an experimentation to figure out what is going on anyway.
With this modified copying behavior, I also get my program running successfully. Both printf and constant strings in both main process and library are working expected.

I am still not sure if this is a bug in wasm-ld's pie option interacting with TLS data, or it's me did some wrong in handling the data, which is something I will further investigate

[JustinCappos]

This is really interesting. Good job tracking this down! Seems like a perfect thing to ask the wasm folks about.

…

On Fri, Feb 14, 2025 at 6:37 PM Qianxi Chen ***@***.***> wrote: I did some more experiments to figure out what is going on. For the above issue, I found that if I intentionally subtract the 4 bytes when assigning memory base address of library, all the strings can be correctly printed. So it seems like the entire data section is misaligned by the 4 bytes. Then I tried to create some more thread local variables to see what is going to happen, and I found out that the data section is always misaligned by exact TLS size bytes. In the previous example, there is only one thread variable which is int type (so the TLS size is 4 bytes), so the data is misaligned by 4 bytes. If I create more thread variables for diferent size of TLS and its always misaligned for exactly TLS bytes. So if I intentionally minus TLS size bytes when assigning memory base, the constant data can be correctly pointed. However, when I tried to replace write with printf in library function, I still get issues even if I intentionally subtract TLS bytes to memory base. I looked into the issue, and found out that even all the constant strings are correctly aligned, there are a few globally defined struct which becomes misaligned. And what's more interesting is that if I do not subtract TLS bytes to memory base, then these globally defined struct will be correctly aligned but the constant strings will become misaligned. I did some more investigation and found out that these globally defined struct has a little bit difference access mode than constant strings. Unlike directly using the pointers adding memory base to access like constant string, it is defined as a wasm global (GOT.data.internal.name) and utilized intermediate pointer (data section) to access. There are three data sections when compiling c to wasm with TLS support (without -pie): 1. tdata section: essentially TLS data 2. rodata section: places where global data (like global string) live 3. data section: Holds pointer to the global data in rodata section. When -pie option is added to wasm-ld, all of these three sections will be (sequentially) merged into one section. Since rodata section is the place where the string lives, in the merged data section, some of this (originally rodata) section is misaligned by TLS size, but the (originally data) section isn't I did two more experiments to verify this: 1. I tried to manually modify the wasm binary to subtract TLS size to the constant string pointer used, and it works: all the string, no matter from main process or library is correctly printed, and printf on both main process and library is working normally (which means the globally defined struct used by printf is also correctly aligned). 2. I tried not to modify the wasm binary, but slightly changed how I copy the constant data to memory: Instead of ptr::copy_nonoverlapping(src.as_ptr(), addr as *mut u8, src.len()); where src is the library's constant data, and addr is the destination address in main process's linear memory, allocated by rawposix mmap. I experimennted with let end = 30000; ptr::copy_nonoverlapping(src.as_ptr(), (addr + 1312) as *mut u8, end); // TLS size = 1312 ptr::copy_nonoverlapping(src.as_ptr().add(end), (addr + end as u64) as *mut u8, src.len() - end); which intentionally copied most of the rodata section with TLS size off, which counteracts the misalignment in string accessing, and remains the data section not affected since they are already correctly aligned. This will lost the 1312 bytes of data in the middle, but this is an experimentation to figure out what is going on anyway. With this modified copying behavior, I also get my program running successfully. Both printf and constant strings in both main process and library are working expected. I am still not sure if this is a bug in wasm-ld's pie option interacting with TLS data, or it's me did some wrong in handling the data, which is something I will further investigate — Reply to this email directly, view it on GitHub <#13 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGROD6OFJQ2CHYMTBTEWZL2PZ43BAVCNFSM6AAAAABNFGMBHSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNRQGQ4DENRQGM> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***> [image: qianxichen233]*qianxichen233* left a comment (Lind-Project/lind-wasm#13) <#13 (comment)> I did some more experiments to figure out what is going on. For the above issue, I found that if I intentionally subtract the 4 bytes when assigning memory base address of library, all the strings can be correctly printed. So it seems like the entire data section is misaligned by the 4 bytes. Then I tried to create some more thread local variables to see what is going to happen, and I found out that the data section is always misaligned by exact TLS size bytes. In the previous example, there is only one thread variable which is int type (so the TLS size is 4 bytes), so the data is misaligned by 4 bytes. If I create more thread variables for diferent size of TLS and its always misaligned for exactly TLS bytes. So if I intentionally minus TLS size bytes when assigning memory base, the constant data can be correctly pointed. However, when I tried to replace write with printf in library function, I still get issues even if I intentionally subtract TLS bytes to memory base. I looked into the issue, and found out that even all the constant strings are correctly aligned, there are a few globally defined struct which becomes misaligned. And what's more interesting is that if I do not subtract TLS bytes to memory base, then these globally defined struct will be correctly aligned but the constant strings will become misaligned. I did some more investigation and found out that these globally defined struct has a little bit difference access mode than constant strings. Unlike directly using the pointers adding memory base to access like constant string, it is defined as a wasm global (GOT.data.internal.name) and utilized intermediate pointer (data section) to access. There are three data sections when compiling c to wasm with TLS support (without -pie): 1. tdata section: essentially TLS data 2. rodata section: places where global data (like global string) live 3. data section: Holds pointer to the global data in rodata section. When -pie option is added to wasm-ld, all of these three sections will be (sequentially) merged into one section. Since rodata section is the place where the string lives, in the merged data section, some of this (originally rodata) section is misaligned by TLS size, but the (originally data) section isn't I did two more experiments to verify this: 1. I tried to manually modify the wasm binary to subtract TLS size to the constant string pointer used, and it works: all the string, no matter from main process or library is correctly printed, and printf on both main process and library is working normally (which means the globally defined struct used by printf is also correctly aligned). 2. I tried not to modify the wasm binary, but slightly changed how I copy the constant data to memory: Instead of ptr::copy_nonoverlapping(src.as_ptr(), addr as *mut u8, src.len()); where src is the library's constant data, and addr is the destination address in main process's linear memory, allocated by rawposix mmap. I experimennted with let end = 30000; ptr::copy_nonoverlapping(src.as_ptr(), (addr + 1312) as *mut u8, end); // TLS size = 1312 ptr::copy_nonoverlapping(src.as_ptr().add(end), (addr + end as u64) as *mut u8, src.len() - end); which intentionally copied most of the rodata section with TLS size off, which counteracts the misalignment in string accessing, and remains the data section not affected since they are already correctly aligned. This will lost the 1312 bytes of data in the middle, but this is an experimentation to figure out what is going on anyway. With this modified copying behavior, I also get my program running successfully. Both printf and constant strings in both main process and library are working expected. I am still not sure if this is a bug in wasm-ld's pie option interacting with TLS data, or it's me did some wrong in handling the data, which is something I will further investigate — Reply to this email directly, view it on GitHub <#13 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGROD6OFJQ2CHYMTBTEWZL2PZ43BAVCNFSM6AAAAABNFGMBHSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNRQGQ4DENRQGM> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[qianxichen233]

I just posted an issue in LLVM repo about this. Hopefully we can get some feedbacks about what is going on.

Besides that, I found a temporary workaround for the issue: just disable TLS build of glibc by removing -matomics -mbulk-memory options in clang. This would definitely cause all the threading feature to crash, but at least it could make those do not need threading to work correctly. I recompiled my test programs without TLS build, and this time, I do not need any modification in terms of handling data address to be able to have both constant data/function pointers and printf on both main process and library working correctly

[rennergade]

I'm combing through the code and see that when you redid pthreads you never used __wasilibc_pthread_self which I believe wasi-threads use. Could this be causing any issues?

[qianxichen233]

I'm combing through the code and see that when you redid pthreads you never used __wasilibc_pthread_self which I believe wasi-threads use. Could this be causing any issues?

This was originally handled here:

lind-wasm/src/glibc/nptl/pthread_create.c

Line 278 in 003a83d

__wasilibc_pthread_self = *self;

But when we migrate the old threading method to clone_syscall based, I forgot the copy this line to its desired place in new implementation. I originally thought that TLS is not working in lind-wasm currently, but I just tested and found out it is working at least on basic level, that the variable defined with __thread keyword is indeed not shared by threads. I'll fix the pthread_self later, but I think this shouldn't be related to dynamic loading, since I reproduced the weird misalign behavior without using any standard library (glibc)

[rennergade]

Ok makes sense. Can you link to the LLVM issue for our reference?