Updating LLD to use bulk memory for threads

[tlively]

In the near future I plan to update LLD to emit passive segments when a shared memory is requested. This will prevent memory from being reinitialized every time a thread is spawned. Currently toolchains need to do something like separate the contents of memory into a separate file and manually initialize memory on the main thread. With this update to LLD, the linker will synthesize a function called __wasm_init_memory. This function will automatically be called at the beginning of __wasm_call_ctors, but @fitzgen reminded me that Rust does not have any global constructors, so you will probably be able to call __wasm_init_memory directly.

The existing memory separation code will probably break since the data segments will no longer declare their own offsets, so I just wanted to give a heads up that this change is coming. It also means that enabling shared memory in Firefox will be insufficient to test multithreaded modules, which may affect your workflow or testing. I filed a Firefox bug about this at https://bugzilla.mozilla.org/show_bug.cgi?id=1561747.

I will update this issue once the change has landed upstream in LLD and when I update the tool-conventions repo with details about this arrangement.

[fitzgen]

Thanks for the heads up @tlively!

cc @alexcrichton

[alexcrichton]

Indeed thanks for the heads up! It'll take a bit for LLD changes to get into rust-lang/rust since we need to update the LLVM compiler in rust-lang/rust before it reaches nightlies or here, so we've definitely got some good runway.

BTW in case you aren't aware @tlively this is actually something that we all handle today in the wasm-bindgen-threads-xform crate internally, and you can see an example of this running online at - https://rustwasm.github.io/wasm-bindgen/exbuild/raytrace-parallel/. We do exactly what you're changing LLD to do which is to switch all memory segments to passive and then manually initialize them.

We don't currently have __wasm_call_ctors it's true (although in theory Rust code could link to C++ and it would show up), but we can definitely arrange our transformation pass to ensure that __wasm_call_ctors or __wasm_init_memory is hooked up to only happen once on the main thread.

In any case once this lands in LLD I'll start working on the update to rust-lang/rust, and in that process I'll make sure we update the passes here to be compatible with it!

[tlively]

This change to LLD is complete, and although I just landed the last patch today, it should be backported and available on the LLVM 9.0 release branch. The behavior of the linker is documented in the tool conventions. Passive segments will be emitted by default when you pass --shared-memory and all you'll have to do is call __wasm_init_memory from your runtime before anything else happens.

[alexcrichton]

Thanks for the heads up! Our LLVM 9 upgrade recently landed and I'm working on getting the threading support back up and running with this. I'll make sure to include support for --passive-segments as well!

[alexcrichton]

@tlively on issue I've found I opened up on the tool conventions repo WebAssembly/tool-conventions#117, but otherwise looks like everything is working out great

[alexcrichton]

@tlively it's probably also worth mentioning, if y'all have discussions for the general architecture of threaded wasm I wrote up a post awhile back about our experience along with gotchas and some problems to solve, so if it'd help I could help work through some issues as well!