Investigate instance pooling strategy in wasmtime

[pepyakin]

Since some time, wasmtime supports instance pooling. In theory, this mechanism should improve startup latency.

If so, we may not need to employ our own rolled hacked such as "fast instance reuse" mechanism, which already gives us some pain. For example, it takes more effort to implement (#9164 (comment)) or potentially may give way to hard to debug errors (#10095).

[bkchr]

@koute do you maybe want to work on this?

[koute]

Sure I can take a stab at it (since I'm in the area anyway); just one question - does anyone remember if we already have a benchmark for this? (If not I'll add one.)

[bkchr]

We don't have a benchmark for this yet. Feel free to add one 🙂

[koute]

So I've done some experiments. Here's a quick rundown of the numbers based on a simple benchmark (calls new_instance and then calls the test_empty_return within the runtime):

• Each iteration with fast instance reuse currently takes ~5us.
• Each iteration without fast instance reuse currently takes ~1ms.
• After Statically register host WASM functions #10394 (plus one more PR that I'll put up after that one) is merged each iteration without fast instance reuse and without pooling strategy will take ~30us.
• And finally on top of that turning on the pooling only gives us an extra ~5us, so ~25us per iteration.

So pooling doesn't actually net that much extra speed (at least in this benchmark), however even without pooling after my PRs the codepath without fast instance reuse becomes pretty fast, although unfortunately not as fast as our own fast instance reuse method.

So now the question is - what do you want to do here? We could maybe close the gap even further with some optimization work on wasmtime's side, we could just eat the ~25us loss, or we could keep it as-is (with the fast instance reuse being the default) and reinvestigate another day.

[pepyakin]

Thanks for so detailed investigation.

Could you try the same but with a real runtime? One of the biggest source of overhead for Fast Instance Reuse (FIR) is copying over the data segments. In a real runtime data segments could reach 200-300 KiB. I expect that it would also influence the pooling strategy, however, we can also try to use wasmtime's userfaultfd which may help. In that case we should be careful with a more representative workload. Also that would depend on a fresh Linux but that may be ok.

Besides that we could also try to see if things like wasmtime::InstancePre or register the functions in config can shave some more time. I wouldn't hold the breath for those though.

Admittedly, FIR is a hack and I'd be happy to get rid of it. At some point we may run out of tricks in our sleeves to keep it work in case changes related to new features. Ideally we improve wasmtime so that it works for this ultra low latency scenario. I am optimistic because AFAIU the folks behind wasmtime also value low-latency a lot.

[koute]

Okay, here are some numbers calling into the Kusama runtime (the function called within the runtime is exactly the same - I copy-pasted it from the output of cargo expand for the test runtime and recompiled it as part of the full Kusama runtime):

• With fast instance reuse: ~49us
• Without fast instance reuse: ~4.9ms
• Without fast instance reuse, after my PRs, no pooling: ~83us
• Without fast instance reuse, after my PRs, with pooling, without uffd: ~83us
• Without fast instance reuse, after my PRs, with pooling, with uffd: ~48us

...so this does look promising if we enable both pooling and uffd, basically giving us the same performance as with fast instance reuse! (Enabling only one of them gives the normal ~83us.)

So it looks like we can probably just delete the fast instance reuse codepath? Before we commit it'd be nice to make one more test that's more real-world, e.g. importing a bunch of blocks from an actual production chain and timing that, or something along these lines. @pepyakin Any idea what would be good to run?

[pepyakin]

Ok, this is very useful and a good sign.

I think even without UFFD it is not that bad: it's 2x regression compared to 5x we witnessed before.

Relying on UFFD is kind of annoying still though. I checked the code and it seems to require 4.11. It seems that we can afford to impose the minimum kernel version requirement. On the first glance it seems that in case we enable UFFD, executor will fail during the run time on <4.11, which is really annoying.

While I was browsing the code, I stumbled upon the paged_memory_initialization configuration option which I completely forgot about. Did you enable it? Nevermind, paged_memory_initialization is enabled by default with UFFD enabled.

Regarding the tests. I think block importing may be a good start, although once again it may be a bit deceiving, at least in case of the production chain. E.g. with UFFD we can basically skip the data segment initialization, however, at the price of a more costly access to the un-paged memory areas where data segments reside (or maybe even untouched/zeroed areas, I haven't dived into the implementation that deep). In a production chain the first many blocks will be empty and will hit only the same pattern, and as I imagine only a subset of data segments will be actually used. If that's the case, maybe it will be better to come up with a synthetic test.