Optimize `array.fill` and `array.copy` by alexcrichton · Pull Request #13382 · bytecodealliance/wasmtime

alexcrichton · 2026-05-15T02:32:24Z

This commit is a step towards optimizing the codegen and implementation of the array.fill and array.copy GC instructions. This builds on the intrinsics refactored in #13367 and #13368 to be used in the GC implementation as well. For example array.fill and array.new_default for i8 arrays will now use memset to fill an array. The larger change here is the reimplementation of array.copy to inline the loop within generated CLIF.

Prior to this commit the array.copy function was exclusively built as a libcall. There were two versions of this libcall -- one for GC references and one for non GC references. Even with this minor optimization, however, the benchmark in #13279 is still quite slow. The main reason for this is that the GC references version of the libcall has an extremely high constant overhead per element copied. This is due to the nature of the implementation using completely safe APIs which engages a lot more dynamic type checks and GC heap tracking than necessary.

The implementation of array.copy in this commit now looks like:

First the bounds of the copy operation are all validated. Traps happen if either array is out of bounds.
Next raw heap addresses are calculated for the start/end of the src/dst.
For copyable types (e.g. i8, i32, etc), the implementation then delegates to the same intrinsic as memory.copy, a host-defined memmove.
For GC types (e.g. anyref), the implementation instead has an inline loop which copies elements from the source to the destination. This copy is structured as two loops, either a forwards copy or a backwards copy, and only one is taken.

The result of this implementation is that the benchmark in #13279 is massively faster. What previously took minutes-to-hours now completes in seconds, even with the DRC collector.

One caveat with this implementation is that when delegating to memory.copy or memory.fill there's an
unnecessary-if-wasmtime-has-no-bugs bounds check double-check the host will not fault. While wasm code will gracefully catch segfaults the host will not, so this is necessary to ensure that in the face of heap corruption the host never faults.

In the future it might make sense to split out functionality like this into separate helper CLIF functions to keep the source relatively lean. In the meantime though the massive performance benefits seem to outweigh the size costs (to me at least).

Closes #13279

This commit is a step towards optimizing the codegen and implementation of the `array.fill` and `array.copy` GC instructions. This builds on the intrinsics refactored in bytecodealliance#13367 and bytecodealliance#13368 to be used in the GC implementation as well. For example `array.fill` and `array.new_default` for `i8` arrays will now use `memset` to fill an array. The larger change here is the reimplementation of `array.copy` to inline the loop within generated CLIF. Prior to this commit the `array.copy` function was exclusively built as a libcall. There were two versions of this libcall -- one for GC references and one for non GC references. Even with this minor optimization, however, the benchmark in bytecodealliance#13279 is still quite slow. The main reason for this is that the GC references version of the libcall has an extremely high constant overhead per element copied. This is due to the nature of the implementation using completely safe APIs which engages a lot more dynamic type checks and GC heap tracking than necessary. The implementation of `array.copy` in this commit now looks like: * First the bounds of the copy operation are all validated. Traps happen if either array is out of bounds. * Next raw heap addresses are calculated for the start/end of the src/dst. * For copyable types (e.g. `i8`, `i32`, etc), the implementation then delegates to the same intrinsic as `memory.copy`, a host-defined `memmove`. * For GC types (e.g. `anyref`), the implementation instead has an inline loop which copies elements from the source to the destination. This copy is structured as two loops, either a forwards copy or a backwards copy, and only one is taken. The result of this implementation is that the benchmark in bytecodealliance#13279 is massively faster. What previously took minutes-to-hours now completes in seconds, even with the DRC collector. One caveat with this implementation is that when delegating to `memory.copy` or `memory.fill` there's an unnecessary-if-wasmtime-has-no-bugs bounds check double-check the host will not fault. While wasm code will gracefully catch segfaults the host will not, so this is necessary to ensure that in the face of heap corruption the host never faults. In the future it might make sense to split out functionality like this into separate helper CLIF functions to keep the source relatively lean. In the meantime though the massive performance benefits seem to outweigh the size costs (to me at least).

github-actions · 2026-05-15T09:36:31Z

Subscribe to Label Action

cc @fitzgen

Details

This issue or pull request has been labeled: "wasmtime:api", "wasmtime:ref-types"

Thus the following users have been cc'd because of the following labels:

fitzgen: wasmtime:ref-types

To subscribe or unsubscribe from this label, edit the .github/subscribe-to-label.json configuration file.

Learn more.

fitzgen · 2026-05-15T12:55:36Z

+    // If this is a byte array then specialize its fill to use the same libcall
+    // as `memory.fill`. This gets us to `memset` on the host which for larger
+    // arrays can be a big boost due to the vectorized implementation. Note
+    // though that this explicitly checks to make sure that the final address
+    // being written is in-bounds in the GC heap to detect corruption and traps
+    // here in-wasm. Once we're in the host GC heap corruption can't be caught,
+    // so it needs to be an up-front check here.
+    if let WasmStorageType::I8 = elem_ty {


Do we want to special case all POD types here? Like why not do this for i16 and v128 and f32 and even i31ref?

I guess memory_fill only takes a byte value to fill? Might make sense to have 8/16/32/64/128 versions. File a follow up issue?

I was wondering the same, yeah, and you're right the reason is that it'd require new libcalls and I wasn't looking to expand the set of libcalls here. I also figure that languages will probably expect byte-arrays to be the most optimal so I figured we'd at least have a fast path for that. Can file a follow-up issue yeah.

I've filed an issue at #13386, and upon considering this more I realized that one thing we'd want to do, even in the future world of larger bit-widths, is that we still want to specialize based on what's actually being filled in. For example array.new_default with zero-initialization should be a memset for pretty much all types.

I've gone ahead and implemented that here, too. Mind re-reviewing the latest commit?

fitzgen

Ship it!

This commit is the dual of bytecodealliance#13382 but for tables. The goal here is to move the copying operation into WebAssembly to enable fuel/epoch checking during larger copies (bytecodealliance#13387). This additionally enables removing quite a lot of library code to implement this in the host. Finally, most of the code to do this was already implemented for compilation, so much of this PR focuses on refactoring all of that to reuse all of the existing logic in a way that table support "falls out by default".

Fixes a bug from bytecodealliance#13382 where `translate_loop_header` happened too soon. Needs to happen after block params are inserted, not before.

This commit is the dual of bytecodealliance#13382 but for tables. The goal here is to move the copying operation into WebAssembly to enable fuel/epoch checking during larger copies (bytecodealliance#13387). This additionally enables removing quite a lot of library code to implement this in the host. Finally, most of the code to do this was already implemented for compilation, so much of this PR focuses on refactoring all of that to reuse all of the existing logic in a way that table support "falls out by default".

* Move `table.copy` implementation to compiled code This commit is the dual of #13382 but for tables. The goal here is to move the copying operation into WebAssembly to enable fuel/epoch checking during larger copies (#13387). This additionally enables removing quite a lot of library code to implement this in the host. Finally, most of the code to do this was already implemented for compilation, so much of this PR focuses on refactoring all of that to reuse all of the existing logic in a way that table support "falls out by default". * Refactor manual copy loop * Take src/dst into account separately when reading/writing. * Use `table.{get,set}` translation to reuse formats/flags/etc.

alexcrichton requested review from a team as code owners May 15, 2026 02:32

alexcrichton requested review from cfallin and removed request for a team May 15, 2026 02:32

alexcrichton mentioned this pull request May 15, 2026

wasmtime works extremely slow on kotlin benchmarks #13279

Closed

alexcrichton requested review from fitzgen and removed request for cfallin May 15, 2026 02:34

Fix conditional compile

cead1e6

github-actions Bot added wasmtime:api Related to the API of the `wasmtime` crate itself wasmtime:ref-types Issues related to reference types and GC in Wasmtime labels May 15, 2026

fitzgen approved these changes May 15, 2026

View reviewed changes

Review comments

639a52d

alexcrichton mentioned this pull request May 15, 2026

Optimize array.fill with more libcalls, optimize array.copy for more collectors #13386

Open

fitzgen approved these changes May 15, 2026

View reviewed changes

Comment thread crates/cranelift/src/func_environ/gc/enabled.rs

Comment thread crates/cranelift/src/func_environ/gc/enabled.rs

fitzgen mentioned this pull request May 15, 2026

Huge array allocations are timing out during fuzzing #11427

Closed

Specialize more bit patterns in array.fill

96f3be9

fitzgen mentioned this pull request May 15, 2026

Should we add fuel/epoch checks into the middle of {array,memory,table}.{fill,copy,init}? #13387

Closed

fitzgen approved these changes May 15, 2026

View reviewed changes

fitzgen added this pull request to the merge queue May 15, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks May 15, 2026

Fix 32-bit test

7ed2d30

alexcrichton enabled auto-merge May 15, 2026 21:57

alexcrichton added this pull request to the merge queue May 15, 2026

Merged via the queue into bytecodealliance:main with commit 2b7c854 May 15, 2026
51 checks passed

alexcrichton deleted the refactor-array-copy branch May 15, 2026 22:33

Ari4ka approved these changes May 16, 2026

View reviewed changes

alexcrichton mentioned this pull request May 17, 2026

Fix fuel/epoch checks in array.copy loop #13402

Merged

alexcrichton mentioned this pull request May 19, 2026

Move table.copy implementation to compiled code #13407

Merged

ttraenkler mentioned this pull request Jun 6, 2026

Enable Wasm GC by default #13216

Closed

14 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Optimize `array.fill` and `array.copy`#13382

Optimize `array.fill` and `array.copy`#13382
alexcrichton merged 5 commits into
bytecodealliance:mainfrom
alexcrichton:refactor-array-copy

alexcrichton commented May 15, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented May 15, 2026

Uh oh!

fitzgen May 15, 2026

Uh oh!

alexcrichton May 15, 2026

Uh oh!

alexcrichton May 15, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fitzgen left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

alexcrichton commented May 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented May 15, 2026

Subscribe to Label Action

Uh oh!

fitzgen May 15, 2026

Choose a reason for hiding this comment

Uh oh!

alexcrichton May 15, 2026

Choose a reason for hiding this comment

Uh oh!

alexcrichton May 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fitzgen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

alexcrichton commented May 15, 2026 •

edited

Loading