cmd/compile: create GOARCH=wasm32

[johanbrandhorst]

Background

The GOARCH=wasm architecture was first introduced alongside GOOS=js in 2019. The choice was made to use a 64 bit architecture because of WebAssembly’s (Wasm) native support for 64 bit integer types and the existing Wasm proposal to introduce a 64 bit memory address space, as detailed in the original Go Wasm design doc. The assumption at the time was that the Wasm ecosystem was going to switch to a 64 bit address space over time, and that Go would benefit from having used a 64 bit architecture from the start.

On the browser side, Firefox and Chrome both have experimental support for 64 bit memory, and it seems poised to become stable in the coming year. However, on the server side, all existing hosts use a 32 bit architecture, and it has become clear that it is here to stay.

In most uses of Wasm with Go, the architecture is transparent to the user, but with the introduction of go:wasmimport in #38248 and #59149, knowing the memory layout of input and output parameters becomes very important, as described in #59156. A short term solution of restricting the types of input and output parameters to scalar types and unsafe.Pointer was adopted to make this clear to users, but this user experience is not a desirable long term solution.

As an example, the fd_write import from wasi_snapshot_preview1 accepts an *iovec of this type:

type iovec struct {
    buf *byte
    len uint32
}

Because GOARCH=wasm has 64 bit pointers we have currently defined this type as:

type iovec struct {
    buf uint32
    len uint32
}

which gives the correct alignment but has issues:

• The conversion from *byte to uint32 causes the compiler to lose track of the pointer, potentially causing the GC to reclaim the memory before the imported function is called. It requires using runtime.KeepAlive on inner pointers passed to imported function calls to ensure that they remain alive.

• It creates a poor and error prone user experience, as the conversion of the pointer type looks like this:

  iov := &iovec{
      buf: uint32(uintptr(unsafe.Pointer(unsafe.SliceData(buf)))),
      len: uint32(len(buf)),
  }
  
  

This issue is already affecting early adopters of the wasip1 port: Fastly SDK.

Performance

A 32 bit architecture would allow the compiled Go code to be more performant, due to the effects of locality, the larger size of certain structs and the difficulty of avoiding bounds checking. It would also use less memory.

Proposal

Create a new GOARCH=wasm32 architecture which uses 32 bit pointers and integers. The architecture would only be supported in a new wasip1/wasm32 port. int, uint and uintptr would all be 32 bit in length.

The maintainers of the existing wasip1/wasm port (@johanbrandhorst , @Pryz, @evanphx) volunteer to become maintainers of this new port.

Discussion

The introduction of GOARCH=wasm32 would allow us to write safer code, because the pointer size matches what the host expects:

type iovec struct {
    buf *byte // pointers are 32 bits
    len int   // int is 32 bits
}

iov := &iovec{
    buf: unsafe.SliceData(buf),
    len: len(buf),
}

In this case, the compiler can track the use of the slice pointer, and there is no risk that the GC will collect the objects before calling the imported function. There is also stronger type safety since there is no need to bypass the compiler to perform unsafe type conversions.

This provides a much improved user experience for users writing wrappers for host provided functions.

The existing wasip1/wasm port would remain and retain the go:wasmimport type restrictions. It may eventually become deprecated and removed, in accordance with the Go porting policy. Such a change would be subject to a separate proposal.

There are no plans for introducing a js/wasm32 port.

[eliben]

What is the plan for when 64-bit will be finally supported for the browser? Which additional port(s) will be added? What about when WASI starts supporting 64-bit too?

[cherrymui]

Thanks for the proposal. SGTM overall. A few questions:

• on the server side, all existing hosts use a 32 bit architecture, and it has become clear that it is here to stay.

I guess this is probably true. Some supporting links would be great. It is unlikely for the server side to have a plan to go to 64-bit?

• uses 32 bit pointers and integers

Integers are interesting. Wasm support 64-bit integer operations. I assume we'll continue to use those for 64-bit integer operations (Go's int64 and uint64), which should be more efficient than decomposing to 32-bit operations? Will int and uint become 32-bit? Do int64 and uint64 have 4-byte alignment (like we have for other 32-bit architectures)?

• Currently, how much do users need to directly interact with this? I guess in many cases this is hidden in the internals of low-level packages (like runtime and syscall), and users will just use high-level APIs? Maybe this matters mostly for people using wasmimport'd functions that are not included in packages like os and syscall?

• How much work user would need to do to migrate from the current 64-bit wasm to wasm32? Is there a plan to help user migrate, maybe with some automation?

• For the implementation side, I guess we can share a good amount of code for both 32-bit and 64-bit Wasm, at least for the toolchain? Do you have an estimate of the amount of the code that is needed?

Thanks.

[evanphx]

Hi @cherrymui

Since WASI is one of the primary motivators behind this additional port, I'll use it as an example. Referencing https://github.com/WebAssembly/WASI/blob/main/legacy/preview1/docs.md, you'll see they (thankfully) are explicit in their integer sizes.

The question of the width of the Go int type is absolutely relevant, but I think it ends up being mostly relevant in how it interplays with uintptr and pointer sizes. Since we're talking about going from the large version today to a smaller version, it's unlikely to cause any issues.

The user interacts for this are pretty minimal so we're not worried about it having much an an impact. I'll say that as far as migrations are concerned, people who use Go's wasm support today are having to migrate existing code today because of the use of 64bit pointers today. So when wasm32 is introduced, they'll be able to effectively undo those migrations and go back to doing it the same way they do with all the other wasm compiled code.

Coding wise, our hope is that because of the structure of the Go compiler abstracting most of this, it should be a fairly small change. But that change is likely to be diffuse amongst a bunch of places.

Next week during Gophercon I'm hoping to do pick up where @dgryski left off here https://github.com/dgryski/go-wasi/tree/dgryski/wasm32/ and see how far it can be pushed forward quickly. That should give us a better idea of how much more is required.

[evanphx]

@eliben I wish I had a better of how long before we see wasm64 uptick in browsers. To be honest, we sort of expected to see it by now.

I think that beyond browser's adopting, one of the bigger roadblocks to adoption is the wasm ABI interactions with javascript. Let's assume for a moment that the javascript code wants to pass a string to a function implemented in webassembly. The wasm ABI requires today that javascript know the exact byte layout, in memory, that the webassembly code will see the string in memory. In fact it requires it so much, that the caller has to write the string into the linear memory that is shared with webassembly, and then pass the offset of the value in the linear memory byte array, which the webassembly sees as a pointer. 😓

Put another way, wasm doesn't provide any ABI abstractions between JS and Webassembly code, which exacerbates issues like the number of bytes a pointer takes up because JS is constantly manually constructing these values by hand.

[cherrymui]

@evanphx thanks!

Referencing https://github.com/WebAssembly/WASI/blob/main/legacy/preview1/docs.md

Thanks for the link. From the API it is clear that pointer is 32-bit. For integers, it seems the system APIs use explicitly sized types like u32. I think for the Go side we may also want to use explicitly sized integer types like uint32 when mapping the system APIs. E.g. for the example above, iovec probably should be defined as

type iovec struct {
    buf *byte  // pointers are 32 bits
    len uint32 // map to wasm u32
}

instead of using int or uint for the second field.

The question of the width of the Go int type is absolutely relevant, but I think it ends up being mostly relevant in how it interplays with uintptr and pointer sizes.

Yes, for the completeness of the port, we need to decide the size of Go's int and uint types. Currently on all platforms that the gc toolchain supports, int and uintptr have the same width (although the spec doesn't require it and gccgo does support platforms where they differ). So it probably makes sense to define int and uint 32-bit. (The proposal probably needs to mention it.)

We also need to determine the alignment of 64-bit types like int64, uint64 (and perhaps float64). Currently on all other 32-bit platforms they are 4-byte aligned. It looks like the WASI API https://github.com/WebAssembly/WASI/blob/main/legacy/preview1/docs.md requires 64-bit values 8-byte aligned (e.g. filesize: u64 has "Alignment: 8"). So should we choose to align them to 8-byte in Go? This would probably need some work, to get an alignment beyond pointer size, but probably makes sense and worth it as we start a new port. Another option is that they are usually 4-byte aligned in Go code, but we explicitly align them when interacting with the system API.

https://github.com/dgryski/go-wasi/tree/dgryski/wasm32/

Thanks for sharing this! This includes some interesting changes that are more internal to the compiler and irrelevant to the system API, such as cf5a3fa . This switches to use 32-bit operations for 32-bit values. Currently in the current Wasm port, the "registers" (for the Go compiler, see the original design doc) are 64-bit. Do you plan to switch to 32-bit "registers"? A mixture of 32-bit and 64-bit (which seems more complex to me)? Is there a clear benefit for changing the internals to 32-bit? I assume 64-bit operations are just as efficient when the Wasm execution engine is running on a 64-bit machine (probably most servers).

Thanks.

[johanbrandhorst]

What is the plan for when 64-bit will be finally supported for the browser? Which additional port(s) will be added? What about when WASI starts supporting 64-bit too?

When the browser supports 64-bit addresses, I think the existing js/wasm port should be well positioned to take advantage of it, though it is not yet clear to me what the mechanism would be. There would be no need for an additional port for the js OS, in my opinion.

If the WASI ecosystem starts moving towards 64 bit (which there is little interest in, as far as I can tell, though I don't have any hard sources on that) the existing wasip1/wasm port would be well suited to target that environment. This is why the potential deprecation of this port would have to be considered in a separate proposal, as I think we'd need to very clearly prove that 64 bit WASI hosts are not going to be relevant to users to make such a change.

I guess this is probably true. Some supporting links would be great. It is unlikely for the server side to have a plan to go to 64-bit?

I would love to provide some supporting links but unfortunately I've only heard this from various people working closer to the runtimes, so I don't know that I have any concrete evidence for it. Wasmtime does support 64 bit addresses behind a flag. Regardless, all runtimes use 32 bit by default today, so there is a gap in the user experience when using Go. As mentioned before, we'd keep the 64 bit arch around and only seriously consider deprecating it if we knew for sure 64 bit runtimes were not going to become a standard in the ecosystem.

Integers are interesting. Wasm support 64-bit integer operations. I assume we'll continue to use those for 64-bit integer operations (Go's int64 and uint64), which should be more efficient than decomposing to 32-bit operations? Will int and uint become 32-bit? Do int64 and uint64 have 4-byte alignment (like we have for other 32-bit architectures)?

int, uint and uintptr would become 32 bit. I've updated the proposal. I'm not sure yet about the 64 bit integer type alignments, as Evan says we're still experimenting with the implementation.

[evanphx]

Hi @cherrymui

I think for the Go side we may also want to use explicitly sized integer types like uint32 when mapping the system APIs.
Yup! We're doing that today, we only use explicitly sized int types when interfacing with WASI.

Currently on all platforms that the gc toolchain supports, int and uintptr have the same width ...
A good call out, we'll look to keep that constant by making int 32bits wide.

So should we choose to align them to 8-byte in Go?
Another great call out, we had only been scratching around the edges of this. The idea of doing the wasm32 port is to bring Go's ABI into sync with other wasm languages, so we probably should 8-byte align them. We'll do a little research to double check what clang and rust are doing here as well.

Do you plan to switch to 32-bit "registers"?
The registers we use are wasm locals, which do hold a type. But nicely, each local can have it's own different type, just like locals in a normal programming language. This means we can easily experiment with using mixed locals. But I expect, all and all, that leaving the locals as uint64 will be the right thing to do, so that 64bit ops are nice and obvious, and 32bit ops overlay correctly.

One thing we'll have to double check (and I'm pretty sure about this) is that 32bit integer ops will wrap around correctly at the 32bit boundary even when a 64bit local is involved.

[joeshaw]

Currently on all platforms that the gc toolchain supports, int and uintptr have the same width (although the spec doesn't require it and gccgo does support platforms where they differ). So it probably makes sense to define int and uint 32-bit. (The proposal probably needs to mention it.)

FYI, in TinyGo's WASI support, both int and uintptr are 32 bits.

[cherrymui]

Thanks. SGTM overall. We could determine the compiler's implementation detail later (e.g. on a CL, as it is internal to the compiler).

One more question: it sounds to me the issue is mostly about interacting with the system API. Besides that, the current 64-bit wasip1/wasm port works reasonably well? Instead of a whole new port, would it be possible to solve it in a API level? Like, add a syscall/wasm32.Ptr[T] that maps to 32-bit pointers on the Wasm side, with a helper function to do the conversion (say wasm32.P converts a *T to Ptr[T])? And make wasmimport directive accept wasm32.Ptr. We could also make the GC do the right thing for tracking wasm32.Ptr. This way, one can write wasm32.P(unsafe.SliceData(buf)), which is slightly more code than without it, but maybe not too bad? Is this too much boilerplate?

I guess passing (pointer to) struct is still tricky. But I think that is a tricky problem anyway. On other platforms, when interacting with the system API or C, we still need to ensure the data are laid out the same on both side. In general we don't want to assume a Go struct has identical layout as a C struct. We can do the same for here. And it is usually only needed in a few low-level packages. With the current wasm port, where int64 and uint64 have the alignment matching WASI API, with the addition of the Ptr type, it probably shouldn't be hard to write Go data structures that match the WASI layout.

What is the down side of handling it in an API level?

Thanks.

[joeshaw]

One more question: it sounds to me the issue is mostly about interacting with the system API. Besides that, the current 64-bit wasip1/wasm port works reasonably well? Instead of a whole new port, would it be possible to solve it in a API level? Like, add a syscall/wasm32.Ptr[T] that maps to 32-bit pointers on the Wasm side, with a helper function to do the conversion (say wasm32.P converts a *T to Ptr[T])? And make wasmimport directive accept wasm32.Ptr. We could also make the GC do the right thing for tracking wasm32.Ptr. This way, one can write wasm32.P(unsafe.SliceData(buf)), which is slightly more code than without it, but maybe not too bad? Is this too much boilerplate?

This is what we've done in the Fastly Compute Go SDK, except to uint32 as that's what's currently required: https://github.com/fastly/compute-sdk-go/blob/c3a63de93dcb2cf090f431846d601c1302886c3e/internal/abi/prim/prim.go#L28-L34

Previously we were passing pointers to structs (in TinyGo), but this was a largely mechanical change and the use of generics does offer us the type safety we would have otherwise lost going straight to uint32.

As for whether a port is necessary or not, it's unfortunate that every pointer wastes 4 bytes, especially as these serverless compute platforms are fairly limited in terms of memory.