cmd/compile: replace CallImport with go:wasmimport directive

[neelance]

The wasm architecture currently uses a special CallImport assembler instruction to call function imports. This approach is quite inflexible and only compatible with the ABI used by wasm_exec.js.

The WebAssembly ecosystem is converging on using WASI (WebAssembly System Interface) as a standardized interface between the WebAssembly binary and its host (similar to system calls). I am working on adding support for WASI to Go. As a first step, Go needs to be able to use function imports with the ABI used by WASI.

For this reason I propose to replace CallImport with a new compiler directive go:wasmimport. It can be used like this:

//go:wasmimport __wasi_proc_exit wasi_unstable proc_exit
func __wasi_proc_exit(code int32)

By default go:wasmimport will use the ABI used by WASI. An optional parameter can be used for backwards compatibility with wasm_exec.js. This parameter is called abi0 because wasm_exec.js is reading the parameters form memory with Go's normal ABI0 layout:

//go:wasmimport wasmExit go runtime.wasmExit abi0
func wasmExit(code int32)

I do not plan to add other ABIs. On the contrary, the abi0 mode is supposed to go away once WASI can be used as the primary interface (it is still missing certain features).

The go:wasmimport directive will not be covered by Go's compatibility promise as long as the wasm architecture itself is not considered stable.

[rsc]

I'm confused about there being two different ABIs and also about what is optional and not. It would be nice if the optional fields were last, not in the middle of the list. Then you can tell what things mean by looking at just the argument count - you don't have to know that "go" is apparently a magic word and that when it is arg[2] it is not the real arg[2].

Could you please write the suggested docs describing the syntax in full? And maybe rotate the optional words to the end? It would be even better if there were not two different optional ABI words. Can it be reduced to one?

[neelance]

Right, I forgot to include a clear explanation of the arguments of the directive. The usage is like this:

//go:wasmimport localname importmodule importname [abi0]

localname is the name of the Go function, same as with //go:linkname.

importmodule and importname identify the function to import. WebAssembly functions always live within a module. For WASI this module is currently called wasi_unstable, see here. For wasm_exec.js it is called go, see

go/misc/wasm/wasm_exec.js

Lines 261 to 262 in aa3413c

	this.importObject = {
	go: {

.

[abi0] is the optional flag to indicate that this import is using the ABI0 way for passing arguments.

Hope this helps.

[rsc]

//go:linkname is kind of the odd one out for compiler directives (see go doc compile) in that it doesn't just implicitly apply to the next function declaration. Instead of adding another like that, it would probably be better to attach it to a function implicitly, like the others (example: //go:noinline).

Then instead of:

//go:wasmimport wasmExit go runtime.wasmExit abi0
func wasmExit(code int32)

you'd have

//go:wasmimport go runtime.wasmExit abi0
func wasmExit(code int32)

Is there always a function declaration to attach to?

The syntax would be down to:

//go:wasmimport importmodule importname [abi0]

I'm skeptical about exposing the name abi0. If the default is likely to be Go code, maybe use [wasi] as the optional mode? Or maybe if the importmodule is go, assume the Go ABI?
It still feels like there is more simplification possible here.

[neelance]

I fully agree that a version without localname would be better. In fact this was my initial plan.

However, all the current directives that apply to the next function are just flags without any parameters. They get processed by the pragmaValue function in lex.go and the return type is type Pragma uint16. No arguments possible.

This is why I opted for not doing a large change to Go's lexer/parser and instead copied a solution that already exists: //go:linkname.

About the abi flag: The default should be the abi that wasi uses, since it is a more native way to pass arguments to a WebAssembly function. The abi0 flag only needs to exist so I don't have to change everything in a massive CL. This way I can still support the abi that wasm_exec.js uses and slowly migrate to wasi. Having a special case for the importmodule go is possible, sure, but such implicit behavior change seems bad to me.

[rsc]

However, all the current directives that apply to the next function are just flags without any parameters. They get processed by the pragmaValue function in lex.go and the return type is type Pragma uint16. No arguments possible.

That was true a couple weeks ago but is no longer true (I just changed that, in preparation for some other possible //go: directives). So let's make the proposal what we both want, that it applies to the next function. I'm happy to work with you off-issue if you need help making that work.

So now we're just down to the ABI flag. The problem with abi0 is that we have not yet exposed that name and don't really plan to. I would rather refer to it as the go ABI. And it seems kind of redundant to have to say that the go importmodule uses the go ABI. Will other importmodules use that same ABI?

I'm also a little confused about "The abi0 flag only needs to exist so I don't have to change everything in a massive CL." since the //go:wasmimport directive doesn't exist today.

@neelance is it the case that long term nothing will say abi0? That is, will it be that some future Go release removes support for the abi0 annotation?

/cc @aclements how do you feel about exposing the name abi0 here?

[neelance]

So let's make the proposal what we both want, that it applies to the next function.

Great!

Will other importmodules use that same ABI?

They should not.

is it the case that long term nothing will say abi0? That is, will it be that some future Go release removes support for the abi0 annotation?

Yes, that's the long term plan, but I can't say when this will be the case. It depends a lot on how fast the development of WASI goes until it has enough features to completely replace the custom interface that wasm_exec.js currently uses.

I don't mind picking a different flag name than abi0. Making it implicit with the name of the importmodule feels a bit bad to me, but I do not strongly oppose it either.

[aclements]

I'm actually fine with exposing the name abi0. The intent of calling them ABI0 and ABIInternal was that we could expose the name "ABI0" if we ever froze another, faster ABI as ABI1. I'm not sure if we'd want to spell it "abi0" or "ABI0" given that it is an initialism.

However, it actually seems more reasonable to me to assume everything in the "go" module follows the "Go ABI" and omit the ABI from the directive. If I understand correctly, this is effectively a "closed" module for wasm_exec.js's exports and users can't define new abi0 interfaces in either this module or any other module, so the set of things that use abi0 is fixed and bounded and exactly those in the "go" module.

[neelance]

All right, I agree that it is good to discourage anyone else from using abi0 by exposing it less. Shall I edit the proposal in the first post or add a new version below?

[rsc]

@neelance, please add a new comment below, so it's easier to see the history. Thanks.

[neelance]

Here is the new full proposal text:

---

The wasm architecture currently uses a special CallImport assembler instruction to call function imports. This approach is quite inflexible and only compatible with the ABI used by wasm_exec.js.

The WebAssembly ecosystem is converging on using WASI (WebAssembly System Interface) as a standardized interface between the WebAssembly binary and its host (similar to system calls). I am working on adding support for WASI to Go. As a first step, Go needs to be able to use function imports with the ABI used by WASI.

For this reason I propose to replace CallImport with a new compiler directive go:wasmimport. It can be used like this:

//go:wasmimport importmodule importname

Concrete example:

//go:wasmimport wasi_unstable proc_exit
func __wasi_proc_exit(code int32)

importmodule and importname identify the function to import. WebAssembly functions always live within a module. For WASI this module is currently called wasi_unstable, see here. For wasm_exec.js it is called go, see

go/misc/wasm/wasm_exec.js

Lines 261 to 262 in aa3413c

	this.importObject = {
	go: {

By default go:wasmimport will use the ABI used by WASI. There is a special case for backwards compatibility with wasm_exec.js: If importmodule is go, then the arguments will get passed in a way so wasm_exec.js can read them from memory with Go's ABI0 layout. This special case will be removed once the interface of wasm_exec.js has been fully superseded by WASI (currently it is still missing certain features).

The go:wasmimport directive will not be covered by Go's compatibility promise as long as the wasm architecture itself is not considered stable.

[aykevl]

From the TinyGo side, I fully support this proposal. Right now we use something like this (following your example):

//go:wasm-module wasi_unstable
//export proc_exit
func __wasi_proc_exit(code int32)

It works as a stopgap, but if this proposal is accepted I'll make sure TinyGo will switch over to the new format.

I have one comment:

By default go:wasmimport will use the ABI used by WASI.

What is this ABI? Is it specified somewhere? It seems to me that it's really just the C ABI and if so, I think it should be specified as such. Some related discussions:

• Compute layout of datatypes according to C ABI WebAssembly/WASI#181
• Pass small structs in parameters instead of memory WebAssembly/tool-conventions#88

[neelance]

On the lowest level the ABI consists of passing data via arguments (on the WebAssembly stack) to WebAssembly's call instruction. There may be higher level additions in the future, such as a standardized way of how to pass strings, but currently this seems to be still in development. So by "ABI used by WASI" I really mean whatever WASI is doing right now and in the future.

[aykevl]

On the lowest level the ABI consists of passing data via arguments (on the WebAssembly stack) to WebAssembly's call instruction.

Nitpick: ABI is much more than a calling convention! It also involves struct layout, alignment, etc. (relevant when passing pointers around).

But in general, because you can use this facility for more than just WASI (e.g. -buildmode=c-shared) I think it's important to specify the things that WASI doesn't specify. For example, the WASI ABI passes all structs by reference, sidestepping the issue whether small structs may be passed directly in (virtual) registers instead of on the stack. Go will have to pick a side here. Therefore I would suggest investigating which ABI is used by Clang or Rust and following that.

[neelance]

My goal with go:wasmimport is to support WASI and it will do whatever WASI needs now or in the future. I think this is very reasonable, because WASI will set certain standards in the WebAssembly ecosystem on how to pass data between WebAssembly modules. This may include other aspects than the calling convention.

I do not want to define some "WebAssembly interface for Go" here, since I believe this is too early and Go should rather follow the WebAssembly ecosystem (namely WASI) than to diverge with its own interface. So if there is some concern about other people using this interface, then I would be okay with restricting go:wasmimport to the runtime and syscall/js packages. If people really want to experiment in other contexts, then they would need to patch the Go compiler first.

[rsc]

Everyone seems in favor of what we've converged on. Thanks for the good discussion.

Based on the discussion, this seems like a likely accept.

[cherrymui]

Sorry about being late.

Does go:wasmimport work for arbitrary modules and functions, or only for a list of special functions?

A bit bikeshedding: is "go" the best name for wasm_exec.js? From the Go code side, everything is go except the imported functions. It is true that the module is currently named "go", but we can change that. Maybe "jsgo" or "gojs"?

[neelance]

Does go:wasmimport work for arbitrary modules and functions, or only for a list of special functions?

There is no list. The parameter types need to be supported. Like I said above, I would be okay with restricting it to runtime and syscall/js if people are worried about usage outside of the Go project itself.

A bit bikeshedding: is "go" the best name for wasm_exec.js?

I am fine with changing this name. No preference on my end.

[rsc]

Replying to the previous two comments:

• It certainly seems reasonable to start with restricting go:wasmimport to runtime and syscall/js, to make sure we have time to correct mistakes.

• @cherrymui suggested changing the importmodule field from go to jsgo or gojs. @neelance said fine either way. Any other opinions? If we do restrict go:wasmimport to just runtime and syscall/js, it will be easily changed.

Leaving as likely accept for another week but we seem to have converged.

[cherrymui]

Thanks. I also think it is reasonable to start with restricting it to runtime and syscall/js. We can make it more general later if necessary.

[rsc]

No change in consensus, so accepted.

[gopherbot]

Change https://go.dev/cl/463018 mentions this issue: all: implement wasmimport directive

[mdempsky]

Should a malformed //go:wasmimport directive be a compiler error even when GOARCH is not wasm? Or should it just be silently ignored by the compiler?

[johanbrandhorst]

I don't know if there's any precedent but I'd like to see it always error.

[achille-roussel]

Hello!

I am following up here on a topic we discussed live among the group working on this issue, and believe we need input to make the right decision.

The question that came up was how to deal with //go:wasmimport functions which accept pointers as arguments in relation with escape analysis?

The WebAssembly type system also only knows about integers and floats, it does not have the notion of pointers. Pointers are represented by 32 bits integers since the addressable space of WebAssemble is 4GiB.

However, the GC does not know that the underlying host is a WebAssembly host, and it assumes the //go:wasmimport functions (which do not have a body) may retain the pointers and therefore it places them on the heap.

We believe that the application can expect the host not to retain any pointers passed to those functions, since they cannot be tracked by its GC.

We considered adding //go:noescape to these function declarations, but since it is impossible for the runtime and application to negotiate the lifetime of pointers, it seems that these two statements should hold true:

• the host must never hold a pointer to the memory space of a WebAssembly application after it returned from a //go:wasmimport function call
• all //go:wasmimport functions should imply //go:noescape

Alternatively, we considered forbidding the use of pointer types in the signature of //go:wasmimport functions. However, this forces the use of unsafe to convert Go pointers to integers when calling these functions. It also likely requires the application code to think about objects lifetime and add the necessary calls to runtime.KeepAlive (given, there are no threads so the application cannot do GC while it's calling to the host, but it seems like fertile ground for breaking programs if/when WebAssembly gets threads?).

[james-lawrence]

I believe the two assumptions are true. is the usecase that the host function uses the pointer to know what offset to access in the wasm memory?

Alternatively, we considered forbidding the use of pointer types in the signature of //go:wasmimport functions

you can always relax such a restriction later once the problem space is better understood. so it might be worth assuming go:noescape and prohibiting pointers for the time being. get the rest of the work in place and then revisit once people are able to use and experiment with wasi.

[achille-roussel]

I believe the two assumptions are true. is the usecase that the host function uses the pointer to know what offset to access in the wasm memory?

Yes, in what I have experienced with pointers are passed so the host can read or write a buffer in the program memory (e.g. for I/O, or output parameters of syscalls like stat).