misc/wasm, cmd/link: Go 1.17.2 causes WASM builds to throw `command line too long` with many environment variables

[evanw]

What version of Go are you using (go version)?

$ go version
go version go1.17.2 darwin/amd64

Does this issue reproduce with the latest release?

Yes. It's caused by the latest release.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/evan/Library/Caches/go-build"
GOENV="/Users/evan/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/evan/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/evan/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.17.2"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/evan/dev/esbuild/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/j6/np400cw17sz0n5ljd67byrzw0000gn/T/go-build2118346093=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Here's a simple way to reproduce the issue:

$ echo 'package main;import"fmt";func main(){fmt.Println(0)}' > main.go
$ GOOS=js GOARCH=wasm go build -o main.wasm main.go
$ X=longlonglonglong && X="$X$X$X$X" && X="$X$X$X$X" && X="$X$X$X$X" && X="$X$X$X$X" node $(go env GOROOT)/misc/wasm/wasm_exec.js main.wasm

What did you expect to see?

This should print 0.

What did you see instead?

This fails with the following error:

Error: command line too long
    at global.Go.run (/usr/local/go/misc/wasm/wasm_exec.js:574:11)
    at /usr/local/go/misc/wasm/wasm_exec.js:630:14

There are at least two problems:

1. With Go 1.17.2, it's now impossible to use Go WASM programs in situations with many environment variables. This comes up when trying to run something via GitHub Actions, for example.

2. The error message command line too long is confusing because there are no command-line arguments. Instead the problem is that there are too many environment variables.

See also

• security: fix CVE-2021-38297 #48797
• https://go-review.googlesource.com/c/go/+/354571/

[evanw]

I just discovered that this also impacts running tests for the same reason:

$ echo 'package main;import"testing";func TestFoo(*testing.T){}' > main_test.go
$ X=longlonglonglong && X="$X$X$X$X" && X="$X$X$X$X" && X="$X$X$X$X" && X="$X$X$X$X" PATH="$(go env GOROOT)/misc/wasm:$PATH" GOOS=js GOARCH=wasm go test main_test.go
Error: command line too long
    at global.Go.run (/usr/local/go/misc/wasm/wasm_exec.js:574:11)
    at /usr/local/go/misc/wasm/wasm_exec.js:630:14

[ALTree]

cc @rolandshoemaker @mknyszek

[dmitshur]

Also CC @cherrymui, @neelance.

[dmitshur]

Both command line arguments and environment variables are currently passed to WebAssembly by writing them to linear memory, and CL 354571 added a fixed limit of 4096 bytes for both.

The error message can certainly be improved to also mention environment variables.

I'm wondering if you can speak about how much more than 4096 bytes you're running into in your use case? Is it just a little over, or is it significantly over? Or is it unpredictable?

[evanw]

I'm the author of esbuild and I use GitHub Actions to run my Go WASM tests. Here's the full list of environment variables I'm seeing in my tests:

Click to expand the list of environment variables

_=/usr/bin/make
ACCEPT_EULA=Y
AGENT_TOOLSDIRECTORY=/opt/hostedtoolcache
ANDROID_HOME=/usr/local/lib/android/sdk
ANDROID_NDK_HOME=/usr/local/lib/android/sdk/ndk-bundle
ANDROID_NDK_LATEST_HOME=/usr/local/lib/android/sdk/ndk/23.0.7599858
ANDROID_NDK_ROOT=/usr/local/lib/android/sdk/ndk-bundle
ANDROID_SDK_ROOT=/usr/local/lib/android/sdk
ANT_HOME=/usr/share/ant
AZURE_EXTENSION_DIR=/opt/az/azcliextensions
BOOTSTRAP_HASKELL_NONINTERACTIVE=1
CHROME_BIN=/usr/bin/google-chrome
CHROMEWEBDRIVER=/usr/local/share/chrome_driver
CI=true
CONDA=/usr/share/miniconda
DEBIAN_FRONTEND=noninteractive
DEPLOYMENT_BASEPATH=/opt/runner
DOTNET_MULTILEVEL_LOOKUP=0
DOTNET_NOLOGO=1
DOTNET_SKIP_FIRST_TIME_EXPERIENCE=1
GECKOWEBDRIVER=/usr/local/share/gecko_driver
GITHUB_ACTION_REF=
GITHUB_ACTION_REPOSITORY=
GITHUB_ACTION=__run_19
GITHUB_ACTIONS=true
GITHUB_ACTOR=evanw
GITHUB_API_URL=https://api.github.com
GITHUB_BASE_REF=
GITHUB_ENV=/home/runner/work/_temp/_runner_file_commands/set_env_05e5de09-0c30-4b02-97e2-b15aa3c3b3ca
GITHUB_EVENT_NAME=push
GITHUB_EVENT_PATH=/home/runner/work/_temp/_github_workflow/event.json
GITHUB_GRAPHQL_URL=https://api.github.com/graphql
GITHUB_HEAD_REF=
GITHUB_JOB=esbuild
GITHUB_PATH=/home/runner/work/_temp/_runner_file_commands/add_path_05e5de09-0c30-4b02-97e2-b15aa3c3b3ca
GITHUB_REF=refs/heads/temp-go-wasm
GITHUB_REPOSITORY_OWNER=evanw
GITHUB_REPOSITORY=evanw/esbuild
GITHUB_RETENTION_DAYS=90
GITHUB_RUN_ATTEMPT=1
GITHUB_RUN_ID=1347838728
GITHUB_RUN_NUMBER=2312
GITHUB_SERVER_URL=https://github.com
GITHUB_SHA=8447ef9b4da93f3897dcb2f280d4181d5cd1f008
GITHUB_WORKFLOW=CI
GITHUB_WORKSPACE=/home/runner/work/esbuild/esbuild
GOROOT_1_14_X64=/opt/hostedtoolcache/go/1.14.15/x64
GOROOT_1_15_X64=/opt/hostedtoolcache/go/1.15.15/x64
GOROOT_1_16_X64=/opt/hostedtoolcache/go/1.16.9/x64
GOROOT_1_17_X64=/opt/hostedtoolcache/go/1.17.2/x64
GOROOT=/opt/hostedtoolcache/go/1.17.2/x64
GRAALVM_11_ROOT=/usr/local/graalvm/graalvm-ce-java11-21.2.0
GRADLE_HOME=/usr/share/gradle-7.2
HOME=/home/runner
HOMEBREW_CELLAR=/home/linuxbrew/.linuxbrew/Cellar
HOMEBREW_CLEANUP_PERIODIC_FULL_DAYS=3650
HOMEBREW_NO_AUTO_UPDATE=1
HOMEBREW_PREFIX=/home/linuxbrew/.linuxbrew
HOMEBREW_REPOSITORY=/home/linuxbrew/.linuxbrew/Homebrew
HOMEBREW_SHELLENV_PREFIX=/home/linuxbrew/.linuxbrew
ImageOS=ubuntu20
ImageVersion=20211011.1
INVOCATION_ID=e89f572507984074a8ef4d7b4677e023
JAVA_HOME_11_X64=/usr/lib/jvm/adoptopenjdk-11-hotspot-amd64
JAVA_HOME_8_X64=/usr/lib/jvm/adoptopenjdk-8-hotspot-amd64
JAVA_HOME=/usr/lib/jvm/adoptopenjdk-11-hotspot-amd64
JOURNAL_STREAM=8:22524
LANG=C.UTF-8
LEIN_HOME=/usr/local/lib/lein
LEIN_JAR=/usr/local/lib/lein/self-installs/leiningen-2.9.7-standalone.jar
MAKEFLAGS=
MAKELEVEL=1
MFLAGS=
NVM_DIR=/home/runner/.nvm
PATH=/home/runner/.deno/bin:/opt/hostedtoolcache/deno/1.15.1/x64:/opt/hostedtoolcache/node/14.18.0/x64/bin:/home/runner/go/bin:/opt/hostedtoolcache/go/1.17.2/x64/bin:/home/linuxbrew/.linuxbrew/bin:/home/linuxbrew/.linuxbrew/sbin:/home/runner/.local/bin:/opt/pipx_bin:/usr/share/rust/.cargo/bin:/home/runner/.config/composer/vendor/bin:/usr/local/.ghcup/bin:/home/runner/.dotnet/tools:/snap/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
PERFLOG_LOCATION_SETTING=RUNNER_PERFLOG
PIPX_BIN_DIR=/opt/pipx_bin
PIPX_HOME=/opt/pipx
POWERSHELL_DISTRIBUTION_CHANNEL=GitHub-Actions-ubuntu20
PWD=/home/runner/work/esbuild/esbuild
RUNNER_NAME=GitHub Actions 2
RUNNER_OS=Linux
RUNNER_PERFLOG=/home/runner/perflog
RUNNER_TEMP=/home/runner/work/_temp
RUNNER_TOOL_CACHE=/opt/hostedtoolcache
RUNNER_TRACKING_ID=github_c55eb5e4-2f62-4ef4-87cb-caa86ebdb0aa
RUNNER_USER=runner
RUNNER_WORKSPACE=/home/runner/work/esbuild
SELENIUM_JAR_PATH=/usr/share/java/selenium-server-standalone.jar
SGX_AESM_ADDR=1
SHLVL=1
STATS_KEEPALIVE=false
SWIFT_PATH=/usr/share/swift/usr/bin
USER=runner
VCPKG_INSTALLATION_ROOT=/usr/local/share/vcpkg
XDG_CONFIG_HOME=/home/runner/.config

The text I pasted above comes in at 3919 bytes for 96 environment variables, so you'd need to add an extra 98*8=768 for the pointers for a total of 3919+768=4687 bytes just for environment variable data alone. So it's a little over 4096 bytes.

The maximum value of the offset variable in wasm_exec.js that I observe when running my tests is 9512, which is 5416 bytes of data. This is larger than 4687 bytes because I do have a few command-line arguments in my tests. And I'll probably need to go a little higher in the future if a future test needs another command-line argument, so it would be nice to have a little more wiggle room than exactly 5416 bytes.

[neelance]

I see no reason against increasing the limit from 4096 to 8192 bytes.

Later, after https://golang.org/cl/350737 landed, we can switch to the WASI interface for getting the arguments and environment which would remove the limit entirely.

[gopherbot]

Change https://golang.org/cl/358194 mentions this issue: cmd/link: increase reserved space for passing env on wasm