Tests fail on macos with go 1.18

[Llandy3d]

Brief summary

Trying to run tests with make tests on macos with golang 1.18 results in errors while the expected output is for them to complete.

The problem seems to not be present by using golang 1.17 and it seems related to his bug:
golang/go#52010

k6 version

3e21e67 <- commit hash, but it doesn't matter since the go version it's the problem

OS

MacOs Monterey 12.1

Docker version and image (if applicable)

No response

Steps to reproduce the problem

with go 1.18:

make tests

Expected behaviour

tests complete without errors.

Actual behaviour

❯ make tests
go test -race -timeout 210s ./...
?   	go.k6.io/k6	[no test files]
ok  	go.k6.io/k6/api	(cached)
?   	go.k6.io/k6/api/common	[no test files]
ok  	go.k6.io/k6/api/v1	(cached)
?   	go.k6.io/k6/api/v1/client	[no test files]
ok  	go.k6.io/k6/cloudapi	(cached)
ok  	go.k6.io/k6/cmd	19.669s
ok  	go.k6.io/k6/cmd/integration_tests/eventloop	(cached)
ok  	go.k6.io/k6/cmd/integration_tests/testmodules/events	(cached)
ok  	go.k6.io/k6/converter/har	(cached)
ok  	go.k6.io/k6/core	(cached)
ok  	go.k6.io/k6/core/local	(cached)
ok  	go.k6.io/k6/errext	(cached)
?   	go.k6.io/k6/errext/exitcodes	[no test files]
time="2022-06-27T12:54:58+02:00" level=info msg="{\"text\":\"nativeObject\"}" source=console
time="2022-06-27T12:54:58+02:00" level=warning msg="There were unknown fields in the options exported in the script" error="json: unknown field \"myOption\""
time="2022-06-27T12:54:58+02:00" level=warning msg="The moduleSpecifier \"buffer\" has no scheme but we will try to resolve it as remote module. This will be deprecated in the future and all remote modules will need to explicitly use \"https\" as scheme."
time="2022-06-27T12:54:58+02:00" level=info msg=1 source=console
time="2022-06-27T12:54:58+02:00" level=warning msg="The moduleSpecifier \"buffer\" has no scheme but we will try to resolve it as remote module. This will be deprecated in the future and all remote modules will need to explicitly use \"https\" as scheme."
time="2022-06-27T12:55:00+02:00" level=warning msg="Couldn't load source map for file:///script.js" error="The moduleSpecifier \"file:///test.min.js.map\" couldn't be found on local disk. Make sure that you've specified the right path to the file. If you're running k6 using the Docker image make sure you have mounted the local directory (-v /local/path/:/inside/docker/path) containing your script and modules so that they're accessible by k6 from inside of the container, see https://k6.io/docs/using-k6/modules#using-local-modules-with-docker."
time="2022-06-27T12:55:00+02:00" level=warning msg="Couldn't load source map for file:///script.js" error="The moduleSpecifier \"file:///test.min.js.map\" couldn't be found on local disk. Make sure that you've specified the right path to the file. If you're running k6 using the Docker image make sure you have mounted the local directory (-v /local/path/:/inside/docker/path) containing your script and modules so that they're accessible by k6 from inside of the container, see https://k6.io/docs/using-k6/modules#using-local-modules-with-docker."
time="2022-06-27T12:55:03+02:00" level=info msg="{\"text\":\"test1\"}" source=console
time="2022-06-27T12:55:04+02:00" level=warning msg="Request Failed" error="Get \"http://httpbin.local:59824/html\": lookup httpbin.local: no such host"
time="2022-06-27T12:55:04+02:00" level=info msg="{\"text\":\"test2\"}" source=console
time="2022-06-27T12:55:05+02:00" level=info msg="{\"text\":\"test3\"}" source=console
time="2022-06-27T12:55:11+02:00" level=warning msg="Request Failed" error="Get \"http://127.0.0.1:1\": dial tcp 127.0.0.1:1: connect: connection refused"
time="2022-06-27T12:55:11+02:00" level=warning msg="Request Failed" error="Get \"http://127.0.0.1:1\": dial tcp 127.0.0.1:1: connect: connection refused"
time="2022-06-27T12:55:11+02:00" level=info msg=1 source=console
time="2022-06-27T12:55:12+02:00" level=warning msg="The moduleSpecifier \"buffer\" has no scheme but we will try to resolve it as remote module. This will be deprecated in the future and all remote modules will need to explicitly use \"https\" as scheme."
time="2022-06-27T12:55:12+02:00" level=warning msg="The moduleSpecifier \"buffer\" has no scheme but we will try to resolve it as remote module. This will be deprecated in the future and all remote modules will need to explicitly use \"https\" as scheme."
--- FAIL: TestVUIntegrationClientCerts (0.00s)
    --- FAIL: TestVUIntegrationClientCerts/VerifyServerCert (0.00s)
        --- FAIL: TestVUIntegrationClientCerts/VerifyServerCert/Source (0.01s)
            runner_test.go:1611:
                	Error Trace:	runner_test.go:1611
                	Error:      	"GoError: Get \"https://127.0.0.1:59828\": x509: “127.0.0.1:6969” certificate is not standards compliant\n\tat go.k6.io/k6/js/modules/k6/http.(*RootModule).NewModuleInstance.func2 (native)\n\tat file:///script.js:6:17(4)\n\tat native\n" does not contain "certificate signed by unknown authority"
                	Test:       	TestVUIntegrationClientCerts/VerifyServerCert/Source
        --- FAIL: TestVUIntegrationClientCerts/VerifyServerCert/Archive (0.01s)
            runner_test.go:1611:
                	Error Trace:	runner_test.go:1611
                	Error:      	"GoError: Get \"https://127.0.0.1:59828\": x509: “127.0.0.1:6969” certificate is not standards compliant\n\tat go.k6.io/k6/js/modules/k6/http.(*RootModule).NewModuleInstance.func2 (native)\n\tat file:///script.js:6:17(4)\n\tat native\n" does not contain "certificate signed by unknown authority"
                	Test:       	TestVUIntegrationClientCerts/VerifyServerCert/Archive
--- FAIL: TestVUIntegrationInsecureRequests (0.00s)
    --- FAIL: TestVUIntegrationInsecureRequests/False (0.00s)
        --- FAIL: TestVUIntegrationInsecureRequests/False/Source (0.53s)
            runner_test.go:877:
                	Error Trace:	runner_test.go:877
                	Error:      	"GoError: Get \"https://expired.badssl.com/\": x509: “*.badssl.com” certificate is expired\n\tat go.k6.io/k6/js/modules/k6/http.(*RootModule).NewModuleInstance.func2 (native)\n\tat file:///script.js:3:37(4)\n\tat native\n" does not contain "x509: certificate has expired or is not yet valid"
                	Test:       	TestVUIntegrationInsecureRequests/False/Source
        --- FAIL: TestVUIntegrationInsecureRequests/False/Archive (0.48s)
            runner_test.go:877:
                	Error Trace:	runner_test.go:877
                	Error:      	"GoError: Get \"https://expired.badssl.com/\": x509: “*.badssl.com” certificate is expired\n\tat go.k6.io/k6/js/modules/k6/http.(*RootModule).NewModuleInstance.func2 (native)\n\tat file:///script.js:3:37(4)\n\tat native\n" does not contain "x509: certificate has expired or is not yet valid"
                	Test:       	TestVUIntegrationInsecureRequests/False/Archive
    --- FAIL: TestVUIntegrationInsecureRequests/Null (0.00s)
        --- FAIL: TestVUIntegrationInsecureRequests/Null/Source (0.40s)
            runner_test.go:877:
                	Error Trace:	runner_test.go:877
                	Error:      	"GoError: Get \"https://expired.badssl.com/\": x509: “*.badssl.com” certificate is expired\n\tat go.k6.io/k6/js/modules/k6/http.(*RootModule).NewModuleInstance.func2 (native)\n\tat file:///script.js:3:37(4)\n\tat native\n" does not contain "x509: certificate has expired or is not yet valid"
                	Test:       	TestVUIntegrationInsecureRequests/Null/Source
        --- FAIL: TestVUIntegrationInsecureRequests/Null/Archive (0.50s)
            runner_test.go:877:
                	Error Trace:	runner_test.go:877
                	Error:      	"GoError: Get \"https://expired.badssl.com/\": x509: “*.badssl.com” certificate is expired\n\tat go.k6.io/k6/js/modules/k6/http.(*RootModule).NewModuleInstance.func2 (native)\n\tat file:///script.js:3:37(4)\n\tat native\n" does not contain "x509: certificate has expired or is not yet valid"
                	Test:       	TestVUIntegrationInsecureRequests/Null/Archive
FAIL
FAIL	go.k6.io/k6/js	32.570s
ok  	go.k6.io/k6/js/common	(cached)
ok  	go.k6.io/k6/js/compiler	(cached)
ok  	go.k6.io/k6/js/eventloop	(cached)
?   	go.k6.io/k6/js/modules	[no test files]
ok  	go.k6.io/k6/js/modules/k6	(cached)
ok  	go.k6.io/k6/js/modules/k6/crypto	(cached)
ok  	go.k6.io/k6/js/modules/k6/crypto/x509	(cached)
ok  	go.k6.io/k6/js/modules/k6/data	(cached)
ok  	go.k6.io/k6/js/modules/k6/encoding	(cached)
ok  	go.k6.io/k6/js/modules/k6/execution	(cached)
?   	go.k6.io/k6/js/modules/k6/experimental	[no test files]
2022/06/27 12:54:50 http: TLS handshake error from 127.0.0.1:59660: remote error: tls: bad certificate
--- FAIL: TestClient (0.00s)
    --- FAIL: TestClient/BadTLS (1.00s)
        client_test.go:702:
            	Error Trace:	client_test.go:702
            	            				client_test.go:745
            	Error:      	"GoError: context deadline exceeded: connection error: desc = \"transport: authentication handshake failed: x509: “Acme Co” certificate is not trusted\" at reflect.methodValueCall (native)" does not contain "certificate signed by unknown authority"
            	Test:       	TestClient/BadTLS
FAIL
FAIL	go.k6.io/k6/js/modules/k6/grpc	1.624s
ok  	go.k6.io/k6/js/modules/k6/html	(cached)
?   	go.k6.io/k6/js/modules/k6/html/gen	[no test files]
ok  	go.k6.io/k6/js/modules/k6/http	(cached)
ok  	go.k6.io/k6/js/modules/k6/metrics	(cached)
ok  	go.k6.io/k6/js/modules/k6/ws	(cached)
?   	go.k6.io/k6/js/modulestest	[no test files]
ok  	go.k6.io/k6/js/tc39	(cached)
ok  	go.k6.io/k6/lib	(cached)
?   	go.k6.io/k6/lib/consts	[no test files]
ok  	go.k6.io/k6/lib/executor	(cached)
ok  	go.k6.io/k6/lib/fsext	(cached)
ok  	go.k6.io/k6/lib/netext	(cached)
ok  	go.k6.io/k6/lib/netext/grpcext	(cached)
time="2022-06-27T12:54:50+02:00" level=warning msg="Request Failed" error="Get \"http://127.0.0.1:59431\": dial: i/o timeout"
2022/06/27 12:54:50 http: TLS handshake error from 127.0.0.1:59424: remote error: tls: bad certificate
2022/06/27 12:54:50 http2: Transport received unsolicited DATA frame; closing connection
2022/06/27 12:54:50 http2: panic serving 127.0.0.1:59434: expected internal error
goroutine 94 [running]:
net/http.(*http2serverConn).runHandler.func1()
	/opt/homebrew/Cellar/go/1.18.1/libexec/src/net/http/h2_bundle.go:5896 +0x1d0
panic({0x1014a5900, 0x1015ae1d0})
	/opt/homebrew/Cellar/go/1.18.1/libexec/src/runtime/panic.go:844 +0x258
go.k6.io/k6/lib/netext/httpext.TestHTTP2StreamError.func1({0x1015b5550, 0xc000596008}, 0xc000162a38?)
	/Users/llandy/dev/k6/lib/netext/httpext/error_codes_test.go:210 +0x134
net/http.HandlerFunc.ServeHTTP(0x1015abcf8, {0x1015b5550, 0xc000596008}, 0x1012b8b50?)
	/opt/homebrew/Cellar/go/1.18.1/libexec/src/net/http/server.go:2084 +0x50
net/http.(*ServeMux).ServeHTTP(0x0?, {0x1015b5550, 0xc000596008}, 0xc0000d6b00)
	/opt/homebrew/Cellar/go/1.18.1/libexec/src/net/http/server.go:2462 +0xa4
go.k6.io/k6/lib/testutils/httpmultibin.NewHTTPMultiBin.func1({0x1015b5550, 0xc000596008}, 0xc0000d6b00)
	/Users/llandy/dev/k6/lib/testutils/httpmultibin/httpmultibin.go:315 +0xa8
net/http.HandlerFunc.ServeHTTP(0xc0006421e0, {0x1015b5550, 0xc000596008}, 0x1?)
	/opt/homebrew/Cellar/go/1.18.1/libexec/src/net/http/server.go:2084 +0x50
net/http.serverHandler.ServeHTTP({0xc00029e2a0?}, {0x1015b5550, 0xc000596008}, 0xc0000d6b00)
	/opt/homebrew/Cellar/go/1.18.1/libexec/src/net/http/server.go:2916 +0x6d0
net/http.initALPNRequest.ServeHTTP({{0x1015b5bf8?, 0xc00057ca80?}, 0xc000922380?, {0xc00029e2a0?}}, {0x1015b5550, 0xc000596008}, 0xc0000d6b00)
	/opt/homebrew/Cellar/go/1.18.1/libexec/src/net/http/server.go:3523 +0x298
net/http.(*http2serverConn).runHandler(0xc000768c00, 0xc000596008, 0x10101e080?, 0xc00000e078)
	/opt/homebrew/Cellar/go/1.18.1/libexec/src/net/http/h2_bundle.go:5903 +0x9c
created by net/http.(*http2serverConn).processHeaders
	/opt/homebrew/Cellar/go/1.18.1/libexec/src/net/http/h2_bundle.go:5633 +0x948
2022/06/27 12:54:50 http: TLS handshake error from 127.0.0.1:59444: remote error: tls: bad certificate
2022/06/27 12:54:50 http: TLS handshake error from 127.0.0.1:59450: remote error: tls: bad certificate
--- FAIL: TestX509UnknownAuthorityError (0.01s)
    error_codes_test.go:269:
        	Error Trace:	error_codes_test.go:269
        	Error:      	Not equal:
        	            	expected: 0x51e
        	            	actual  : 0x3e8
        	Test:       	TestX509UnknownAuthorityError
    error_codes_test.go:270:
        	Error Trace:	error_codes_test.go:270
        	Error:      	"x509: “Acme Co” certificate is not trusted" does not contain "x509: unknown authority"
        	Test:       	TestX509UnknownAuthorityError
time="2022-06-27T12:54:50+02:00" level=warning msg="Request Failed" error="Get \"http://127.0.0.1:59426\": request timeout"
time="2022-06-27T12:54:50+02:00" level=warning msg="Request Failed" error="request timeout"
2022/06/27 12:54:50 http: TLS handshake error from 127.0.0.1:59498: read tcp 127.0.0.1:59422->127.0.0.1:59498: read: connection reset by peer
2022/06/27 12:54:50 http: TLS handshake error from 127.0.0.1:59513: EOF
2022/06/27 12:54:50 http: TLS handshake error from 127.0.0.1:59565: EOF
2022/06/27 12:54:50 http: TLS handshake error from 127.0.0.1:59566: EOF
2022/06/27 12:54:50 http: TLS handshake error from 127.0.0.1:59567: EOF
2022/06/27 12:54:50 http: TLS handshake error from 127.0.0.1:59569: EOF
2022/06/27 12:54:50 http: TLS handshake error from 127.0.0.1:59571: EOF
2022/06/27 12:54:50 http: TLS handshake error from 127.0.0.1:59576: EOF
2022/06/27 12:54:50 http: TLS handshake error from 127.0.0.1:59581: EOF
2022/06/27 12:54:50 http: TLS handshake error from 127.0.0.1:59633: EOF
2022/06/27 12:54:50 http: TLS handshake error from 127.0.0.1:59659: read tcp 127.0.0.1:59422->127.0.0.1:59659: read: connection reset by peer
2022/06/27 12:54:50 http: TLS handshake error from 127.0.0.1:59668: EOF
2022/06/27 12:54:50 http: TLS handshake error from 127.0.0.1:59683: EOF
2022/06/27 12:54:50 http: TLS handshake error from 127.0.0.1:59687: EOF
2022/06/27 12:54:50 http: TLS handshake error from 127.0.0.1:59688: EOF
2022/06/27 12:54:50 http: TLS handshake error from 127.0.0.1:59690: read tcp 127.0.0.1:59422->127.0.0.1:59690: read: connection reset by peer
2022/06/27 12:54:50 http: TLS handshake error from 127.0.0.1:59693: EOF
2022/06/27 12:54:50 http: TLS handshake error from 127.0.0.1:59695: EOF
2022/06/27 12:54:50 http: TLS handshake error from 127.0.0.1:59703: EOF
2022/06/27 12:54:51 http: TLS handshake error from 127.0.0.1:59713: EOF
2022/06/27 12:54:51 http: TLS handshake error from 127.0.0.1:59722: EOF
2022/06/27 12:54:51 http: TLS handshake error from 127.0.0.1:59734: read tcp 127.0.0.1:59422->127.0.0.1:59734: read: connection reset by peer
2022/06/27 12:54:51 http: TLS handshake error from 127.0.0.1:59743: EOF
2022/06/27 12:54:51 http: TLS handshake error from 127.0.0.1:59746: EOF
2022/06/27 12:54:51 http: TLS handshake error from 127.0.0.1:59749: read tcp 127.0.0.1:59422->127.0.0.1:59749: read: connection reset by peer
2022/06/27 12:54:51 http: TLS handshake error from 127.0.0.1:59751: read tcp 127.0.0.1:59422->127.0.0.1:59751: read: connection reset by peer
2022/06/27 12:54:51 http: TLS handshake error from 127.0.0.1:59752: read tcp 127.0.0.1:59422->127.0.0.1:59752: read: connection reset by peer
2022/06/27 12:54:51 http: TLS handshake error from 127.0.0.1:59758: EOF
2022/06/27 12:54:51 http: TLS handshake error from 127.0.0.1:59769: EOF
2022/06/27 12:54:51 http: TLS handshake error from 127.0.0.1:59777: EOF
2022/06/27 12:54:51 http: TLS handshake error from 127.0.0.1:59789: read tcp 127.0.0.1:59422->127.0.0.1:59789: read: connection reset by peer
2022/06/27 12:54:51 http: TLS handshake error from 127.0.0.1:59792: EOF
FAIL
FAIL	go.k6.io/k6/lib/netext/httpext	1.430s
?   	go.k6.io/k6/lib/testutils	[no test files]
?   	go.k6.io/k6/lib/testutils/httpmultibin	[no test files]
?   	go.k6.io/k6/lib/testutils/httpmultibin/grpc_any_testing	[no test files]
?   	go.k6.io/k6/lib/testutils/minirunner	[no test files]
?   	go.k6.io/k6/lib/testutils/mockoutput	[no test files]
?   	go.k6.io/k6/lib/testutils/mockresolver	[no test files]
ok  	go.k6.io/k6/lib/types	(cached)
ok  	go.k6.io/k6/loader	(cached)
ok  	go.k6.io/k6/log	(cached)
ok  	go.k6.io/k6/metrics	(cached)
?   	go.k6.io/k6/metrics/engine	[no test files]
ok  	go.k6.io/k6/output	(cached)
ok  	go.k6.io/k6/output/cloud	(cached)
ok  	go.k6.io/k6/output/csv	(cached)
ok  	go.k6.io/k6/output/influxdb	(cached)
ok  	go.k6.io/k6/output/json	(cached)
ok  	go.k6.io/k6/output/statsd	(cached)
ok  	go.k6.io/k6/ui	(cached)
ok  	go.k6.io/k6/ui/pb	(cached)
FAIL
make: *** [tests] Error 1

[na--]

Thanks for opening this! The badssl tests are probably easy enough to fix by just changing the string that is checked, though it's probably better to take care of that in #2543? 🤔 Not sure why the panic happens though 😕

[na--]

We should probably also try re-enabling the automated CI tests on MacOS... 🤔 IIRC, we disabled them a long time ago because the MacOS runners in GitHub Actions had terrible performance that caused a bunch of tests to be flaky, but they might have fixed these issues, or we can disable a few of the flakiest tests on Mac and leave the rest running 🤷‍♂️

[codebien]

I see both the linked issues have been resolved and we don't support go1.18 anymore.

@Llandy3d is this still an issue for you with the current master? Can you try with go1.21, please? Otherwise, we may focus this issue on the CI automation for macOS.

[Llandy3d]

I see both the linked issues have been resolved and we don't support go1.18 anymore.

@Llandy3d is this still an issue for you with the current master? Can you try with go1.21, please? Otherwise, we may focus this issue on the CI automation for macOS.

Last try was on 1st September 2023 on go version go1.21.0 darwin/arm64. The issue was reproduced.

[jsolana]

Hi mates! Any news?

With go version go1.15.3 darwin/amd64 and k6 v0.48.0, the issue still persists.

--- FAIL: TestVUIntegrationClientCerts (0.00s)
    --- FAIL: TestVUIntegrationClientCerts/VerifyServerCert (0.00s)
        --- FAIL: TestVUIntegrationClientCerts/VerifyServerCert/Source (0.04s)
            runner_test.go:1855:
                	Error Trace:	/dev/k6/k6/js/runner_test.go:1855
                	Error:      	Error "GoError: Get \"https://127.0.0.1:50319\": tls: failed to verify certificate: x509: “127.0.0.1:6969” certificate is not standards compliant\n\tat go.k6.io/k6/js/modules/k6/http.(*RootModule).NewModuleInstance.func2 (native)\n\tat file:///script.js:7:25(4)\n" does not contain "certificate signed by unknown authority"
                	Test:       	TestVUIntegrationClientCerts/VerifyServerCert/Source
        --- FAIL: TestVUIntegrationClientCerts/VerifyServerCert/Archive (0.01s)
            runner_test.go:1855:
                	Error Trace:	/dev/k6/k6/js/runner_test.go:1855
                	Error:      	Error "GoError: Get \"https://127.0.0.1:50319\": tls: failed to verify certificate: x509: “127.0.0.1:6969” certificate is not standards compliant\n\tat go.k6.io/k6/js/modules/k6/http.(*RootModule).NewModuleInstance.func2 (native)\n\tat file:///script.js:7:25(4)\n" does not contain "certificate signed by unknown authority"
                	Test:       	TestVUIntegrationClientCerts/VerifyServerCert/Archive

Thanks!

[codebien]

Hey @jsolana,
go version go1.15.3 is no longer supported by k6. You should try using a go version >= 1.19. Do you see the same error with it?

[jsolana]

Sorry not sure how I wrote this value 😓
I ve just tested with go version go1.21.3 darwin/amd64

[oleiade]

Confirming the maintainters team was able to reproduce the issue on up to date MacOS (sonoma), M1 Pro architecture, and using the last Go version (go.1.21.3) indeed 👍🏻