storage: "stream error: stream ID x; INTERNAL_ERROR" when upgrading from Go 1.19.5 to 1.20 or 1.20.1

[garan82]

Client
Storage
https://pkg.go.dev/cloud.google.com/go/storage

Environment
GKE

Go Environment
$ go version
go version go1.20 linux/amd64

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home//.cache/go-build"
GOENV="/home//.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home//go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home//go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.20"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/usr/local/home//go.mod"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build824194778=/tmp/go-build -gno-record-gcc-switches"

When reading objects from GCS, we're getting a lots of "stream error: stream ID x; INTERNAL_ERROR" errors. It comes and goes, there is no clear pattern, essentially it's very similar to #784

Additional context
We have no problem with Go 1.19.5 & https://pkg.go.dev/cloud.google.com/go/storage 1.28 or 1.29. Once upgrading to Go 1.20 or Go 1.20.1, we encounter many GCS errors.

[tritone]

Thanks for reporting. The fact that it changes with an upgrade to the Go version makes me think that there is an issue in net/http, which the storage client uses internally.

Can you clarify which version of go you were using previously (1.95 doesn't exist I think...)? Also, are you using http/1 or http/2 (default)?

I can escalate this to the Go language team, or take a look to see if any similar issues are reported there.

[garan82]

Sorry, I meant Go 1.19.5 I'm using default http2.

…

On Tue, Feb 21, 2023 at 3:07 PM Chris Cotter ***@***.***> wrote: Thanks for reporting. The fact that it changes with an upgrade to the Go version makes me think that there is an issue in net/http, which the storage client uses internally. Can you clarify which version of go you were using previously (1.95 doesn't exist I think...)? Also, are you using http/1 or http/2 (default)? I can escalate this to the Go language team, or take a look to see if any similar issues are reported there. — Reply to this email directly, view it on GitHub <#7440 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA7GVQ4JQLDOYRI2CEGFQ6DWYVDELANCNFSM6AAAAAAU64AR5Q> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[tritone]

Sounds good. I will look into it with the Go team. In the mean time, if you want to upgrade the language version, one option is to disable HTTP/2 (by setting GODEBUG=http2client=0)

[neild]

Does the error contain the string "; received from peer", as in "stream error: stream ID ####; INTERNAL_ERROR; received from peer"?

If so, this is an error produced by the GFE (the destination webserver), indicating an internal error on its part. Generally, GFEs generate this error when the backend (the GCS server behind the GFE) halts mid-response.

I've heard other reports of these errors showing up more with Go 1.20. I don't know what might have changed from 1.19 to affect this, but my current hypothesis is that we've always been receiving these errors but are surfacing them more frequently now for some reason. (For example, perhaps if we get an error and a clean stream termination at about the same time, maybe we're more likely to process the error now--although I haven't found any changes that seem like they'd cause that to happen.)

A reliable reproduction case would help a lot in narrowing down the root cause of the changed behavior on the Go end, at least. However, the actual error here seems to be on the remote end. (Assuming I'm right and this is a "received from peer" error.)

[tritone]

@garan82 it would also be helpful if you have a rough estimate on what percentage of the time these errors occur, and how large the files you are reading are. I am going to try to reproduce.

[garan82]

@neild yes the error contain "; received from peer".

@neild/@tritone: it doesn't happen consistently, hence we first thought it was intermittent.

• It happens with Seek, but not Read.
• Our GCS files are ~ 100MB-200MB.
• I'm not sure about the percentage of time this happens, still digging into that.

[mikewiacek]

To clarify, not Seek, but NewRangeReader

Our file format is somewhat analogous to SSTables, and we must read the index at the end of a file to efficiently find where the data we want within the larger file. So most operations on our end use NewRangeReader multiple times.

I've tried to reproduce this outside of our production env on GKE but to no avail. It's a sporadic issue. One theory I have is that it occurs more often when we're doing a large number of reads of objects from hundreds (or thousands) of simultaneous readers. We've hit on numerous occasions the 50Gbps read limit from GCS multi-region buckets. One hard to verify theory is that this is an artifact of that rate limiting with that 50Gbps limit manifesting in the GFEs in some way? It's a remote chance because it was the same with Go 1.19, but I'm out of ideas.