Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Panic in etcd gateway start #13810

Closed
tskarman opened this issue Mar 16, 2022 · 10 comments · Fixed by #13824
Closed

Panic in etcd gateway start #13810

tskarman opened this issue Mar 16, 2022 · 10 comments · Fixed by #13824

Comments

@tskarman
Copy link

tskarman commented Mar 16, 2022

What happened?

A panic happens when running etcd gateway start --discovery-srv someplace.com --listen-addr=127.0.5.6:23790 --trusted-ca-file somepath/ca.pem in v3.5.2 where no panic happens when running it in v3.4.18.

What did you expect to happen?

No panic, establish connection without error or fail with error message.

How can we reproduce it (as minimally and precisely as possible)?

The above command on Windows would do the trick.
I can give you the CA file and the DNS name privately. Just tell me where to send the information.

Anything else we need to know?

  • When specifying --insecure-discovery the panic does not occur and things operate normally.
  • The problem does not occur in v3.4.18
  • The cluster etcd version is v3.4.18
  • I tried setting GODEBUG=x509sha1=1,x509ignoreCN=0,http2debug=2 and combinations thereof - no change and not further diagnostic output. Some of those options prob. even aren't valid anymore.

Etcd version (please run commands below)

$ etcd --version
etcd Version: 3.5.2
Git SHA: 99018a77b
Go Version: go1.16.3
Go OS/Arch: windows/amd64

$ etcdctl version
etcdctl version: 3.5.2
API version: 3.5

Etcd configuration (command line flags or environment variables)

gateway start --discovery-srv someplace.com --listen-addr=127.0.5.6:23790 --trusted-ca-file somepath/ca.pem

Etcd debug information (please run commands blow, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
+------------------+---------+-------+----------------------------------+----------------------------------+------------+
|        ID        | STATUS  | NAME  |          PEER ADDRS              |         CLIENT ADDRS             | IS LEARNER |
+------------------+---------+-------+----------------------------------+----------------------------------+------------+
| 237a24d9080da0db | started | etcd4 | https://etcd4.someplace.com:2380 | https://etcd4.someplace.com:2379 |      false |
| 42b03093b37f16a9 | started | etcd5 | https://etcd5.someplace.com:2380 | https://etcd5.someplace.com:2379 |      false |
| 6aea52aba38d03f1 | started | etcd3 | https://etcd3.someplace.com:2380 | https://etcd3.someplace.com:2379 |      false |
| a8cf2d8a635f6a96 | started | etcd2 | https://etcd2.someplace.com:2380 | https://etcd2.someplace.com:2379 |      false |
| f5f365ee667c3149 | started | etcd1 | https://etcd1.someplace.com:2380 | https://etcd1.someplace.com:2379 |      false |
+------------------+---------+-------+----------------------------------+----------------------------------+------------+
$ etcdctl --endpoints=<member list> endpoint status -w table
+----------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|           ENDPOINT               |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://etcd1.someplace.com:2379 | f5f365ee667c3149 |  3.4.18 |  9.5 MB |      true |      false |     11648 |   82909472 |           82909472 |        |
| https://etcd2.someplace.com:2379 | a8cf2d8a635f6a96 |  3.4.18 |  9.5 MB |     false |      false |     11648 |   82909472 |           82909472 |        |
| https://etcd3.someplace.com:2379 | 6aea52aba38d03f1 |  3.4.18 |  9.5 MB |     false |      false |     11648 |   82909472 |           82909472 |        |
| https://etcd4.someplace.com:2379 | 237a24d9080da0db |  3.4.18 |  9.5 MB |     false |      false |     11648 |   82909472 |           82909472 |        |
| https://etcd5.someplace.com:2379 | 42b03093b37f16a9 |  3.4.18 |  9.5 MB |     false |      false |     11648 |   82909472 |           82909472 |        |
+----------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

Relevant log output

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x0 pc=0x5920d9]

goroutine 1 [running]:
go.etcd.io/etcd/client/pkg/v3/transport.ValidateSecureEndpoints(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000038180, 0x17, ...)
        /tmp/etcd-release-3.5.2/etcd/release/etcd/client/pkg/transport/tls.go:37 +0x1d9
go.etcd.io/etcd/server/v3/etcdmain.discoverEndpoints(0xc00045a280, 0xc00003c100, 0xa, 0xc000038180, 0x17, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /tmp/etcd-release-3.5.2/etcd/release/etcd/server/etcdmain/util.go:63 +0x250
go.etcd.io/etcd/server/v3/etcdmain.startGateway(0xc000455180, 0xc00045a230, 0x0, 0x5)
        /tmp/etcd-release-3.5.2/etcd/release/etcd/server/etcdmain/gateway.go:105 +0x2a5
github.com/spf13/cobra.(*Command).execute(0xc000455180, 0xc00045a1e0, 0x5, 0x5, 0xc000455180, 0xc00045a1e0)
        /usr/local/google/home/siarkowicz/.gvm/pkgsets/go1.16.3/global/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:856 +0x2c2
github.com/spf13/cobra.(*Command).ExecuteC(0x187ac60, 0x60, 0xc0000a8000, 0xc00008ff48)
        /usr/local/google/home/siarkowicz/.gvm/pkgsets/go1.16.3/global/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:960 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
        /usr/local/google/home/siarkowicz/.gvm/pkgsets/go1.16.3/global/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:897
go.etcd.io/etcd/server/v3/etcdmain.Main(0xc0000dc000, 0x8, 0x8)
        /tmp/etcd-release-3.5.2/etcd/release/etcd/server/etcdmain/main.go:32 +0x85
main.main()
        /tmp/etcd-release-3.5.2/etcd/release/etcd/server/main.go:32 +0x4c
@tskarman
Copy link
Author

I'd be happy to supply more log output. I did not find a way to get more verbose output from etcd gateway start.
Just let me know what I can do to help. Thank you!

@eval-exec
Copy link
Contributor

I'd be happy to supply more log output. I did not find a way to get more verbose output from etcd gateway start. Just let me know what I can do to help. Thank you!

what is your go version and go env?

eval-exec added a commit to eval-exec/etcd that referenced this issue Mar 19, 2022
`ValidateSecureEndpoints()` should call `t.DialContext()` instead of `t.Dial()`, because `t.Dial` is `nil`
eval-exec added a commit to eval-exec/etcd that referenced this issue Mar 19, 2022
`ValidateSecureEndpoints()` should call `t.DialContext()` instead of `t.Dial()`, because `t.Dial` is `nil`
@tskarman
Copy link
Author

I'd be happy to supply more log output. I did not find a way to get more verbose output from etcd gateway start. Just let me know what I can do to help. Thank you!

what is your go version and go env?

The panics happens both on systems with and on systems without a Go installation. But I am guessing you are asking for a different reason, so I ran the commands you suggested on two of the affected systems that do have a Go installation:

System 1:

$ go version
go version go1.17.2 windows/amd64
$ go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\tobias\AppData\Local\go-build
set GOENV=C:\Users\tobias\AppData\Roaming\go\env
set GOEXE=.exe
set GOEXPERIMENT=
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\tobias\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\tobias\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=C:\Program Files\Go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=C:\Program Files\Go\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.17.2
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=NUL
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\tobias\AppData\Local\Temp\go-build4036475500=/tmp/go-build -gno-record-gcc-switches

System 2:

$ go version
go version go1.18 windows/amd64
$ go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\tobias\AppData\Local\go-build
set GOENV=C:\Users\tobias\AppData\Roaming\go\env
set GOEXE=.exe
set GOEXPERIMENT=
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\tobias\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\tobias\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=C:\Program Files\Go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=C:\Program Files\Go\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.18
set GCCGO=gccgo
set GOAMD64=v1
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=NUL
set GOWORK=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\tobias\AppData\Local\Temp\go-build4094784673=/tmp/go-build -gno-record-gcc-switches

I am flexible in regards to Go versions and can install other versions if that's needed.

eval-exec added a commit to eval-exec/etcd that referenced this issue Mar 22, 2022
`ValidateSecureEndpoints()` should call `t.DialContext()` instead of `t.Dial()`, because `t.Dial` is `nil`
eval-exec added a commit to eval-exec/etcd that referenced this issue Mar 23, 2022
`ValidateSecureEndpoints()` should call `t.DialContext()` instead of `t.Dial()`, because `t.Dial` is `nil`
eval-exec added a commit to eval-exec/etcd that referenced this issue Mar 23, 2022
`ValidateSecureEndpoints()` should call `t.DialContext()` instead of `t.Dial()`, because `t.Dial` is `nil`
serathius added a commit that referenced this issue Mar 24, 2022
Fix panic in etcd validate secure endpoints #13810
@tskarman
Copy link
Author

Thank you all! :-)

yankay pushed a commit to yankay/etcd that referenced this issue Mar 25, 2022
`ValidateSecureEndpoints()` should call `t.DialContext()` instead of `t.Dial()`, because `t.Dial` is `nil`
@ahrtr ahrtr mentioned this issue Aug 3, 2022
25 tasks
@sheyt0
Copy link
Contributor

sheyt0 commented Feb 17, 2023

Can the fix be ported to version 3.5?

@tskarman
Copy link
Author

tskarman commented Sep 8, 2023

Ah yes, this isn't actually fixed in the v3.5 branch.
So the issue I reported for v3.5.2 still presents in v3.5.9.

Any plans to get that merged? That would be great and the original intent.
I now had to resort to the --insecure-discovery (which still works as a workaround) as I didn't want to stay on v3.4 any longer.

@ahrtr ahrtr reopened this Sep 8, 2023
@fuweid fuweid self-assigned this Sep 9, 2023
@fuweid
Copy link
Member

fuweid commented Sep 9, 2023

@tskarman would you please provide the full log about this? I would like to help fix it. thanks

Updated:

the issue shows in windows platform?

etcd Version: 3.5.2
Git SHA: 99018a77b
Go Version: go1.16.3
Go OS/Arch: windows/amd64

@fuweid
Copy link
Member

fuweid commented Sep 9, 2023

Sorry. It just needs to backport. File pr right now

@fuweid
Copy link
Member

fuweid commented Sep 12, 2023

@tskarman the patch has been merged into 3.5. Closing it because 3.5.10 is coming soon.

@fuweid fuweid closed this as completed Sep 12, 2023
@tskarman
Copy link
Author

@tskarman the patch has been merged into 3.5. Closing it because 3.5.10 is coming soon.

Cool :-)

Thank you very much!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

Successfully merging a pull request may close this issue.

5 participants