panic on checkNegotiationNeeded

[tarrencev]

Seems there is a case where Track is nil while executing checkNegotiationNeeded. I don't think this should be possible, since the Senders direction is Send* but it doesn't have a track set. I think all senders in Send* state should have a track defined.

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x15caf3c]
goroutine 4980 [running]:
github.com/pion/webrtc/v3.(*RTPSender).Track(0x0, 0x0)
	/Users/tvanas/go/pkg/mod/github.com/pion/webrtc/v3@v3.0.0-beta.10/rtpsender.go:80 +0x3c
github.com/pion/webrtc/v3.(*PeerConnection).checkNegotiationNeeded(0xc00011eb60, 0x15dc500)
	/Users/tvanas/go/pkg/mod/github.com/pion/webrtc/v3@v3.0.0-beta.10/peerconnection.go:359 +0x91a
github.com/pion/webrtc/v3.(*PeerConnection).negotiationNeededOp(0xc00011eb60)
	/Users/tvanas/go/pkg/mod/github.com/pion/webrtc/v3@v3.0.0-beta.10/peerconnection.go:301 +0x90
github.com/pion/webrtc/v3.(*operations).start(0xc0009aa240)
	/Users/tvanas/go/pkg/mod/github.com/pion/webrtc/v3@v3.0.0-beta.10/operations.go:84 +0x61
created by github.com/pion/webrtc/v3.(*operations).Enqueue
	/Users/tvanas/go/pkg/mod/github.com/pion/webrtc/v3@v3.0.0-beta.10/operations.go:35 +0xcb
exit status 2

[tomsun]

Have had the same thing happen, possibly a few times

	2020-11-05T10:10:59.953+02:00	[2020-11-05 08:10:59.953] [INFO] [62][sfu.go][Handle] => Handling......
	2020-11-05T10:11:00.805+02:00	[2020-11-05 08:11:00.805] [INFO] [62][sfu.go][Handle] => Handling......
	2020-11-05T10:11:04.881+02:00	[2020-11-05 08:11:04.881] [INFO] [2041][server.go][ServeHTTP] => Closing peer
	2020-11-05T10:11:04.881+02:00	[2020-11-05 08:11:04.881] [ERROR] [180][simplesender.go][func1] => Error closing sender: InvalidStateError: connection closed
	2020-11-05T10:11:04.885+02:00	panic: runtime error: invalid memory address or nil pointer dereference
	2020-11-05T10:11:04.885+02:00	[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x8c783c]
	2020-11-05T10:11:04.885+02:00	goroutine 30185328 [running]:
	2020-11-05T10:11:04.885+02:00	github.com/pion/webrtc/v3.(*RTPSender).Track(0x0, 0x0)
	2020-11-05T10:11:04.885+02:00	/go/pkg/mod/github.com/pion/webrtc/v3@v3.0.0-beta.10.0.20201025013753-76bc99210140/rtpsender.go:80 +0x3c
	2020-11-05T10:11:04.885+02:00	github.com/pion/webrtc/v3.(*PeerConnection).checkNegotiationNeeded(0xc0390151e0, 0x10000c0a82e8200)
	2020-11-05T10:11:04.885+02:00	/go/pkg/mod/github.com/pion/webrtc/v3@v3.0.0-beta.10.0.20201025013753-76bc99210140/peerconnection.go:359 +0x91a
	2020-11-05T10:11:04.885+02:00	github.com/pion/webrtc/v3.(*PeerConnection).negotiationNeededOp(0xc0390151e0)
	2020-11-05T10:11:04.885+02:00	/go/pkg/mod/github.com/pion/webrtc/v3@v3.0.0-beta.10.0.20201025013753-76bc99210140/peerconnection.go:301 +0x90
	2020-11-05T10:11:04.885+02:00	github.com/pion/webrtc/v3.(*operations).start(0xc10de379b0)
	2020-11-05T10:11:04.885+02:00	/go/pkg/mod/github.com/pion/webrtc/v3@v3.0.0-beta.10.0.20201025013753-76bc99210140/operations.go:84 +0x61
	2020-11-05T10:11:04.885+02:00	created by github.com/pion/webrtc/v3.(*operations).Enqueue
	2020-11-05T10:11:04.885+02:00
	/go/pkg/mod/github.com/pion/webrtc/v3@v3.0.0-beta.10.0.20201025013753-76bc99210140/operations.go:35 +0xcb
	/go/pkg/mod/github.com/pion/webrtc/v3@v3.0.0-beta.10.0.20201025013753-76bc99210140/operations.go:35 +0xcb

[Sean-Der]

Really sorry about that @tomsun!

We are planning on making this feature only fire if you set OnNegotiationNeeded and that should help. There is a lot of complexity here we are still working on nailing down :)

@OrlandoCo Did you get a chance to investigate this at all? Any idea how hard/easy it is? I can probably grab it tomorrow if you haven't start yet!

[OrlandoCo]

@Sean-Der Sorry I have been fighting with a nasty chrome bug for a week now, andI have not had time, hope we can solve it soon!

[tomsun]

Same in beta 12:

2020-11-23T00:42:37.752+02:00	[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x93aaec]
2020-11-23T00:42:37.752+02:00	goroutine 23477494 [running]:
2020-11-23T00:42:37.752+02:00	github.com/pion/webrtc/v3.(*RTPSender).Track(0x0, 0x0)
2020-11-23T00:42:37.752+02:00	/go/pkg/mod/github.com/pion/webrtc/v3@v3.0.0-beta.12/rtpsender.go:80 +0x3c
2020-11-23T00:42:37.752+02:00	github.com/pion/webrtc/v3.(*PeerConnection).checkNegotiationNeeded(0xc02200ed00, 0x0)
2020-11-23T00:42:37.752+02:00	/go/pkg/mod/github.com/pion/webrtc/v3@v3.0.0-beta.12/peerconnection.go:356 +0x8e5
2020-11-23T00:42:37.752+02:00	github.com/pion/webrtc/v3.(*PeerConnection).negotiationNeededOp(0xc02200ed00)
2020-11-23T00:42:37.752+02:00	/go/pkg/mod/github.com/pion/webrtc/v3@v3.0.0-beta.12/peerconnection.go:300 +0x90
2020-11-23T00:42:37.752+02:00	github.com/pion/webrtc/v3.(*operations).start(0xc049bcd860)
2020-11-23T00:42:37.753+02:00	/go/pkg/mod/github.com/pion/webrtc/v3@v3.0.0-beta.12/operations.go:84 +0x61
2020-11-23T00:42:37.753+02:00	created by github.com/pion/webrtc/v3.(*operations).Enqueue
2020-11-23T00:42:37.753+02:00	/go/pkg/mod/github.com/pion/webrtc/v3@v3.0.0-beta.12/operations.go:35 +0xcb

Also a similar but different one:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x154d06b]

goroutine 1190 [running]:
github.com/pion/webrtc/v3.(*Track).Label(0x0, 0x0, 0x0)
	/Users/a16/go/pkg/mod/github.com/pion/webrtc/v3@v3.0.0-beta.12/track.go:73 +0x3b
github.com/pion/webrtc/v3.(*Track).Msid(0x0, 0x0, 0x4)
	/Users/a16/go/pkg/mod/github.com/pion/webrtc/v3@v3.0.0-beta.12/track.go:87 +0x2f
github.com/pion/webrtc/v3.(*PeerConnection).checkNegotiationNeeded(0xc000239ba0, 0x0)
	/Users/a16/go/pkg/mod/github.com/pion/webrtc/v3@v3.0.0-beta.12/peerconnection.go:356 +0x8f3
github.com/pion/webrtc/v3.(*PeerConnection).negotiationNeededOp(0xc000239ba0)
	/Users/a16/go/pkg/mod/github.com/pion/webrtc/v3@v3.0.0-beta.12/peerconnection.go:300 +0x90
github.com/pion/webrtc/v3.(*operations).start(0xc000afc960)
	/Users/a16/go/pkg/mod/github.com/pion/webrtc/v3@v3.0.0-beta.12/operations.go:84 +0x61
created by github.com/pion/webrtc/v3.(*operations).Enqueue
	/Users/a16/go/pkg/mod/github.com/pion/webrtc/v3@v3.0.0-beta.12/operations.go:35 +0xcb
exit status 2
[Sentry] 2020/11/23 02:28:37 Buffer flushed successfully.

[Sean-Der]

@tomsun sorry about that! I opened #1548

We will get that merged today and tagged so you can stop seeing that :(

[Sean-Der]

@tomsun I just tagged v3.0.0-beta.14, so you shouldn't hit this anymore!

We are two PRs away from v3.0.0 as well. Will be working on lots of stabilization tasks as we get closer :)

thanks for using Pion, really appreciate the patience with the bugs.

[tomsun]

🙌 thanks @Sean-Der - awesome!

(Mainly trying out https://github.com/pion/webrtc through https://github.com/pion/ion-sfu - maybe you're right that the sfu doesn't implement renegotiate itself..? Trying out your patch 00ec48a now 🙏 , on top of where we otherwise were, using some replace statements in relevant go.mod:s )

(Also using pion/webrtc client-side, with peerConnection.OnNegotiationNeeded implemented. A crash in that client has a slightly better recovery path though, as it only has to recover itself. Haven't actually seen the above crashes in the client though (although the logs in that app are quite noisy))