stability: panic on blue after upgrade

[mberhault]

blue was just upgraded from 1eaa457 to b820060

Shortly after startup, multiple nodes (40.79.33.40 and many others) crashed with:

F161130 15:18:19.117192 23528 storage/replica_raftstorage.go:701  [n1,s1,r1433/?:{-},@c42eca4000] unexpected range ID 0
goroutine 23528 [running]:
github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0x5b01, 0x6fc37d3, 0x29ed240, 0xc42920aac8)
        /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:849 +0xa5
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).outputLogEntry(0x29ee840, 0xc400000004, 0x21f181f, 0x1e, 0x2bd, 0xc427cc0180, 0x35)
        /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:714 +0x7b1
github.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x25322a0, 0xc424b52f30, 0x4, 0x2, 0x1992d55, 0x16, 0xc42920aee8, 0x1, 0x1)
        /go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:140 +0x296
github.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x25322a0, 0xc424b52f30, 0x1, 0xc400000004, 0x1992d55, 0x16, 0xc42920aee8, 0x1, 0x1)
        /go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:88 +0x8c
github.com/cockroachdb/cockroach/pkg/util/log.Fatalf(0x25322a0, 0xc424b52f30, 0x1992d55, 0x16, 0xc42920aee8, 0x1, 0x1)
        /go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:172 +0x7e
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).applySnapshot(0xc42eca4000, 0x25322a0, 0xc424b52f30, 0x844aeb441995c4ae, 0x1dfde5bec556fba4, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_raftstorage.go:701 +0xd3d
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftRequest(0xc420a9ee00, 0x25322a0, 0xc424b52f30, 0xc425bb18b8, 0x844aeb441995c4ae, 0x1dfde5bec556fba4, 0x0, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:2942 +0x8a3
github.com/cockroachdb/cockroach/pkg/storage.(*Store).HandleSnapshot(0xc420a9ee00, 0xc425bb1860, 0x7f7cc8ab71f8, 0xc42c602170, 0x0, 0x0)
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:2628 +0x746
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot.func1.1(0x2537d20, 0xc42c602170, 0xc420168540, 0x25322a0, 0xc4276590e0, 0xc420db4738, 0xc420db4740)
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:362 +0x17f
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot.func1(0x25322a0, 0xc4276590e0)
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:363 +0x5d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc420416000, 0x21f1b09, 0x19, 0x16c, 0x0, 0x0, 0xc427659110, 0x25322a0, 0xc4276590e0)
        /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:264 +0xdf
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:265 +0x234

Another two (40.123.43.133) crashed with:

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xb48d4e]

goroutine 1089 [running]:
panic(0x1801040, 0xc420018040)
        /usr/local/go/src/runtime/panic.go:500 +0x1a1
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Recover(0xc4203d4000)
        /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0x6e
panic(0x1801040, 0xc420018040)
        /usr/local/go/src/runtime/panic.go:458 +0x243
github.com/cockroachdb/cockroach/pkg/storage.(*Store).canApplySnapshotLocked(0xc420986380, 0x2533960, 0xc421672360, 0x0, 0x2533960, 0xc421683dd0, 0x2533960)
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3575 +0x7e
github.com/cockroachdb/cockroach/pkg/storage.(*Store).canApplySnapshot(0xc420986380, 0x2533960, 0xc421672360, 0x0, 0x0, 0x0, 0x0)
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3569 +0xad
github.com/cockroachdb/cockroach/pkg/storage.(*Store).HandleSnapshot(0xc420986380, 0xc4241586c0, 0x7fca0f2c7ce0, 0xc4224537c0, 0x0, 0x0)
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:2587 +0x312
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot.func1.1(0x25393e0, 0xc4224537c0, 0xc42006ed80, 0x2533960, 0xc421683dd0, 0xc420024f38, 0xc420024f40)
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:362 +0x182
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot.func1(0x2533960, 0xc421683dd0)
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:363 +0x5d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc4203d4000, 0x21f2f21, 0x19, 0x16c, 0x0, 0x0, 0xc421672270, 0x2533960, 0xc421683dd0)
        /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:264 +0xdf
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:265 +0x234

[petermattis]

Looks like you did a rolling upgrade. #10931 requires a stop the world upgrade.

[mberhault]

my bad, I ended up with two different versions, one just before that change, the other one at that change.
Restarting now with everything at b820060

[mberhault]

blue looks ok after a "proper" upgrade. Sorry about that.