panic: runtime error: invalid memory address or nil pointer dereference

[ymettier]

Hello,

I'm running a nightly built version(0.9.4-nightly-cf58c38, branch master, commit cf58c38) and InfluxDB crashs.
I downloaded the rpm and installed with "yum localinstall..."

And InfluxDB does not work on 2nd node.
panic: runtime error: invalid memory address or nil pointer dereference

Here is how I reproduced it :

1/ remove everything in /var/opt/influxdb/ and remove influxdb.log on all nodes (so I start with a clean environment - did I miss anything ?)

2/ start influxdb on HOST0.DOMAIN (/etc/init.d/influxdb start)

Note : I have some collectors already sending metrics to the OpenTSDB connector of InfluxDB, that is enabled in my InfluxDB config.

3/ Edit /etc/default/influxdb and add the "INFLUXD_OPTS="-join HOST0.DOMAIN:8088" on HOST1.DOMAIN server

4/ start influxdb on HOST1.DOMAIN (/etc/init.d/influxdb start)

5/ see it connect to the other node and see it crash just after. Less than 30 seconds after.

Note : I created no database (but I think that the opentsdb connector created one for me)
Note : I created no user, no grant, nothing...
Note : I had not even time to connect my 3rd node. 2nd node had already crashed.

Here are the logs

HOST0.DOMAIN (still running)

[httpd] 2015/08/28 11:21:03 Starting HTTP service
[httpd] 2015/08/28 11:21:03 Authentication enabled: false
[httpd] 2015/08/28 11:21:03 Listening on HTTP: 0.0.0.0:8086
[opentsdb] 2015/08/28 11:21:03 Starting OpenTSDB service
[opentsdb] 2015/08/28 11:21:04 Listening on: 0.0.0.0:4242
[retention] 2015/08/28 11:21:04 Starting retention policy enforcement service with check interval of 10m0s
2015/08/28 11:21:04 InfluxDB starting, version 0.9.4-nightly-cf58c38, branch master, commit cf58c38995f64a45bd8f8b985654facee1892e33
2015/08/28 11:21:04 Go version go1.5, GOMAXPROCS set to 4
[run] 2015/08/28 11:21:04 Listening for signals
[wal] 2015/08/28 11:21:05 WAL starting with 30720 ready series size, 0.50 compaction threshold, and 20971520 partition size threshold
[wal] 2015/08/28 11:21:05 WAL writing to /var/opt/influxdb/wal/poc1/default/1



[metastore] 2015/08/28 11:21:29 adding new raft peer: nodeId=2 addr=HOST1.DOMAIN:8088
[tcp] 2015/08/28 11:21:34 accept remote write connection from X.X.X.X:43496
[tcp] 2015/08/28 11:21:34 close remote write connection from X.X.X.X:43496

Note : X.X.X.X is the IP of HOST1.DOMAIN.

[metastore] 2015/08/28 11:21:35 Node at HOST0.DOMAIN:8088 [Follower]. peers=[HOST1.DOMAIN:8088 HOST0.DOMAIN:8088]
[retention] 2015/08/28 11:31:04 retention policy shard deletion check commencing
[wal] 2015/08/28 11:31:06 Flushing 79 measurements and 5645 series to index
[wal] 2015/08/28 11:31:06 Metadata flush took 40.424703ms

OK, not too bad on HOST0.

HOST1.DOMAIN (crash less than 30 seconds after it started)

[metastore] 2015/08/28 11:21:29 Using data dir: /var/opt/influxdb/meta
[metastore] 2015/08/28 11:21:29 Node at HOST1.DOMAIN:8088 [Follower]
[metastore] 2015/08/28 11:21:29 Joining cluster at: [HOST0.DOMAIN:8088]
[metastore] 2015/08/28 11:21:29 Joined remote node HOST0.DOMAIN:8088
[metastore] 2015/08/28 11:21:29 nodeId=2 raftEnabled=true peers=[HOST1.DOMAIN:8088 HOST0.DOMAIN:8088]
[store] 2015/08/28 11:21:29 Using data dir: /var/opt/influxdb/data
[handoff] 2015/08/28 11:21:29 Starting hinted handoff service
[handoff] 2015/08/28 11:21:29 Using data dir: /var/opt/influxdb/hh
[tcp] 2015/08/28 11:21:29 Starting cluster service
[shard-precreation] 2015/08/28 11:21:29 Starting precreation service with check interval of 10m0s, advance period of 30m0s
[snapshot] 2015/08/28 11:21:29 Starting snapshot service
[admin] 2015/08/28 11:21:29 Starting admin service
[admin] 2015/08/28 11:21:29 Listening on HTTP: 0.0.0.0:8083
[continuous_querier] 2015/08/28 11:21:29 Starting continuous query service
[httpd] 2015/08/28 11:21:29 Starting HTTP service
[httpd] 2015/08/28 11:21:29 Authentication enabled: false
[httpd] 2015/08/28 11:21:29 Listening on HTTP: 0.0.0.0:8086
[opentsdb] 2015/08/28 11:21:29 Starting OpenTSDB service
[metastore] 2015/08/28 11:21:29 Updated node id=2 hostname=HOST1.DOMAIN:8088
[opentsdb] 2015/08/28 11:21:29 Listening on: 0.0.0.0:4242
[retention] 2015/08/28 11:21:29 Starting retention policy enforcement service with check interval of 10m0s
2015/08/28 11:21:29 InfluxDB starting, version 0.9.4-nightly-cf58c38, branch master, commit cf58c38995f64a45bd8f8b985654facee1892e33
2015/08/28 11:21:29 Go version go1.5, GOMAXPROCS set to 4
[run] 2015/08/28 11:21:29 Listening for signals
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x40 pc=0x62805d]

goroutine 16 [running]:
github.com/influxdb/influxdb/cluster.(*ShardMapper).CreateMapper(0xc820150240, 0x1, 0xc8201077f0, 0x1, 0x1, 0x7fe19f882428, 0xc8201380c0, 0x2710, 0x0, 0x0, ...)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cluster/shard_mapper.go:58 +0x4fd
github.com/influxdb/influxdb/tsdb.(*QueryExecutor).PlanSelect(0xc820150280, 0xc8201380c0, 0x2710, 0x0, 0x0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/query_executor.go:240 +0xa34
github.com/influxdb/influxdb/tsdb.(*QueryExecutor).executeSelectStatement(0xc820150280, 0x0, 0xc8201380c0, 0xc820234ae0, 0x2710, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/query_executor.go:258 +0x5c
github.com/influxdb/influxdb/tsdb.(*QueryExecutor).ExecuteQuery.func1(0xc820239340, 0xc8200e050e, 0x4, 0xc820150280, 0xc820234ae0, 0x2710)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/query_executor.go:146 +0x632
created by github.com/influxdb/influxdb/tsdb.(*QueryExecutor).ExecuteQuery
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/query_executor.go:199 +0x9b

goroutine 1 [chan receive]:
main.(*Main).Run(0xc820111f00, 0xc82000a0f0, 0x6, 0x6, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cmd/influxd/main.go:91 +0xa16
main.main()
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cmd/influxd/main.go:42 +0x29b

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1

goroutine 5 [syscall]:
os/signal.loop()
        /root/.gvm/gos/go1.5/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
        /root/.gvm/gos/go1.5/src/os/signal/signal_unix.go:28 +0x37

goroutine 8 [IO wait]:
net.runtime_pollWait(0x7fe19f8818f8, 0x72, 0xc8200101c0)
        /root/.gvm/gos/go1.5/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc82000b5d0, 0x72, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82000b5d0, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc82000b570, 0x0, 0x7fe19c0610d8, 0xc82016c5a0)
        /root/.gvm/gos/go1.5/src/net/fd_unix.go:408 +0x27c
net.(*TCPListener).AcceptTCP(0xc82002c120, 0x9888e0, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/tcpsock_posix.go:254 +0x4d
net.(*TCPListener).Accept(0xc82002c120, 0x0, 0x0, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/tcpsock_posix.go:264 +0x3d
github.com/influxdb/influxdb/tcp.(*Mux).Serve(0xc820156ea0, 0x7fe19f880950, 0xc82002c120, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tcp/mux.go:45 +0x5d
created by github.com/influxdb/influxdb/cmd/influxd/run.(*Server).Open.func1
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:313 +0xa81

goroutine 9 [chan receive]:
github.com/influxdb/influxdb/tcp.(*listener).Accept(0xc82002c128, 0x0, 0x0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tcp/mux.go:117 +0x6f
github.com/influxdb/influxdb/meta.(*raftLayer).Accept(0xc820156ff0, 0x0, 0x0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/meta/store.go:2058 +0x51
github.com/hashicorp/raft.(*NetworkTransport).listen(0xc82000b650)
        /tmp/tmp.SXplMOFgOF/src/github.com/hashicorp/raft/net_transport.go:346 +0x79
created by github.com/hashicorp/raft.NewNetworkTransport
        /tmp/tmp.SXplMOFgOF/src/github.com/hashicorp/raft/net_transport.go:138 +0x284

goroutine 10 [select]:
github.com/hashicorp/raft.(*Raft).runFollower(0xc8200941c0)
        /tmp/tmp.SXplMOFgOF/src/github.com/hashicorp/raft/raft.go:606 +0x91d
github.com/hashicorp/raft.(*Raft).run(0xc8200941c0)
        /tmp/tmp.SXplMOFgOF/src/github.com/hashicorp/raft/raft.go:591 +0x91
github.com/hashicorp/raft.(*Raft).(github.com/hashicorp/raft.run)-fm()
        /tmp/tmp.SXplMOFgOF/src/github.com/hashicorp/raft/raft.go:245 +0x20
github.com/hashicorp/raft.(*raftState).goFunc.func1(0xc8200941c0, 0xc820155f50)
        /tmp/tmp.SXplMOFgOF/src/github.com/hashicorp/raft/state.go:152 +0x44
created by github.com/hashicorp/raft.(*raftState).goFunc
        /tmp/tmp.SXplMOFgOF/src/github.com/hashicorp/raft/state.go:153 +0x4d

goroutine 11 [select]:
github.com/hashicorp/raft.(*Raft).runFSM(0xc8200941c0)
        /tmp/tmp.SXplMOFgOF/src/github.com/hashicorp/raft/raft.go:502 +0x10eb
github.com/hashicorp/raft.(*Raft).(github.com/hashicorp/raft.runFSM)-fm()
        /tmp/tmp.SXplMOFgOF/src/github.com/hashicorp/raft/raft.go:246 +0x20
github.com/hashicorp/raft.(*raftState).goFunc.func1(0xc8200941c0, 0xc820155f60)
        /tmp/tmp.SXplMOFgOF/src/github.com/hashicorp/raft/state.go:152 +0x44
created by github.com/hashicorp/raft.(*raftState).goFunc
        /tmp/tmp.SXplMOFgOF/src/github.com/hashicorp/raft/state.go:153 +0x4d

goroutine 12 [select]:
github.com/hashicorp/raft.(*Raft).runSnapshots(0xc8200941c0)
        /tmp/tmp.SXplMOFgOF/src/github.com/hashicorp/raft/raft.go:1613 +0x493
github.com/hashicorp/raft.(*Raft).(github.com/hashicorp/raft.runSnapshots)-fm()
        /tmp/tmp.SXplMOFgOF/src/github.com/hashicorp/raft/raft.go:247 +0x20
github.com/hashicorp/raft.(*raftState).goFunc.func1(0xc8200941c0, 0xc820155f70)
        /tmp/tmp.SXplMOFgOF/src/github.com/hashicorp/raft/state.go:152 +0x44
created by github.com/hashicorp/raft.(*raftState).goFunc
        /tmp/tmp.SXplMOFgOF/src/github.com/hashicorp/raft/state.go:153 +0x4d

goroutine 13 [select]:
github.com/influxdb/influxdb/meta.(*localRaft).logLeaderChanges(0xc820069590)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/meta/state.go:187 +0x4f5
created by github.com/influxdb/influxdb/meta.(*localRaft).open
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/meta/state.go:177 +0x1344

goroutine 34 [chan receive]:
github.com/influxdb/influxdb/tcp.(*listener).Accept(0xc82002c130, 0x0, 0x0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tcp/mux.go:117 +0x6f
github.com/influxdb/influxdb/meta.(*Store).serveExecListener(0xc82000cb40)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/meta/store.go:619 +0xa2
created by github.com/influxdb/influxdb/meta.(*Store).Open
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/meta/store.go:239 +0x364

goroutine 35 [chan receive]:
github.com/influxdb/influxdb/tcp.(*listener).Accept(0xc82002c138, 0x0, 0x0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tcp/mux.go:117 +0x6f
github.com/influxdb/influxdb/meta.(*Store).serveRPCListener(0xc82000cb40)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/meta/store.go:723 +0xa2
created by github.com/influxdb/influxdb/meta.(*Store).Open
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/meta/store.go:242 +0x3b6

goroutine 51 [IO wait]:
net.runtime_pollWait(0x7fe19f8815f8, 0x72, 0xc8200101c0)
        /root/.gvm/gos/go1.5/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc8201ac290, 0x72, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8201ac290, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc8201ac230, 0x0, 0x7fe19c0610d8, 0xc820116600)
        /root/.gvm/gos/go1.5/src/net/fd_unix.go:408 +0x27c
net.(*TCPListener).AcceptTCP(0xc8201ae018, 0x455470, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/tcpsock_posix.go:254 +0x4d
net.(*TCPListener).Accept(0xc8201ae018, 0x0, 0x0, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/tcpsock_posix.go:264 +0x3d
github.com/influxdb/influxdb/services/opentsdb.(*Service).serve(0xc8200c91e0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/opentsdb/service.go:150 +0xae
created by github.com/influxdb/influxdb/services/opentsdb.(*Service).Open
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/opentsdb/service.go:115 +0xadf

goroutine 52 [select]:
github.com/influxdb/influxdb/services/retention.(*Service).deleteShardGroups(0xc820069220)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/retention/service.go:69 +0x351
created by github.com/influxdb/influxdb/services/retention.(*Service).Open
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/retention/service.go:45 +0x1b4

goroutine 19 [IO wait]:
net.runtime_pollWait(0x7fe19f881838, 0x72, 0xc8200101c0)
        /root/.gvm/gos/go1.5/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc8201be1b0, 0x72, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8201be1b0, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8201be150, 0xc8201c4000, 0x1000, 0x1000, 0x0, 0x7fe19f87b050, 0xc8200101c0)
        /root/.gvm/gos/go1.5/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc820108030, 0xc8201c4000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/net.go:172 +0xe4
bufio.(*Reader).fill(0xc820180120)
        /root/.gvm/gos/go1.5/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).ReadByte(0xc820180120, 0x1, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/bufio/bufio.go:229 +0x7a
github.com/hashicorp/raft.(*NetworkTransport).handleCommand(0xc82000b650, 0xc820180120, 0xc820180180, 0xc8201be1c0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/hashicorp/raft/net_transport.go:386 +0x45
github.com/hashicorp/raft.(*NetworkTransport).handleConn(0xc82000b650, 0x7fe1975bf058, 0xc820108030)
        /tmp/tmp.SXplMOFgOF/src/github.com/hashicorp/raft/net_transport.go:370 +0x361
created by github.com/hashicorp/raft.(*NetworkTransport).listen
        /tmp/tmp.SXplMOFgOF/src/github.com/hashicorp/raft/net_transport.go:357 +0x3a9

goroutine 50 [chan receive]:
github.com/influxdb/influxdb/services/opentsdb.(*chanListener).Accept(0xc8201a06a0, 0x0, 0x0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/opentsdb/handler.go:149 +0x83
net/http.(*Server).Serve(0xc820184480, 0x7fe19c021418, 0xc8201a06a0, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1887 +0xb3
github.com/influxdb/influxdb/services/opentsdb.(*Service).serveHTTP(0xc8200c91e0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/opentsdb/service.go:280 +0x1c7
created by github.com/influxdb/influxdb/services/opentsdb.(*Service).Open
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/opentsdb/service.go:114 +0xaba

goroutine 21 [select]:
github.com/influxdb/influxdb/cmd/influxd/run.(*Server).monitorErrorChan(0xc82008e620, 0xc82001a900)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:442 +0x145
created by github.com/influxdb/influxdb/cmd/influxd/run.(*Server).Open.func1
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:319 +0xbff

goroutine 22 [select]:
github.com/influxdb/influxdb/services/hh.(*Service).retryWrites(0xc82007c5a0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/hh/service.go:103 +0x314
created by github.com/influxdb/influxdb/services/hh.(*Service).Open
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/hh/service.go:67 +0x210

goroutine 23 [select]:
github.com/influxdb/influxdb/services/hh.(*Service).expireWrites(0xc82007c5a0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/hh/service.go:121 +0x2d1
created by github.com/influxdb/influxdb/services/hh.(*Service).Open
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/hh/service.go:68 +0x232

goroutine 24 [chan receive]:
github.com/influxdb/influxdb/tcp.(*listener).Accept(0xc82002c140, 0x0, 0x0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tcp/mux.go:117 +0x6f
github.com/influxdb/influxdb/cluster.(*Service).serve(0xc82000b3b0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cluster/service.go:84 +0xdb
created by github.com/influxdb/influxdb/cluster.(*Service).Open
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cluster/service.go:61 +0x135

goroutine 25 [select]:
github.com/influxdb/influxdb/services/precreator.(*Service).runPrecreation(0xc8201502c0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/precreator/service.go:75 +0x44f
created by github.com/influxdb/influxdb/services/precreator.(*Service).Open
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/precreator/service.go:53 +0x21f

goroutine 26 [chan receive]:
github.com/influxdb/influxdb/tcp.(*listener).Accept(0xc82002c148, 0x0, 0x0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tcp/mux.go:117 +0x6f
github.com/influxdb/influxdb/services/snapshotter.(*Service).serve(0xc820068f50)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/snapshotter/service.go:75 +0xa9
created by github.com/influxdb/influxdb/services/snapshotter.(*Service).Open
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/snapshotter/service.go:48 +0x130

goroutine 27 [IO wait]:
net.runtime_pollWait(0x7fe19f881778, 0x72, 0xc8200101c0)
        /root/.gvm/gos/go1.5/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc8201f0ca0, 0x72, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8201f0ca0, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc8201f0c40, 0x0, 0x7fe19c0610d8, 0xc8201165c0)
        /root/.gvm/gos/go1.5/src/net/fd_unix.go:408 +0x27c
net.(*TCPListener).AcceptTCP(0xc8201080a8, 0x455470, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/tcpsock_posix.go:254 +0x4d
net.(*TCPListener).Accept(0xc8201080a8, 0x0, 0x0, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/tcpsock_posix.go:264 +0x3d
net/http.(*Server).Serve(0xc8201843c0, 0x7fe19f880950, 0xc8201080a8, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1887 +0xb3
net/http.Serve(0x7fe19f880950, 0xc8201080a8, 0x7fe19c0213c8, 0xc820114330, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1765 +0x85
github.com/influxdb/influxdb/services/admin.(*Service).serve(0xc820068ff0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/admin/service.go:107 +0x12f
created by github.com/influxdb/influxdb/services/admin.(*Service).Open
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/admin/service.go:70 +0x491

goroutine 28 [select]:
github.com/influxdb/influxdb/services/continuous_querier.(*Service).backgroundLoop(0xc82000b420)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/continuous_querier/service.go:157 +0x434
created by github.com/influxdb/influxdb/services/continuous_querier.(*Service).Open
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/continuous_querier/service.go:93 +0x294

goroutine 29 [IO wait]:
net.runtime_pollWait(0x7fe19f8816b8, 0x72, 0xc8200101c0)
        /root/.gvm/gos/go1.5/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc8201f0d10, 0x72, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8201f0d10, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc8201f0cb0, 0x0, 0x7fe19c0610d8, 0xc820238d40)
        /root/.gvm/gos/go1.5/src/net/fd_unix.go:408 +0x27c
net.(*TCPListener).AcceptTCP(0xc8201080b0, 0xc820039dc0, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/tcpsock_posix.go:254 +0x4d
net.(*TCPListener).Accept(0xc8201080b0, 0x0, 0x0, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/tcpsock_posix.go:264 +0x3d
net/http.(*Server).Serve(0xc820180d80, 0x7fe19f880950, 0xc8201080b0, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1887 +0xb3
net/http.Serve(0x7fe19f880950, 0xc8201080b0, 0x7fe19c065808, 0xc82000b490, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1765 +0x85
github.com/influxdb/influxdb/services/httpd.(*Service).serve(0xc820069130)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/service.go:108 +0x6e
created by github.com/influxdb/influxdb/services/httpd.(*Service).Open
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/service.go:76 +0x6ca

goroutine 53 [select]:
github.com/influxdb/influxdb/services/retention.(*Service).deleteShards(0xc820069220)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/retention/service.go:102 +0x7a0
created by github.com/influxdb/influxdb/services/retention.(*Service).Open
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/retention/service.go:46 +0x1d9

goroutine 54 [select]:
github.com/influxdb/influxdb/cmd/influxd/run.(*Command).monitorServerErrors(0xc820088280)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cmd/influxd/run/command.go:137 +0x2c5
created by github.com/influxdb/influxdb/cmd/influxd/run.(*Command).Run
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cmd/influxd/run/command.go:119 +0xe81

goroutine 55 [select, locked to thread]:
runtime.gopark(0xcea110, 0xc82021e728, 0xbaba58, 0x6, 0x18, 0x2)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163
runtime.selectgoImpl(0xc82021e728, 0x0, 0x18)
        /root/.gvm/gos/go1.5/src/runtime/select.go:392 +0xa64
runtime.selectgo(0xc82021e728)
        /root/.gvm/gos/go1.5/src/runtime/select.go:212 +0x12
runtime.ensureSigM.func1()
        /root/.gvm/gos/go1.5/src/runtime/signal1_unix.go:227 +0x353
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1

goroutine 14 [runnable]:
syscall.Syscall(0x1, 0xb, 0xc82022e000, 0x1a, 0x1a, 0x1a, 0x0)
        /root/.gvm/gos/go1.5/src/syscall/asm_linux_amd64.s:18 +0x5
syscall.write(0xb, 0xc82022e000, 0x1a, 0x1000, 0x77, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/syscall/zsyscall_linux_amd64.go:1064 +0x5f
syscall.Write(0xb, 0xc82022e000, 0x1a, 0x1000, 0x0, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/syscall/syscall_unix.go:176 +0x4d
net.(*netFD).Write(0xc82000b960, 0xc82022e000, 0x1a, 0x1000, 0x0, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/fd_unix.go:310 +0x1c7
net.(*conn).Write(0xc82002c1b0, 0xc82022e000, 0x1a, 0x1000, 0x0, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/net.go:184 +0xe4
bufio.(*Writer).flush(0xc820150b80, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/bufio/bufio.go:562 +0xe0
bufio.(*Writer).Flush(0xc820150b80, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/bufio/bufio.go:551 +0x2d
github.com/hashicorp/raft.(*NetworkTransport).handleConn(0xc82000b650, 0x7fe1975bf058, 0xc82002c1b0)
        /tmp/tmp.SXplMOFgOF/src/github.com/hashicorp/raft/net_transport.go:376 +0x4e2
created by github.com/hashicorp/raft.(*NetworkTransport).listen
        /tmp/tmp.SXplMOFgOF/src/github.com/hashicorp/raft/net_transport.go:357 +0x3a9

goroutine 15 [chan receive]:
github.com/influxdb/influxdb/services/httpd.(*Handler).serveQuery(0xc82000b490, 0x7fe19f882068, 0xc820238e40, 0xc82008e700, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:260 +0xbf9
github.com/influxdb/influxdb/services/httpd.(*Handler).(github.com/influxdb/influxdb/services/httpd.serveQuery)-fm(0x7fe19f882068, 0xc820238e40, 0xc82008e700, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:95 +0x48
github.com/influxdb/influxdb/services/httpd.authenticate.func1(0x7fe19f882068, 0xc820238e40, 0xc82008e700)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:621 +0x70
net/http.HandlerFunc.ServeHTTP(0xc820153680, 0x7fe19f882068, 0xc820238e40, 0xc82008e700)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1422 +0x3a
github.com/influxdb/influxdb/services/httpd.gzipFilter.func1(0x7fe19f881f90, 0xc820238e00, 0xc82008e700)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:679 +0x2af
net/http.HandlerFunc.ServeHTTP(0xc8201536a0, 0x7fe19f881f90, 0xc820238e00, 0xc82008e700)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1422 +0x3a
github.com/influxdb/influxdb/services/httpd.versionHeader.func1(0x7fe19f881f90, 0xc820238e00, 0xc82008e700)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:688 +0xc0
net/http.HandlerFunc.ServeHTTP(0xc8201536c0, 0x7fe19f881f90, 0xc820238e00, 0xc82008e700)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1422 +0x3a
github.com/influxdb/influxdb/services/httpd.cors.func1(0x7fe19f881f90, 0xc820238e00, 0xc82008e700)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:721 +0x328
net/http.HandlerFunc.ServeHTTP(0xc8201536e0, 0x7fe19f881f90, 0xc820238e00, 0xc82008e700)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1422 +0x3a
github.com/influxdb/influxdb/services/httpd.requestID.func1(0x7fe19f881f90, 0xc820238e00, 0xc82008e700)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:731 +0x15d
net/http.HandlerFunc.ServeHTTP(0xc820153700, 0x7fe19f881f90, 0xc820238e00, 0xc82008e700)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1422 +0x3a
github.com/influxdb/influxdb/services/httpd.logging.func1(0x7fe19f881f90, 0xc820238de0, 0xc82008e700)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:739 +0x10b
net/http.HandlerFunc.ServeHTTP(0xc820153720, 0x7fe19f881f90, 0xc820238de0, 0xc82008e700)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1422 +0x3a
github.com/influxdb/influxdb/services/httpd.recovery.func1(0x7fe19f881f58, 0xc8200c93f0, 0xc82008e700)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:749 +0x114
net/http.HandlerFunc.ServeHTTP(0xc820153740, 0x7fe19f881f58, 0xc8200c93f0, 0xc82008e700)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1422 +0x3a
github.com/bmizerany/pat.(*PatternServeMux).ServeHTTP(0xc82002c0e0, 0x7fe19f881f58, 0xc8200c93f0, 0xc82008e700)
        /tmp/tmp.SXplMOFgOF/src/github.com/bmizerany/pat/mux.go:109 +0x244
github.com/influxdb/influxdb/services/httpd.(*Handler).ServeHTTP(0xc82000b490, 0x7fe19f881f58, 0xc8200c93f0, 0xc82008e700)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:167 +0x243
net/http.serverHandler.ServeHTTP(0xc820180d80, 0x7fe19f881f58, 0xc8200c93f0, 0xc82008e700)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1862 +0x19e
net/http.(*conn).serve(0xc8200c9340)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1361 +0xbee
created by net/http.(*Server).Serve
        /root/.gvm/gos/go1.5/src/net/http/server.go:1910 +0x3f6

Sorry for the long logs.

More info...

$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.6 (Santiago)

$ rpm -qi influxdb
Name        : influxdb                     Relocations: /
Version     : 0.9.4_nightly_cf58c38             Vendor: Influxdb
Release     : 1                             Build Date: Fri 28 Aug 2015 06:01:42 AM CEST
Install Date: Fri 28 Aug 2015 10:02:35 AM CEST      Build Host: build.influxdb.com
Group       : default                       Source RPM: influxdb-0.9.4_nightly_cf58c38-1.src.rpm
Size        : 29237764                         License: MIT
Signature   : (none)
Packager    : support@influxdb.com
URL         : influxdb.com
Summary     : Distributed time-series database
Description :
Distributed time-series database

$ df -h /var/opt/influxdb
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/rootvg-influxdbdata
                       99G   38M   94G   1% /var/opt/influxdb

[jipperinbham]

I have the same issue but with a single node cluster running 0.9.3 (released version), see https://gist.github.com/jipperinbham/3f88ffee95508c82aa18

[beckettsean]

@jipperinbham that is a different issue than the initial one reported here. Can you open a new issue and include steps to repro, possibly including a copy of the /data directory?

[jipperinbham]

@beckettsean done, #3941

[jtblin]

I can reproduce the same error in 0.9.4.1 and 0.9.4.2 with collectd enabled. Pointing out as CHANGELOG says it's supposed to be fixed in 0.9.4.

[beckettsean]

@jtblin can you give the repro steps for 0.9.4.2? The panic: runtime error: invalid memory address or nil pointer dereference panic is something of a general error. It's possible you are getting the same panic for a completely different issue.