PANIC: runtime error: invalid memory address or nil pointer dereference

[motyla]

version : 0.7.4_633_gd5ca2bcb-1

Jan  4 11:43:27 metrictank002 metrictank[6702]: [Macaron] PANIC: runtime error: invalid memory address or nil pointer dereference
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/runtime/panic.go:491 (0x42b342)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/runtime/panic.go:63 (0x42a24d)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/runtime/signal_unix.go:367 (0x4426cb)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/idx/memory/memory.go:113 (0xa6739c)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/idx/memory/memory.go:807 (0xa65f5a)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/idx/memory/memory.go:807 (0xa65f5a)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/api/cluster.go:96 (0xa3f04f)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/api/routes.go:38 (0xa5d25a)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/runtime/asm_amd64.s:510 (0x45a35a)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:434 (0x4b8f85)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:302 (0x4b8563)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:177 (0x9ab288)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:137 (0x9aab89)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:113 (0x9c7aad)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:104 (0x9c79e5)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/api/middleware/tracer.go:64 (0x9e658f)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/runtime/asm_amd64.s:509 (0x45a2da)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:434 (0x4b8f85)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:302 (0x4b8563)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:177 (0x9ab288)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:137 (0x9aab89)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:113 (0x9c7aad)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:104 (0x9c79e5)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/api/middleware/stats.go:72 (0x9e5d78)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/runtime/asm_amd64.s:509 (0x45a2da)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:434 (0x4b8f85)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:302 (0x4b8563)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:177 (0x9ab288)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:137 (0x9aab89)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:113 (0x9c7aad)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:104 (0x9c79e5)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/raintank/gziper/gzip.go:73 (0xa3c257)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/runtime/asm_amd64.s:509 (0x45a2da)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:434 (0x4b8f85)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:302 (0x4b8563)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:177 (0x9ab288)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:137 (0x9aab89)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:113 (0x9c7aad)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:104 (0x9c79e5)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/recovery.go:161 (0x9d8dba)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/runtime/asm_amd64.s:509 (0x45a2da)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:434 (0x4b8f85)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:302 (0x4b8563)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:177 (0x9ab288)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:137 (0x9aab89)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:113 (0x9c7aad)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:104 (0x9c79e5)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/logger.go:43 (0x9d808b)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/runtime/asm_amd64.s:509 (0x45a2da)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:434 (0x4b8f85)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:302 (0x4b8563)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:177 (0x9ab288)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:137 (0x9aab89)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:113 (0x9c7aad)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/router.go:184 (0x9da0a2)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/router.go:286 (0x9d36eb)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/macaron.go:177 (0x9cbc27)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/net/http/server.go:2619 (0x6e22b3)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/net/http/server.go:1801 (0x6de5cc)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/runtime/asm_amd64.s:2337 (0x45cad0)
Jan  4 11:43:27 metrictank002 metrictank[6702]: [Macaron] #033[1;32m2018-01-04 11:43:27: Completed /index/find 200 OK in 4.635476ms#033[0m
Jan  4 11:43:27 metrictank002 metrictank[6702]: [Macaron] #033[1;36m2018-01-04 11:43:27: Completed /index/find 500 Internal Server Error in 2.983263ms#033[0m
Jan  4 11:43:27 metrictank002 metrictank[6702]: [Macaron] #033[1;32m2018-01-04 11:43:27: Completed /index/find 200 OK in 5.357149ms#033[0m
Jan  4 11:43:27 metrictank002 metrictank[6702]: [Macaron] #033[1;32m2018-01-04 11:43:27: Completed /index/find 200 OK in 3.039755ms#033[0m
Jan  4 11:43:27 metrictank002 metrictank[6702]: [Macaron] #033[1;32m2018-01-04 11:43:27: Completed /index/find 200 OK in 5.15759ms#033[0m
Jan  4 11:43:27 metrictank002 metrictank[6702]: [Macaron] #033[1;32m2018-01-04 11:43:27: Completed /index/find 200 OK in 383.664763ms#033[0m
Jan  4 11:43:27 metrictank002 metrictank[6702]: [Macaron] 2018-01-04 11:43:27: Started POST /getdata for 10.106.38.183
Jan  4 11:43:27 metrictank002 metrictank[6702]: [Macaron] #033[1;32m2018-01-04 11:43:27: Completed /index/find 200 OK in 3.571271ms#033[0m
Jan  4 11:43:27 metrictank002 metrictank[6702]: [Macaron] #033[1;32m2018-01-04 11:43:27: Completed /index/find 200 OK in 156.45785ms#033[0m
Jan  4 11:43:27 metrictank002 metrictank[6702]: [Macaron] #033[1;32m2018-01-04 11:43:27: Completed /index/find 200 OK in 356.869488ms#033[0m
Jan  4 11:43:27 metrictank002 metrictank[6702]: [Macaron] #033[1;32m2018-01-04 11:43:27: Completed /index/find 200 OK in 5.362755ms#033[0m
Jan  4 11:43:27 metrictank002 metrictank[6702]: [Macaron] #033[1;32m2018-01-04 11:43:27: Completed /index/find 200 OK in 6.321715ms#033[0m
Jan  4 11:43:27 metrictank002 metrictank[6702]: [Macaron] #033[1;32m2018-01-04 11:43:27: Completed /index/find 200 OK in 4.746094ms#033[0m
Jan  4 11:43:27 metrictank002 metrictank[6702]: [Macaron] PANIC: runtime error: invalid memory address or nil pointer dereference
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/runtime/panic.go:491 (0x42b342)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/runtime/panic.go:63 (0x42a24d)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/runtime/signal_unix.go:367 (0x4426cb)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/idx/memory/memory.go:113 (0xa6739c)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/idx/memory/memory.go:807 (0xa65f5a)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/idx/memory/memory.go:807 (0xa65f5a)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/api/cluster.go:96 (0xa3f04f)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/api/routes.go:38 (0xa5d25a)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/runtime/asm_amd64.s:510 (0x45a35a)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:434 (0x4b8f85)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:302 (0x4b8563)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:177 (0x9ab288)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:137 (0x9aab89)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:113 (0x9c7aad)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:104 (0x9c79e5)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/api/middleware/tracer.go:64 (0x9e658f)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/runtime/asm_amd64.s:509 (0x45a2da)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:434 (0x4b8f85)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:302 (0x4b8563)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:177 (0x9ab288)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:137 (0x9aab89)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:113 (0x9c7aad)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:104 (0x9c79e5)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/api/middleware/stats.go:72 (0x9e5d78)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/runtime/asm_amd64.s:509 (0x45a2da)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:434 (0x4b8f85)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:302 (0x4b8563)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:177 (0x9ab288)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:137 (0x9aab89)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:113 (0x9c7aad)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:104 (0x9c79e5)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/raintank/gziper/gzip.go:73 (0xa3c257)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/runtime/asm_amd64.s:509 (0x45a2da)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:434 (0x4b8f85)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:302 (0x4b8563)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:177 (0x9ab288)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:137 (0x9aab89)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:113 (0x9c7aad)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:104 (0x9c79e5)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/recovery.go:161 (0x9d8dba)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/runtime/asm_amd64.s:509 (0x45a2da)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:434 (0x4b8f85)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:302 (0x4b8563)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:177 (0x9ab288)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:137 (0x9aab89)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:113 (0x9c7aad)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:104 (0x9c79e5)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/logger.go:43 (0x9d808b)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/runtime/asm_amd64.s:509 (0x45a2da)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:434 (0x4b8f85)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:302 (0x4b8563)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:177 (0x9ab288)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:137 (0x9aab89)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:113 (0x9c7aad)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/router.go:184 (0x9da0a2)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/router.go:286 (0x9d36eb)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/macaron.go:177 (0x9cbc27)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/net/http/server.go:2619 (0x6e22b3)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/net/http/server.go:1801 (0x6de5cc)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/runtime/asm_amd64.s:2337 (0x45cad0)
Jan  4 11:43:27 metrictank002 metrictank[6702]: [Macaron] #033[1;36m2018-01-04 11:43:27: Completed /index/find 500 Internal Server Error in 4.587958ms#033[0m
Jan  4 11:43:27 metrictank002 metrictank[6702]: [Macaron] #033[1;32m2018-01-04 11:43:27: Completed /index/find 200 OK in 6.628582ms#033[0m
Jan  4 11:43:27 metrictank002 metrictank[6702]: [Macaron] #033[1;32m2018-01-04 11:43:27: Completed /index/find 200 OK in 6.727015ms#033[0m
Jan  4 11:43:27 metrictank002 metrictank[6702]: panic: runtime error: invalid memory address or nil pointer dereference
Jan  4 11:43:27 metrictank002 metrictank[6702]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xa6739c]
Jan  4 11:43:27 metrictank002 metrictank[6702]: goroutine 3245986 [running]:
Jan  4 11:43:27 metrictank002 metrictank[6702]: github.com/grafana/metrictank/idx/memory.(*Node).HasChildren(...)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/idx/memory/memory.go:113
Jan  4 11:43:27 metrictank002 metrictank[6702]: github.com/grafana/metrictank/idx/memory.(*MemoryIdx).find(0xc420828ae0, 0x1, 0xcc8421e400, 0x71, 0xcf6f58, 0x12, 0x4, 0x12, 0x4)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/idx/memory/memory.go:912 +0x65c
Jan  4 11:43:27 metrictank002 metrictank[6702]: github.com/grafana/metrictank/idx/memory.(*MemoryIdx).Find(0xc420828ae0, 0x1, 0xcc8421e400, 0x71, 0x5a4dbf00, 0x0, 0x0, 0x0, 0x0, 0x0)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/idx/memory/memory.go:807 +0x10b
Jan  4 11:43:27 metrictank002 metrictank[6702]: github.com/grafana/metrictank/api.(*Server).findSeriesLocal(0xc420905860, 0x1111240, 0xcb12868640, 0x1, 0xc80da5a9d0, 0x1, 0x1, 0x5a4dbf00, 0x0, 0x0, ...)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/api/graphite.go:133 +0x45c
Jan  4 11:43:27 metrictank002 metrictank[6702]: github.com/grafana/metrictank/api.(*Server).findSeries.func1(0xc420905860, 0x1111240, 0xcb12868640, 0x1, 0xc80da5a9d0, 0x1, 0x1, 0x5a4dbf00, 0xc80da5a9e0, 0xc9216ca5a0, ...)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/api/graphite.go:76 +0x9b
Jan  4 11:43:27 metrictank002 metrictank[6702]: created by github.com/grafana/metrictank/api.(*Server).findSeries
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/api/graphite.go:75 +0x495

[motyla]

@Dieterbe, seems that this looks the same as old issue where opening root namespace in grafana query editor crash the nodes

[Dieterbe]

for an instance that crashes like this, have you run any specific index related commands such as metric deletes? if not, this looks like there's some corrupt state in your index, specifically some child node is nil.
looking at the code, it could either be the startNode or one of its children, or grandchildren. enabling debug logging would help clarify, or getting an index dump may also reveal the problem.

we could also write a custom utility that reads your index, and analyzes the index around where the problem lies, and that should reveal the problem, but that would be custom development work.

[motyla]

thanks @Dieterbe ,
No , we did not ran any index related commands and it's more likely we have a corrupted index.
I think it's related to a bad data ingestion we made due to a misconfiguration at the source.

We will discuss it here this week to see how to handle.

[Dieterbe]

this looks like the same problem as #770 . i have pushed some extra logging via #812 , please run that code and share the error messages you get.

[motyla]

@Dieterbe , here is the log output (now in the right place):

2018/01/08 14:27:52 [D] HTTP Render querying metrictank003/index/find for 1:["*"]
2018/01/08 14:27:52 [D] memory-idx: found first pattern sequence at node * pos 0
2018/01/08 14:27:52 [D] memory-idx: starting search at the root node
2018/01/08 14:27:52 [D] memory-idx: found first pattern sequence at node * pos 0
2018/01/08 14:27:52 [D] memory-idx: searching 17 children of  that match *
2018/01/08 14:27:52 [D] memory-idx: Matching all children
2018/01/08 14:27:52 [D] memory-idx: starting search at the root node
2018/01/08 14:27:52 [D] memory-idx: searching 17 children of  that match *
2018/01/08 14:27:52 [D] memory-idx: Matching all children
2018/01/08 14:27:52 [D] memory-idx: reached pattern length. 17 nodes matched
2018/01/08 14:27:52 [D] memory-idx: reached pattern length. 17 nodes matched
2018/01/08 14:27:52 [D] memory-idx: orgId -1 has no metrics indexed.
2018/01/08 14:27:52 [D] memory-idx: orgId -1 has no metrics indexed.
2018/01/08 14:27:52 [D] memory-idx: 17 nodes matching pattern * found
2018/01/08 14:27:52 [D] memory-idx: 17 nodes matching pattern * found
2018/01/08 14:27:52 [D] memory-idx Find: adding to path host archive id=1.3688022064380aeae3228ecdd381adc7 name=host int=60 schemaId=28 aggId=0 lastSave=1515421173
2018/01/08 14:27:52 [D] memory-idx Find: adding to path host archive id=1.3688022064380aeae3228ecdd381adc7 name=host int=60 schemaId=28 aggId=0 lastSave=1515421173
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0xa68c35]

goroutine 233965 [running]:
github.com/grafana/metrictank/idx/memory.(*MemoryIdx).Find(0xc420bb74a0, 0x1, 0xc81496f845, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /home/circleci/.go_workspace/src/github.com/grafana/metrictank/idx/memory/memory.go:822 +0x475
github.com/grafana/metrictank/api.(*Server).findSeriesLocal(0xc4202f0460, 0x1116280, 0xcb7267d900, 0x1, 0xc51208c300, 0x1, 0x1, 0x0, 0x0, 0x0, ...)
        /home/circleci/.go_workspace/src/github.com/grafana/metrictank/api/graphite.go:133 +0x432
github.com/grafana/metrictank/api.(*Server).findSeries.func1(0xc4202f0460, 0x1116280, 0xcb7267d900, 0x1, 0xc51208c300, 0x1, 0x1, 0x0, 0xc51208c7a0, 0xcbc8fd0360, ...)
        /home/circleci/.go_workspace/src/github.com/grafana/metrictank/api/graphite.go:76 +0x9b
created by github.com/grafana/metrictank/api.(*Server).findSeries
        /home/circleci/.go_workspace/src/github.com/grafana/metrictank/api/graphite.go:75 +0x44d

[Dieterbe]

that log output is not as helpful as what you posted in #812 because this one doesn't show any errors.
but in that ticket, we have the clue i needed:

Jan  9 09:21:16 metrictank004 metrictank[36063]: 2018/01/09 09:21:16 [memory.go:934 find()] [E] memory-idx: grandChild is nil. o
rg=1,patt="*",i=0,pos=0,p="*",path="megaraidsas-metrics"

most likely the problem is those metrics are malformed, as mentioned in the other ticket.
'.megaraidsas-metrics' (starting with a dot)

2 things need to happen:

1. stop sending the badly formed ones. then, delete the bad entries from the index and restart MT. let me know if you need further assistance
2. we should not accept these metrics. how did you ingest them? via kafka-mdm or via carbon?

[Dieterbe]

seems like we had this same issue before (#668) and fixed it via #694 but that was specifically carbon, so please let me know if you used carbon or kafka, and in kafka, how do you put them into kafka? where are these metrics being generated and/or transformed?

[motyla]

We push metrics directly to Kafka and also through carbon-relay-ng. While getting the patch from you we used it to prevent crashes and we also started to put safeties where needed. In our code that write directly and with blacklists in carbon.

From what I saw , bad metrics came from both directions

This is my carbon-relay-ng blacklist:

'regex .*\.$',
'regex .*\.\..*',
'regex ^\..*',

[Dieterbe]

good news. with raintank/fakemetrics#9 i was able to trivially reproduce.
sending metric with name ".foo.bar" and trying to browse in grafana results in the same error:

metrictank_1    | [Macaron] 2018-08-21 13:16:25: Started GET /metrics/find?from=1534835724&query=*&until=1534857444 for 172.18.0.1, 172.18.0.1, 172.18.0.10
metrictank_1    | [Macaron] 2018-08-21 13:16:25: Started POST /index/find for 127.0.0.1
metrictank_1    | [Macaron] 2018-08-21 13:16:25: Completed /index/find 500 Internal Server Error in 445.8µs
metrictank_1    | 2018/08/21 13:16:25 [memory.go:1011 find()] [E] memory-idx: grandChild is nil. org=1,patt="*",i=0,pos=0,p="*",path="foo"
metrictank_1    | [Macaron] 2018-08-21 13:16:25: Completed /metrics/find?from=1534835724&query=*&until=1534857444 200 OK in 1.258231ms
metrictank_1    | 2018/08/21 13:16:25 [asm_amd64.s:2361 goexit()] [E] HTTP Render error querying default/index/find: "500 Internal Server Error"

our options are:

1. graciously allow: in MT adjust incoming data accordingly.
2. invalidate in MT, but that also means invalidate in tsdb-gw; in carbon-relay-ng we can strip it when converting from carbon to metrics2.0, this would be ideal, but unrealistic as we can't expect everyone to upgrade their carbon-relay-ng, so MT will just have to adjust the data.

in this process i also found out not only does graphite allow 1 leading dot, it seems to allow an unlimited amount, and will just chomp them all of. it also reduces sequences of dots in the middle of metric keys, so the previous fix for carbon (#694) will have to be improved.

creating a new ticket, #1008 to track this