panic: runtime error: index out of range

[tomwilkie]

Seen just now in CI, haven't investigated yet:

panic: runtime error: index out of range

goroutine 1 [running]:
github.com/golang/dep/internal/gps.(*solver).findValidVersion(0xc42020c6c0, 0xc4205220a0, 0xc420211090, 0x1, 0x1, 0x0, 0xc420211090)
	/home/ubuntu/.go_workspace/src/github.com/golang/dep/internal/gps/solver.go:823 +0x562
github.com/golang/dep/internal/gps.(*solver).createVersionQueue(0xc42020c6c0, 0xc4202a8d40, 0x10, 0x0, 0x0, 0xc420211090, 0x1, 0x1, 0x0, 0x0, ...)
	/home/ubuntu/.go_workspace/src/github.com/golang/dep/internal/gps/solver.go:780 +0xafc
github.com/golang/dep/internal/gps.(*solver).solve(0xc42020c6c0, 0x0, 0x0, 0x2a)
	/home/ubuntu/.go_workspace/src/github.com/golang/dep/internal/gps/solver.go:397 +0x471
github.com/golang/dep/internal/gps.(*solver).Solve(0xc42020c6c0, 0x3b, 0xa760c0, 0xac4ab8, 0xc4200180e3)
	/home/ubuntu/.go_workspace/src/github.com/golang/dep/internal/gps/solver.go:348 +0xab
main.(*ensureCommand).Run(0xc420127960, 0xc420129880, 0xc42000c4a0, 0x0, 0x0, 0x0, 0x0)
	/home/ubuntu/.go_workspace/src/github.com/golang/dep/cmd/dep/ensure.go:149 +0x3c8
main.(*Config).Run(0xc42001ef60, 0xc42001ef60)
	/home/ubuntu/.go_workspace/src/github.com/golang/dep/cmd/dep/main.go:159 +0x8b3
main.main()
	/home/ubuntu/.go_workspace/src/github.com/golang/dep/cmd/dep/main.go:44 +0x253

https://circleci.com/gh/weaveworks/common/132

[sdboyer]

Definite bad bookkeeping there. Shouldn't be possible to be visiting a dep that isn't depended on by anything.

[tomwilkie]

Its reproducible on CI: https://circleci.com/gh/weaveworks/common/133

But not on my mac...

[sdboyer]

if that's a difference, then the first thing that jumps to mind is import path case sensitivity. i'm not specifically aware of a condition that could result this, but i'm generally aware that this is a problem area for us. any chance you've got some inconsistent casing in your import statements?

[tomwilkie]

A quick hack tells me no:

$ find . -type d | grep -v vendor  | grep -v tools | grep -v .git | grep -v .pkg | xargs go list  -f '{{ .Imports }}' | tr '[]' ' ' | tr ' ' '\n' | sort -u

bufio
bytes
errors
flag
fmt
github.com/weaveworks/common/errors
github.com/weaveworks/common/exec
github.com/weaveworks/common/fs
github.com/weaveworks/common/httpgrpc
github.com/weaveworks/common/mflag
github.com/weaveworks/common/middleware
github.com/weaveworks/common/signals
github.com/weaveworks/common/user
github.com/weaveworks/common/vendor/github.com/Sirupsen/logrus
github.com/weaveworks/common/vendor/github.com/davecgh/go-spew/spew
github.com/weaveworks/common/vendor/github.com/golang/protobuf/proto
github.com/weaveworks/common/vendor/github.com/golang/protobuf/ptypes
github.com/weaveworks/common/vendor/github.com/golang/protobuf/ptypes/any
github.com/weaveworks/common/vendor/github.com/gorilla/mux
github.com/weaveworks/common/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc
github.com/weaveworks/common/vendor/github.com/mwitkow/go-grpc-middleware
github.com/weaveworks/common/vendor/github.com/opentracing-contrib/go-stdlib/nethttp
github.com/weaveworks/common/vendor/github.com/opentracing/opentracing-go
github.com/weaveworks/common/vendor/github.com/opentracing/opentracing-go/ext
github.com/weaveworks/common/vendor/github.com/pmezard/go-difflib/difflib
github.com/weaveworks/common/vendor/github.com/prometheus/client_golang/prometheus
github.com/weaveworks/common/vendor/github.com/prometheus/common/log
github.com/weaveworks/common/vendor/github.com/sercand/kuberesolver
github.com/weaveworks/common/vendor/github.com/weaveworks-experiments/loki/pkg/client
github.com/weaveworks/common/vendor/golang.org/x/net/context
github.com/weaveworks/common/vendor/google.golang.org/genproto/googleapis/rpc/status
github.com/weaveworks/common/vendor/google.golang.org/grpc
github.com/weaveworks/common/vendor/google.golang.org/grpc/metadata
github.com/weaveworks/common/vendor/google.golang.org/grpc/status
io
io/ioutil
math
net
net/http
net/http/httptest
net/http/httputil
net/http/pprof
net/url
os
os/exec
os/signal
regexp
runtime
sort
strconv
strings
sync
syscall
text/tabwriter
time

[sdboyer]

yeah probably not - i mean, this COULD be a thing:

github.com/weaveworks/common/vendor/github.com/Sirupsen/logrus

b/c he has changed his username to lowercase, though that really should be all transparent and totally not be relevant here.

without actually looking at the code, i'm stumped 😦

[tomwilkie]

If I go for a lowecase sirupsen, I get a different error:

ensure Solve(): No versions of golang.org/x/sys met constraints:
	master: Unable to update checked out version: fatal: reference is not a tree: a55a76086885b80f79961eacb876ebd8caf3868d

	master: Unable to update checked out version: fatal: reference is not a tree: a55a76086885b80f79961eacb876ebd8caf3868d

https://circleci.com/gh/weaveworks/common/136

And I can do an ensure locally anymore:

 and vendor: error while writing out vendor tree: error while exporting github.com/sirupsen/logrus: /var/folders/_b/ktq_dxhx0nbbw7gjtdzb3tn40000gn/T/dep283060611/vendor/github.com/sirupsen/logrus/.gitignore already exists, no checkout
/var/folders/_b/ktq_dxhx0nbbw7gjtdzb3tn40000gn/T/dep283060611/vendor/github.com/sirupsen/logrus/.travis.yml already exists, no checkout
/var/folders/_b/ktq_dxhx0nbbw7gjtdzb3tn40000gn/T/dep283060611/vendor/github.com/sirupsen/logrus/CHANGELOG.md already exists, no checkout
/var/folders/_b/ktq_dxhx0nbbw7gjtdzb3tn40000gn/T/dep283060611/vendor/github.com/sirupsen/logrus/LICENSE already exists, no checkout
/var/folders/_b/ktq_dxhx0nbbw7gjtdzb3tn40000gn/T/dep283060611/vendor/github.com/sirupsen/logrus/README.md already exists, no checkout
/var/folders/_b/ktq_dxhx0nbbw7gjtdzb3tn40000gn/T/dep283060611/vendor/github.com/sirupsen/logrus/alt_exit.go already exists, no checkout
/var/folders/_b/ktq_dxhx0nbbw7gjtdzb3tn40000gn/T/dep283060611/vendor/github.com/sirupsen/logrus/alt_exit_test.go already exists, no checkout
/var/folders/_b/ktq_dxhx0nbbw7gjtdzb3tn40000gn/T/dep283060611/vendor/github.com/sirupsen/logrus/doc.go already exists, no checkout
/var/folders/_b/ktq_dxhx0nbbw7gjtdzb3tn40000gn/T/dep283060611/vendor/github.com/sirupsen/logrus/entry.go already exists, no checkout
/var/folders/_b/ktq_dxhx0nbbw7gjtdzb3tn40000gn/T/dep283060611/vendor/github.com/sirupsen/logrus/entry_test.go already exists, no checkout
/var/folders/_b/ktq_dxhx0nbbw7gjtdzb3tn40000gn/T/dep283060611/vendor/github.com/sirupsen/logrus/examples/basic/basic.go already exists, no checkout
/var/folders/_b/ktq_dxhx0nbbw7gjtdzb3tn40000gn/T/dep283060611/vendor/github.com/sirupsen/logrus/examples/hook/hook.go already exists, no checkout
/var/folders/_b/ktq_dxhx0nbbw7gjtdzb3tn40000gn/T/dep283060611/vendor/github.com/sirupsen/logrus/exported.go already exists, no checkout
/var/folders/_b/ktq_dxhx0nbbw7gjtdzb3tn40000gn/T/dep283060611/vendor/github.com/sirupsen/logrus/formatter.go already exists, no checkout
/var/folders/_b/ktq_dxhx0nbbw7gjtdzb3tn40000gn/T/dep283060611/vendor/github.com/sirupsen/logrus/formatter_bench_test.go already exists, no checkout
/var/folders/_b/ktq_dxhx0nbbw7gjtdzb3tn40000gn/T/dep283060611/vendor/github.com/sirupsen/logrus/hook_test.go already exists, no checkout
/var/folders/_b/ktq_dxhx0nbbw7gjtdzb3tn40000gn/T/dep283060611/vendor/github.com/sirupsen/logrus/hooks.go already exists, no checkout
/var/folders/_b/ktq_dxhx0nbbw7gjtdzb3tn40000gn/T/dep283060611/vendor/github.com/sirupsen/logrus/hooks/syslog/README.md already exists, no checkout
/var/folders/_b/ktq_dxhx0nbbw7gjtdzb3tn40000gn/T/dep283060611/vendor/github.com/sirupsen/logrus/hooks/syslog/syslog.go already exists, no checkout
/var/folders/_b/ktq_dxhx0nbbw7gjtdzb3tn40000gn/T/dep283060611/vendor/github.com/sirupsen/logrus/hooks/syslog/syslog_test.go already exists, no checkout
/var/folders/_b/ktq_dxhx0nbbw7gjtdzb3tn40000gn/T/dep283060611/vendor/github.com/sirupsen/logrus/hooks/test/test.go already exists, no checkout
/var/folders/_b/ktq_dxhx0nbbw7gjtdzb3tn40000gn/T/dep283060611/vendor/github.com/sirupsen/logrus/hooks/test/test_test.go already exists, no checkout
/var/folders/_b/ktq_dxhx0nbbw7gjtdzb3tn40000gn/T/dep283060611/vendor/github.com/sirupsen/logrus/json_formatter.go already exists, no checkout
/var/folders/_b/ktq_dxhx0nbbw7gjtdzb3tn40000gn/T/dep283060611/vendor/github.com/sirupsen/logrus/json_formatter_test.go already exists, no checkout
/var/folders/_b/ktq_dxhx0nbbw7gjtdzb3tn40000gn/T/dep283060611/vendor/github.com/sirupsen/logrus/logger.go already exists, no checkout
/var/folders/_b/ktq_dxhx0nbbw7gjtdzb3tn40000gn/T/dep283060611/vendor/github.com/sirupsen/logrus/logger_bench_test.go already exists, no checkout
/var/folders/_b/ktq_dxhx0nbbw7gjtdzb3tn40000gn/T/dep283060611/vendor/github.com/sirupsen/logrus/logrus.go already exists, no checkout
/var/folders/_b/ktq_dxhx0nbbw7gjtdzb3tn40000gn/T/dep283060611/vendor/github.com/sirupsen/logrus/logrus_test.go already exists, no checkout
/var/folders/_b/ktq_dxhx0nbbw7gjtdzb3tn40000gn/T/dep283060611/vendor/github.com/sirupsen/logrus/terminal_appengine.go already exists, no checkout
/var/folders/_b/ktq_dxhx0nbbw7gjtdzb3tn40000gn/T/dep283060611/vendor/github.com/sirupsen/logrus/terminal_bsd.go already exists, no checkout
/var/folders/_b/ktq_dxhx0nbbw7gjtdzb3tn40000gn/T/dep283060611/vendor/github.com/sirupsen/logrus/terminal_linux.go already exists, no checkout
/var/folders/_b/ktq_dxhx0nbbw7gjtdzb3tn40000gn/T/dep283060611/vendor/github.com/sirupsen/logrus/terminal_notwindows.go already exists, no checkout
/var/folders/_b/ktq_dxhx0nbbw7gjtdzb3tn40000gn/T/dep283060611/vendor/github.com/sirupsen/logrus/terminal_solaris.go already exists, no checkout
/var/folders/_b/ktq_dxhx0nbbw7gjtdzb3tn40000gn/T/dep283060611/vendor/github.com/sirupsen/logrus/terminal_windows.go already exists, no checkout
/var/folders/_b/ktq_dxhx0nbbw7gjtdzb3tn40000gn/T/dep283060611/vendor/github.com/sirupsen/logrus/text_formatter.go already exists, no checkout
/var/folders/_b/ktq_dxhx0nbbw7gjtdzb3tn40000gn/T/dep283060611/vendor/github.com/sirupsen/logrus/text_formatter_test.go already exists, no checkout
/var/folders/_b/ktq_dxhx0nbbw7gjtdzb3tn40000gn/T/dep283060611/vendor/github.com/sirupsen/logrus/writer.go already exists, no checkout
: exit status 128

But I guess thats because some of my vendored dependancies depend on the uppercase version, so both are getting pulled in.

[sdboyer]

But I guess thats because some of my vendored dependancies depend on the uppercase version, so both are getting pulled in.

yeah, that'll definitely do it. i had a long chat with someone in slack about this a while ago, i think we actually came to a reasonable, not-terrible solution. didn't make an issue for it, though 😢

ensure Solve(): No versions of golang.org/x/sys met constraints:
  master: Unable to update checked out version: fatal: reference is not a tree: a55a76086885b80f79961eacb876ebd8caf3868d

  master: Unable to update checked out version: fatal: reference is not a tree: a55a76086885b80f79961eacb876ebd8caf3868d

ok, we have to have had a regression, these stale cache errors are suddenly popping up again all over the place.

[tomwilkie]

It does indeed seem to be something wrong with golang.org/x/sys. With verbose ensuring:

(5)	? attempt golang.org/x/sys with 1 pkgs; at least 1 versions to try
(5)	    try golang.org/x/sys@master
(5)	✗   Unable to update checked out version: fatal: reference is not a tree: a55a76086885b80f79961eacb876ebd8caf3868d
(5)	    try golang.org/x/sys@master
(5)	✗   Unable to update checked out version: fatal: reference is not a tree: a55a76086885b80f79961eacb876ebd8caf3868d
panic: runtime error: index out of range

goroutine 1 [running]:
github.com/golang/dep/internal/gps.(*solver).findValidVersion(0xc420214c60, 0xc4203f9ae0, 0xc4201fa570, 0x1, 0x1, 0x0, 0xc4201fa570)
	/home/ubuntu/.go_workspace/src/github.com/golang/dep/internal/gps/solver.go:823 +0x562
github.com/golang/dep/internal/gps.(*solver).createVersionQueue(0xc420214c60, 0xc420128600, 0x10, 0x0, 0x0, 0xc4201fa570, 0x1, 0x1, 0x0, 0x0, ...)
	/home/ubuntu/.go_workspace/src/github.com/golang/dep/internal/gps/solver.go:780 +0xafc
github.com/golang/dep/internal/gps.(*solver).solve(0xc420214c60, 0x0, 0x0, 0x2a)
	/home/ubuntu/.go_workspace/src/github.com/golang/dep/internal/gps/solver.go:397 +0x471
github.com/golang/dep/internal/gps.(*solver).Solve(0xc420214c60, 0x3b, 0xa760c0, 0xac4ab8, 0xc420018163)
	/home/ubuntu/.go_workspace/src/github.com/golang/dep/internal/gps/solver.go:348 +0xab
main.(*ensureCommand).Run(0xc420127940, 0xc4201299c0, 0xc420010530, 0x0, 0x0, 0x0, 0x0)
	/home/ubuntu/.go_workspace/src/github.com/golang/dep/cmd/dep/ensure.go:149 +0x3c8
main.(*Config).Run(0xc42001afc0, 0xc42001afc0)
	/home/ubuntu/.go_workspace/src/github.com/golang/dep/cmd/dep/main.go:159 +0x8b3
main.main()
	/home/ubuntu/.go_workspace/src/github.com/golang/dep/cmd/dep/main.go:44 +0x253

go get -u github.com/golang/dep/cmd/dep && cd ${REPO_ROOT}/${CIRCLE_PROJECT_REPONAME} && dep ensure -v returned exit code 2

Action failed: go get -u github.com/golang/dep/cmd/dep && cd ${REPO_ROOT}/${CIRCLE_PROJECT_REPONAME} && dep ensure -v

[sdboyer]

arrrrrggghhhh. ok, well, that at least gives us a ballpark. thanks 😄

[tomwilkie]

Okay, if I try and use an older version of golang.org/x/sys, I get:

  ✗ solving failed

Solver wall times by segment:
     b-source-exists: 10.144091704s
         b-list-pkgs:  3.051609794s
              b-gmal:  2.252815166s
  b-deduce-proj-root:   430.96164ms
             satisfy:   37.716384ms
         select-atom:   37.603968ms
            unselect:    24.86761ms
            new-atom:   10.096922ms
           backtrack:    4.706169ms
         select-root:    1.075912ms
            add-atom:     452.476µs
               other:     149.914µs
     b-list-versions:     148.497µs

  TOTAL: 15.996296156s

ensure Solve(): No versions of golang.org/x/sys met constraints:
	master: Unable to update checked out version: fatal: reference is not a tree: da118f7b8e5954f39d0d2130ab35d4bf0e3cb344

	master: Unable to update checked out version: fatal: reference is not a tree: a55a76086885b80f79961eacb876ebd8caf3868d

https://circleci.com/gh/weaveworks/common/139

[tomwilkie]

And its something about this PR, as master is still ensuring fine: https://circleci.com/gh/weaveworks/common/140

[tomwilkie]

This looks to be the same / related to #639

[tomwilkie]

I've got it to not fail by reverting all my changes to the lock file and just updated grpc: weaveworks/common#37

Hopefully thats helpful.