dep ensure produces incomplete vendor results

[syphoxy]

I believe this explains itself. my code won't compile if I use dep as my vendor appears to be missing source files. I've checked that the missing directories do actually contain .go files.

this is how I check out a working vendor dependency:

git clone https://github.com/moby/moby.git vendor/github.com/docker/docker
git -C vendor/github.com/docker/docker checkout -b api/v1.32 05121d555475b3d7cfa5b29f22dc85262cd1a1a8

this is what I have in Gopkg.toml:

[[constraint]]
  name = "github.com/docker/docker"
  revision = "05121d555475b3d7cfa5b29f22dc85262cd1a1a8"
  source = "https://github.com/moby/moby.git"

I've verified using api/common.go that in both instances the code is checked out from the correct revision. it appears that dep simply just didn't get some .go files.

for example, common.go has no differences:

% diff -u vendor/github.com/docker/docker{,2}/api/common.go
%

I've also used go ensure -v to see additional output but unfortunately the output does not detail exactly what occurred here.

any help would be appreciated. thank you.

also: I realize that constraining to a revision is an anti-pattern but unfortunately docker/moby upstream developers have failed to properly add a tag for the API version I need to use in my software. please don't remind me.

What version of dep are you using (dep version)?

dep:
 version     : v0.4.1
 build date  : 2018-01-27
 git hash    : 37d9ea0
 go version  : go1.9.3
 go compiler : gc
 platform    : darwin/amd64

What dep command did you run?

dep ensure

What did you expect to see?

docker % ls -l
total 824
-rw-r--r--    1 dariorusso  staff   74397  8 Jun 13:55 AUTHORS
-rw-r--r--    1 dariorusso  staff  206320  8 Jun 13:55 CHANGELOG.md
-rw-r--r--    1 dariorusso  staff   19924  8 Jun 13:55 CONTRIBUTING.md
-rw-r--r--    1 dariorusso  staff    9121  8 Jun 13:55 Dockerfile
-rw-r--r--    1 dariorusso  staff    2631  8 Jun 13:55 Dockerfile.e2e
-rw-r--r--    1 dariorusso  staff    1987  8 Jun 13:55 Dockerfile.simple
-rw-r--r--    1 dariorusso  staff   11376  8 Jun 13:55 Dockerfile.windows
-rw-r--r--    1 dariorusso  staff   10765  8 Jun 13:55 LICENSE
-rw-r--r--    1 dariorusso  staff   15157  8 Jun 13:55 MAINTAINERS
-rw-r--r--    1 dariorusso  staff    9878  8 Jun 13:55 Makefile
-rw-r--r--    1 dariorusso  staff     638  8 Jun 13:55 NOTICE
-rw-r--r--    1 dariorusso  staff    3395  8 Jun 13:55 README.md
-rw-r--r--    1 dariorusso  staff    3776  8 Jun 13:55 ROADMAP.md
-rw-r--r--    1 dariorusso  staff    2664  8 Jun 13:55 TESTING.md
-rw-r--r--    1 dariorusso  staff    1952  8 Jun 13:55 VENDORING.md
drwxr-xr-x   11 dariorusso  staff     374  8 Jun 13:55 api/
drwxr-xr-x    7 dariorusso  staff     238  8 Jun 13:55 builder/
drwxr-xr-x    7 dariorusso  staff     238  8 Jun 13:55 cli/
drwxr-xr-x  214 dariorusso  staff    7276  8 Jun 13:55 client/
drwxr-xr-x    3 dariorusso  staff     102  8 Jun 13:55 cmd/
-rw-r--r--    1 dariorusso  staff     359  8 Jun 13:55 codecov.yml
drwxr-xr-x   22 dariorusso  staff     748  8 Jun 13:55 container/
drwxr-xr-x   33 dariorusso  staff    1122  8 Jun 13:55 contrib/
drwxr-xr-x  148 dariorusso  staff    5032  8 Jun 13:55 daemon/
drwxr-xr-x   21 dariorusso  staff     714  8 Jun 13:55 distribution/
drwxr-xr-x    4 dariorusso  staff     136  8 Jun 13:55 dockerversion/
drwxr-xr-x    5 dariorusso  staff     170  8 Jun 13:55 docs/
drwxr-xr-x    7 dariorusso  staff     238  8 Jun 13:55 errdefs/
drwxr-xr-x   15 dariorusso  staff     510  8 Jun 13:55 hack/
drwxr-xr-x   13 dariorusso  staff     442  8 Jun 13:55 image/
drwxr-xr-x   16 dariorusso  staff     544  8 Jun 13:55 integration/
drwxr-xr-x  115 dariorusso  staff    3910  8 Jun 13:55 integration-cli/
drwxr-xr-x    4 dariorusso  staff     136  8 Jun 13:55 internal/
drwxr-xr-x   21 dariorusso  staff     714  8 Jun 13:55 layer/
drwxr-xr-x   22 dariorusso  staff     748  8 Jun 13:55 libcontainerd/
drwxr-xr-x    3 dariorusso  staff     102  8 Jun 13:55 migrate/
drwxr-xr-x    6 dariorusso  staff     204  8 Jun 13:55 oci/
drwxr-xr-x   21 dariorusso  staff     714  8 Jun 13:55 opts/
drwxr-xr-x   47 dariorusso  staff    1598  8 Jun 13:55 pkg/
drwxr-xr-x   18 dariorusso  staff     612  8 Jun 13:55 plugin/
-rw-r--r--    1 dariorusso  staff    4667  8 Jun 13:55 poule.yml
drwxr-xr-x    4 dariorusso  staff     136  8 Jun 13:55 profiles/
drwxr-xr-x   16 dariorusso  staff     544  8 Jun 13:55 project/
drwxr-xr-x    5 dariorusso  staff     170  8 Jun 13:55 reference/
drwxr-xr-x   21 dariorusso  staff     714  8 Jun 13:55 registry/
drwxr-xr-x    9 dariorusso  staff     306  8 Jun 13:55 reports/
drwxr-xr-x    4 dariorusso  staff     136  8 Jun 13:55 restartmanager/
drwxr-xr-x   14 dariorusso  staff     476  8 Jun 13:55 runconfig/
drwxr-xr-x    8 dariorusso  staff     272  8 Jun 13:55 vendor/
-rw-r--r--    1 dariorusso  staff    7813  8 Jun 13:55 vendor.conf
drwxr-xr-x    8 dariorusso  staff     272  8 Jun 13:55 volume/

% rg DefaultVersion vendor/github.com/docker/docker/api/common.go
17:	// DefaultVersion of Current REST API
18:	DefaultVersion string = "1.32"

What did you see instead?

docker % ls -l
total 176
-rw-r--r--    1 dariorusso  staff  70478  8 Jun 13:47 AUTHORS
-rw-r--r--    1 dariorusso  staff  10765  8 Jun 13:47 LICENSE
-rw-r--r--    1 dariorusso  staff    638  8 Jun 13:47 NOTICE
drwxr-xr-x   10 dariorusso  staff    340  8 Jun 13:47 api/
drwxr-xr-x  117 dariorusso  staff   3978  8 Jun 13:47 client/
drwxr-xr-x    5 dariorusso  staff    170  8 Jun 13:47 contrib/
drwxr-xr-x    3 dariorusso  staff    102  8 Jun 13:47 docs/
drwxr-xr-x    4 dariorusso  staff    136  8 Jun 13:47 hack/
drwxr-xr-x    8 dariorusso  staff    272  8 Jun 13:47 pkg/
drwxr-xr-x    3 dariorusso  staff    102  8 Jun 13:47 project/

% rg DefaultVersion vendor/github.com/docker/docker/api/common.go
17:	// DefaultVersion of Current REST API
18:	DefaultVersion string = "1.32"

[diogobeda]

Getting the same issue when I run dep ensure -add github.com/scylladb/gocqlx. It only copies a portion of the files to my vendor folder, so I can't use some packages.
This is the verbose output:

Fetching sources...
(1/1) github.com/scylladb/gocqlx

Root project is "github.com/diogobeda/todo_api"
 4 transitively valid internal packages
 3 external packages imported from 3 projects
(0)   ✓ select (root)
(1)     ? attempt github.com/gocql/gocql with 1 pkgs; at least 1 versions to try
(1)         try github.com/gocql/gocql@master
(1)     ✓ select github.com/gocql/gocql@master w/4 pkgs
(2)     ? attempt github.com/golang/snappy with 1 pkgs; at least 1 versions to try
(2)         try github.com/golang/snappy@master
(2)     ✓ select github.com/golang/snappy@master w/1 pkgs
(3)     ? attempt github.com/hailocab/go-hostpool with 1 pkgs; at least 1 versions to try
(3)         try github.com/hailocab/go-hostpool@master
(3)     ✓ select github.com/hailocab/go-hostpool@master w/1 pkgs
(4)     ? attempt github.com/gorilla/mux with 1 pkgs; at least 1 versions to try
(4)         try github.com/gorilla/mux@v1.6.2
(4)     ✓ select github.com/gorilla/mux@v1.6.2 w/1 pkgs
(5)     ? attempt github.com/gorilla/context with 1 pkgs; at least 1 versions to try
(5)         try github.com/gorilla/context@v1.1.1
(5)     ✓ select github.com/gorilla/context@v1.1.1 w/1 pkgs
(6)     ? attempt gopkg.in/inf.v0 with 1 pkgs; at least 1 versions to try
(6)         try gopkg.in/inf.v0@v0.9.1
(6)     ✓ select gopkg.in/inf.v0@v0.9.1 w/1 pkgs
(7)     ? attempt github.com/scylladb/gocqlx with 1 pkgs; 2 versions to try
(7)         try github.com/scylladb/gocqlx@v1.0.0
(7)     ✓ select github.com/scylladb/gocqlx@v1.0.0 w/2 pkgs
  ✓ found solution with 11 packages from 7 projects

Solver wall times by segment:
     b-source-exists: 5.000274931s
         b-list-pkgs: 2.372990581s
              b-gmal: 2.346022451s
             satisfy:    597.225µs
         select-atom:    489.671µs
            new-atom:    429.323µs
  b-deduce-proj-root:    177.699µs
         select-root:    176.488µs
               other:     43.613µs
     b-list-versions:     12.029µs

  TOTAL: 9.721214011s

(1/7) Wrote github.com/hailocab/go-hostpool@master
(2/7) Wrote gopkg.in/inf.v0@v0.9.1
(3/7) Wrote github.com/gorilla/context@v1.1.1
(4/7) Wrote github.com/gorilla/mux@v1.6.2
(5/7) Wrote github.com/golang/snappy@master
(6/7) Wrote github.com/scylladb/gocqlx@v1.0.0
(7/7) Wrote github.com/gocql/gocql@master
"github.com/scylladb/gocqlx" is not imported by your project, and has been temporarily added to Gopkg.lock and vendor/.
If you run "dep ensure" again before actually importing it, it will disappear from Gopkg.lock and vendor/.

I checked the gocqlx project folder under $GOPATH/pkg/deps/sources and it is complete. So the issue may be when copying the files?

[sdboyer]

Thanks for your patience with slow responses here - i've been quite consumed with #1912. Nearly done with that, though, and release coming this week!

So the simplest explanation here is that, somehow, dep is pruning something it shouldn't. Step one for that would be adding this in Gopkg.toml:

[prune]
# (whatever your current global prune settings are)

  [[prune.project]]
    name = "github.com/docker/docker"
    unused-packages = false

That, at least, should make sure all of docker is there. However, none of this should be necessary; dep is only pruning out packages that are not a part of the package graph you actually use, directly or transitively. If that's not the case, something's really wrong (and this would be the first instance of it happening). Without those pruning options on, what are the actual compile errors you're seeing that lead you to believe you're missing necessary code?

Also - any chance there are submodules involved? If so, #1909 could be the culprit. Barring pruning issues (which also don't really make sense, per the above), that's my best immediate guess about missing files.

[syphoxy]

thanks for your response. no worries about the delay. I made do in the meantime with some ad hoc git checkouts.

using unused-packages = false worked fine for me. this is apparently in the documentation but it didn't occur to me that it was getting pruned. thanks a lot for pointing that out to me!

I am not using submodules in my respository. doesn't appear that docker/docker is using submodules either.

I'm not sure if this issue is a dup so I'll leave it to you to remain open or to close it.

thanks for all your efforts! much appreciated!

[sdboyer]

I'm glad it helped! but, this is actually a big concern - could you tell me what compilation errors you were getting prior to changing the pruning options?

[syphoxy]

@sdboyer oh, for sure! let me pull the compilation errors for you.

---

What version of dep are you using (dep version)?

% dep version
dep:
 version     : v0.4.1
 build date  : 2018-01-27
 git hash    : 37d9ea0
 go version  : go1.9.3
 go compiler : gc
 platform    : darwin/amd64

What dep command did you run?

% dep ensure
% make
go build redacted/tools/redacted/cmd/redacted
# redacted/tools/redacted/vendor/github.com/docker/docker/client
vendor/github.com/docker/docker/client/container_commit.go:17:15: undefined: reference.ParseNormalizedNamed
vendor/github.com/docker/docker/client/container_commit.go:25:9: undefined: reference.TagNameOnly
vendor/github.com/docker/docker/client/container_commit.go:30:16: undefined: reference.FamiliarName
vendor/github.com/docker/docker/client/image_create.go:16:14: undefined: reference.ParseNormalizedNamed
vendor/github.com/docker/docker/client/image_create.go:22:25: undefined: reference.FamiliarName
vendor/github.com/docker/docker/client/image_import.go:18:16: undefined: reference.ParseNormalizedNamed
vendor/github.com/docker/docker/client/image_pull.go:23:14: undefined: reference.ParseNormalizedNamed
vendor/github.com/docker/docker/client/image_pull.go:29:25: undefined: reference.FamiliarName
vendor/github.com/docker/docker/client/image_pull.go:56:8: undefined: reference.TagNameOnly
vendor/github.com/docker/docker/client/image_push.go:20:14: undefined: reference.ParseNormalizedNamed
vendor/github.com/docker/docker/client/image_push.go:20:14: too many errors
make: *** [redacted] Error 2

---

in case it helps, these are the relevant import statements:

import (
  "github.com/docker/docker/api/types"
  "github.com/docker/docker/api/types/filters"
  "github.com/docker/docker/client"
)

[sdboyer]

@syphoxy ohhh, ok - i tried those imports, and it looks like it's something else: dep was picking an older tagged release of github.com/docker/distribution, but it actually needed master, as those newer symbols were introduced after that latest release.

It's very odd that changing the pruning options would affect outcomes in this way, as there's no way those options can changing solving/version selection behavior, and the only way i can imagine the compilation errors going away is if versions changed.

Any chance you could expand this to more precise reproduction steps? And/or, could you have a look at the [[project]] stanza in Gopkg.lock for github.com/docker/distribution for both when you see the compilation errors, and when you don't?

[mohammed90]

I have the same issue with github.com/satori/go.uuid and changing the pruning config as above doesn't fix it. Of the whole package, only the uuid.go ends up in the directory. I can reproduce it with a project consisting of only one file (content below):

package main

import (
	"fmt"

	"github.com/satori/go.uuid"
)

func main() {

	fmt.Println(uuid.Must(uuid.NewV4()).String())
}

dep version:

dep:
 version     : v0.4.1
 build date  : 2018-01-27
 git hash    : 37d9ea0
 go version  : go1.9.3
 go compiler : gc
 platform    : darwin/amd64

[syphoxy]

@sdboyer strong observation you've made. it is true that I had to add an [[override]] stanza for github.com/docker/distribution because it was pulling an old tag (and thus old incompatible code). I didn't mention this because I thought it was unrelated. sorry!

it appears that when I leave the constraint override in and remove the unused-packages = false prune override, it works as expected as well.

this is the final Gopkg.toml (sans prune override stanza) that works for me:

[[constraint]]
  name = "github.com/docker/docker"
  revision = "05121d555475b3d7cfa5b29f22dc85262cd1a1a8" # API v1.32
  source = "https://github.com/moby/moby.git"

[[constraint]]
  name = "github.com/pkg/errors"
  version = "0.8.0"

[[override]]
  name = "github.com/docker/distribution"
  revision = "245ca4659e09e9745f3cc1217bf56e946509220c" # API v1.32
  source = "https://github.com/docker/distribution"

[prune]
  go-tests = true
  unused-packages = true

does this mean the issue is entirely on github.com/docker/docker and github.com/docker/distribution for not providing appropriate tags and such?

[sdboyer]

@syphoxy sorta on them, yeah. There's some more we could do to get it right - and those things are going to become front-and-center for me after i roll this new release this week. But, basically, yeah, it's not great that there are tagged releases of docker that effectively rely on certain branch positions of its dependencies. 😢

@mohammed90 that's really weird. i can't replicate it locally - i get github.com/satori/go.uuid@1.2.0, and the resulting vendor dir looks like this:

[testuuid] ls -la vendor/github.com/satori/go.uuid
total 80
drwxr-xr-x  9 sdboyer  staff   306 Jul 11 02:13 .
drwxr-xr-x  3 sdboyer  staff   102 Jul 11 02:13 ..
-rw-r--r--  1 sdboyer  staff   377 Jul 11 02:13 .travis.yml
-rw-r--r--  1 sdboyer  staff  1082 Jul 11 02:13 LICENSE
-rw-r--r--  1 sdboyer  staff  1885 Jul 11 02:13 README.md
-rw-r--r--  1 sdboyer  staff  6029 Jul 11 02:13 codec.go
-rw-r--r--  1 sdboyer  staff  6003 Jul 11 02:13 generator.go
-rw-r--r--  1 sdboyer  staff  2382 Jul 11 02:13 sql.go
-rw-r--r--  1 sdboyer  staff  4184 Jul 11 02:13 uuid.go

Any further suggestions on how i might reproduce it?

[mohammed90]

Hmm. With no constraints specified, it pulls github.com/satori/go.uuid@master with most the files missing. When I add the override stanza to the Gopkg.toml specifying v1.2.0, it pulls all the expected files correctly.

[[override]]
  name = "github.com/satori/go.uuid"
  revision = "v1.2.0"

[sdboyer]

OK, so that suggests that there's no structural problem here with writing out vendor - only the known issues with UX. gonna close this one out.

[rkand4]

For someone facing with this horrendous issue. The following helped because I wanted to use docker client with wait for container to launch and stuff:

[prune]
  go-tests = true
  unused-packages = true

[[constraint]]
  name = "github.com/docker/docker"
  source = "https://github.com/moby/moby.git"
  branch = "master"

[[constraint]]
  name = "github.com/pkg/errors"
  version = "0.8.0"

[[override]]
  name = "github.com/docker/distribution"
  source = "https://github.com/docker/distribution"
  branch = "master"