cmd/go: go mod fails to find direct dependencies

[jeanbza]

What version of Go are you using (go version)?

$ go version
go version go1.11.2 darwin/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/deklerk/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/deklerk/workspace/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/lk/zs4m7sv12mq2vzk_wfn2tfvm00h16k/T/go-build259032920=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

git clone https://github.com/golang/oauth2.git
cd oauth2/google
go mod init
go mod tidy

What did you expect to see?

A go.mod file with,

module golang.org/x/oauth2/google

require (
	cloud.google.com/go/compute/metadata v0.0.0-20181107005212-dafb9c8d8707
	golang.org/x/oauth2 v0.0.0-20181106182150-f42d05182288
	google.golang.org/appengine v1.3.0
)

And a populated go.sum.

What did you see instead?

module golang.org/x/oauth2/google

And an empty go.sum.

Note: another weirdness is that when I run go.vendor, I get:

$ go mod vendor
go: finding google.golang.org/appengine/urlfetch latest
go: finding golang.org/x/oauth2 latest
go: finding golang.org/x/oauth2/internal latest
go: finding golang.org/x/oauth2/jws latest
go: finding golang.org/x/oauth2/jwt latest
go: finding cloud.google.com/go/compute/metadata latest
go: no dependencies to vendor
$ cat go.mod
module golang.org/x/oauth2/google

require (
	cloud.google.com/go/compute/metadata v0.0.0-20181107005212-dafb9c8d8707 // indirect
	golang.org/x/oauth2 v0.0.0-20181106182150-f42d05182288 // indirect
	google.golang.org/appengine v1.3.0 // indirect
)

It found several dependencies, but it labeled them indirect. That's odd, because golang.org/x/oauth2/google directly depends on all of these. Furthermore, go mod vendor then goes on to state no dependencies to vendor immediately after it just vendored a bunch of things.

[myitcv]

Is there a reason you did the go mod init inside the google directory?

Doing it at the top level directory works just fine:

$ git clone https://github.com/golang/oauth2.git
Cloning into 'oauth2'...
$ cd oauth2
$ go mod init
go: creating new go.mod: module golang.org/x/oauth2
$ go mod tidy
go: finding google.golang.org/appengine/urlfetch latest
go: finding google.golang.org/appengine v1.3.0
go: finding golang.org/x/net/context/ctxhttp latest
go: finding cloud.google.com/go/compute/metadata latest
go: finding golang.org/x/net/context latest
go: finding golang.org/x/net latest
go: downloading golang.org/x/net v0.0.0-20181106065722-10aee1819953
go: downloading cloud.google.com/go/compute/metadata v0.0.0-20181107005212-dafb9c8d8707
go: downloading google.golang.org/appengine v1.3.0
go: finding golang.org/x/net v0.0.0-20180724234803-3673e40ba225
go: finding github.com/golang/protobuf v1.2.0
go: finding golang.org/x/text v0.3.0
go: downloading github.com/golang/protobuf v1.2.0
go: finding golang.org/x/sync/errgroup latest
go: finding golang.org/x/sync latest
go: downloading golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f
$ cat go.mod
module golang.org/x/oauth2

require (
        cloud.google.com/go/compute/metadata v0.0.0-20181107005212-dafb9c8d8707
        golang.org/x/net v0.0.0-20181106065722-10aee1819953
        golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f // indirect
        google.golang.org/appengine v1.3.0
)
$ go list -m all
golang.org/x/oauth2
cloud.google.com/go/compute/metadata v0.0.0-20181107005212-dafb9c8d8707
github.com/golang/protobuf v1.2.0
golang.org/x/net v0.0.0-20181106065722-10aee1819953
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f
golang.org/x/text v0.3.0
google.golang.org/appengine v1.3.0

I suspect the issue here in the case you report is that:

go: finding golang.org/x/oauth2 latest

find a package golang.org/x/oauth2/google that in effect clashes with the main module, but that this doesn't get reported/handled in the right way.

Can you give details about what you were trying to achieve here?

[jeanbza]

@myitcv Certainly! I was attempting to create an oauth2/google module that can be depended upon and versioned independent of all the other oauth2 subpackages. I had been working under the assumption that's something you can do - please let me know if not!

FWIW, I also tried go mod init at the parent before doing the go mod init in the child, and that didn't seem to have different results.

[myitcv]

@jadekler ah, I see. Yes, submodules (also known as multi-module repos, indeed submodules are a special instance of multi-module repos) are supported. Here's an example:

https://github.com/go-modules-by-example/index/blob/master/009_submodules/README.md

The sequencing of things is the slightly tough bit.

But, I will just draw your attention to #27858 and #28136. The last time the question of splitting x/tools into modules was discussed on the golang-tools fortnightly call the answer from @bcmills was that @rsc et al are thinking about how submodules could/should work and how to version x/* along with Go releases. It's a slowly evolving picture.

I'll leave this issue open however, because I think there is an outstanding issue that there should have been a more clear error in what you originally attempted.

[jeanbza]

Thank you very much! This also needs to happen for cloud.google.com/go, so thanks for the tutorial link. re:oauth2; sounds like I should punt on that for now and go sync up with y'all that are already thinking about this. Will check in with Hana/Bryan about that.

[myitcv]

You're welcome. The advice at the moment is definitely to only create submodules/multi-module repos when you really need to. One indicator of being in that position is:

• your major versions are >= 1
• a major version bump demanded by breaking changes in one subset of packages is not acceptable to other packages (because by definition the major version moves forward for all packages if the entire repo is a single module)

Would certainly be interested in hearing your feedback on submodules/multi-module repos here or over in #27858/#28136 so please keep us posted.

[bcmills]

a package golang.org/x/oauth2/google that in effect clashes with the main module, but that this doesn't get reported/handled in the right way.

Indeed. I believe that makes this a duplicate of #27063.

module$ git clone https://github.com/golang/oauth2.git .
module$ cd google
module/google$ go mod init
go: creating new go.mod: module golang.org/x/oauth2/google

module/google$ go mod tidy
go: finding google.golang.org/appengine v1.3.0
go: finding google.golang.org/appengine/urlfetch latest
go: downloading google.golang.org/appengine v1.3.0
go: finding golang.org/x/oauth2 latest
go: finding golang.org/x/oauth2/jws latest
go: finding golang.org/x/oauth2/internal latest
go: finding golang.org/x/oauth2/jwt latest
go: extracting google.golang.org/appengine v1.3.0
go: finding cloud.google.com/go/compute/metadata latest
go: downloading golang.org/x/oauth2 v0.0.0-20181106182150-f42d05182288
go: extracting golang.org/x/oauth2 v0.0.0-20181106182150-f42d05182288
go: downloading cloud.google.com/go/compute/metadata v0.0.0-20181114215827-b51c7c77b284
go: extracting cloud.google.com/go/compute/metadata v0.0.0-20181114215827-b51c7c77b284
go: finding github.com/golang/protobuf v1.2.0
go: finding golang.org/x/net v0.0.0-20180724234803-3673e40ba225
go: finding golang.org/x/text v0.3.0

module/google$ go list
go: finding golang.org/x/oauth2 latest
go: finding golang.org/x/oauth2/jws latest
go: finding golang.org/x/oauth2/internal latest
go: finding golang.org/x/oauth2/jwt latest
go: finding cloud.google.com/go/compute/metadata latest
can't load package: package golang.org/x/oauth2/google: unknown import path "golang.org/x/oauth2/google": ambiguous import: found golang.org/x/oauth2/google in multiple modules:
        golang.org/x/oauth2/google (/tmp/tmp.hhDmashDWj/module/google)
        golang.org/x/oauth2 v0.0.0-20181106182150-f42d05182288 (/tmp/tmp.hhDmashDWj/gopath/pkg/mod/golang.org/x/oauth2@v0.0.0-20181106182150-f42d05182288/google)