cmd/go: ensure that 'go mod tidy' and `go get -u` do not introduce ambiguous imports

[bcmills]

As noted in #27858 (comment), when a package is moved between two modules that share one of the complete module paths as a prefix, each needs to require some baseline version of the other, for two reasons:

1. To ensure that go get -u doesn't drop any needed packages.

2. To ensure that any third module that depends on both of the common-prefix modules will always end up in a configuration that provides only one copy of each package.

It would be really unfortunate if moving a package meant that you could never run go mod tidy the involved modules again, or that you have to apply some manual edit every time you run it.

We should either modify go mod tidy to avoid dropping requirements that are involved in a cycle with the main module, or add some explicit annotation comment to tell it not to drop an apparently-unnecessary requirement.

[hyangah]

If two modules don't have actual dependency but require is needed for the second reason, there is no cycle to detect. Explicit annotation from the submodule (maybe #keep) seems easier.

[bcmills]

@hyangah, I'm not sure what you mean? If you're changing module boundaries, there should always be a cycle. (Reason # 1 imposes a requirement edge in one direction, while # 2 imposes the other direction: I can't think of a situation where we would need to preserve the edge but not have a cycle.)

[hyangah]

@bcmills my comment is about how to change go mod tidy to detect the (implicit) cycle automatically and prevents dropping the requirement (or even better, adding the implicit requirement automatically), not about the existence of such implicit cycle. The implicit cycle is not encoded in the import graph from .go files so the go mod tidy should go beyond the latest source code analysis. Moreover, the required version is the one that introduces the new module boundary, which is not automatically known yet until the commit to add the new boundary is checked in.

[thepudds]

As noted in #27858 (comment), when a package is moved between a module and one of is submodules, the module and submodule each need to require some baseline version of the other
...
We should either modify go mod tidy to avoid dropping requirements that are involved in a cycle with the main module, or add some explicit annotation comment to tell it not to drop an apparently-unnecessary requirement.

Sorry if this is just noise, and I wouldn't suggest this as a long term solution, but adding a comment here in case it triggers some discussion and/or a better idea.

To avoid go mod tidy removing cyclic dependencies that were put in place to allow a module to be broken apart by moving packages between modules, would it work in 1.11 to do something like add a set of modmove.go files or similar that contain import statements describing the required cyclic dependencies such that a later go mod tidy does not remove the require directives?

I have not tested this on an actual repo, but if you wanted to take a module example.com/my/module/ and break out example.com/my/module/foo into a separate module with its own go.mod, then after manually added the necessary cyclic require statements to the go.mod files, you could then encode those dependencies in a set of new modmove.go file, via something like:

In the root of the repo:

# 1. create a modmove.go for the parent module:

mkdir modmove
cat <<EOF > modmove/modmove.go
// +build modmove

package modmove

import (
        _ "example.com/my/module/foo"
)
EOF

# 2. create a modmove.go for the nested module foo:

mkdir foo/modmove
cat <<EOF > foo/modmove/modmove.go
// +build modmove

package modmove

import (
        _ "example.com/my/module"
)
EOF

In theory, a later go mod tidy would not remove the manually added cyclic require statements.

This technique would be somewhat analogous to the tools.go approach for tool dependencies as described in #25922 (comment), but applied in a different way against a different problem.

(Side note: example above creates modmove subdirectories. That might not be required given the modmove build tag?)

[bcmills]

@thepudds, that's a nice workaround, but it's slightly fragile: if the package that we put in the import line itself moves into or out of some other module with the same prefix, we could end up accidentally creating a cycle between the nested modules rather than the intended cycle with the parent module.

[thepudds]

@bcmills, agreed -- it would take some care to set up initially, and then take some care to not mess up later if there are additional moves.

Hopefully the core go tooling has some additional logic targeting these types of scenarios in 1.12 or so such that something like this is no longer needed.

That said, I was trying to think if there was something that could work in 1.11 (aside from "never run go mod tidy again”, or needing to repeatedly re-apply a manual edit). Perhaps modmoved.go or similar could be viewed as a candidate temporary workaround for 1.11 if someone needs to split a module in 1.11.

[rsc]

I don't think we can a priori assume that every cycle should be kept.
If I'm working on A and I know B uses A, I might run 'go get B@master; go test B/...'. That will create a cycle but one that I probably want 'go mod tidy' to remove.

[bcmills]

In https://golang.org/issue/27858#issuecomment-434713751, @jba suggests:

After a go get -u, the tool should see if any packages required by the main module are no longer in a module, and fetch those packages' modules in the usual way.

That would allow for a single require edge (to remove the duplicate copy) rather than a cycle.