cmd/go: add 'go get' options to update direct and indirect dependencies separately

[thepudds]

What version of Go are you using (go version)?

1.11.1

Summary

Users have more familiarity with their direct dependencies than their indirect dependencies. Indirect dependencies can also be larger in number, with greater potential combinations of pair-wise versions.

For these and other reasons, upgrading direct dependencies has a different risk profile than upgrading indirect dependencies.

Therefore:

• Consider providing users easier control to upgrade just their direct dependencies.
• Consider allowing an easier separation of upgrade strategy for indirect dependencies vs. the upgrade strategy for direct dependencies (e.g., perhaps minor version upgrades for direct dependencies, but only patch version upgrades for resulting indirect dependencies).

This might result in upgrade strategies that retain more benefits of 'High-Fidelity Builds', especially as compared to doing a simple go get -u or go get -u=patch.

Regardless of whether or not this particular suggestion ends up making sense, a more general goal would be to have a small basket of easy-to-run upgrade strategies that could be applied to something like 80-90% of projects.

Background

The "Update Timing & High-Fidelity Builds" section of the official proposal includes:

In the Bundler/Cargo/Dep approach, the package manager always prefers to use the latest version of any dependency. These systems use the lock file to override that behavior, holding the updates back. But lock files only apply to whole-program builds, not to newly imported libraries. If you are working on module A, and you add a new requirement on module B, which in turn requires module C, these systems will fetch the latest of B and then also the latest of C. In contrast, this proposal still fetches the latest of B (because it is what you are adding to the project explicitly, and the default is to take the latest of explicit additions) but then prefers to use the exact version of C that B requires. Although newer versions of C should work, it is safest to use the one that B did.
...
The minimal version selection blog post refers to this kind of build as a “high-fidelity build.”

This is a very nice set of properties of the overall modules system, and is materially different than a more traditional approach.

That section later goes on to say:

Many developers recoil at the idea that adding the latest B would not automatically also add the latest C, but if C was just released, there's no guarantee it works in this build. The more conservative position is to avoid using it until the user asks.
...
users are expected to update on their own schedule, so that they can control when they take on the risk of things breaking
...
If a developer does want to update all dependencies to the latest version, that's easy: go get -u. We may also add a go get -p that updates all dependencies to their latest patch versions

Issue

The initial starting point for a new set of dependencies in the modules system is more conservative than a more traditional approach, and as a result it is likely the case that you have better odds of starting with a working system. The ability to easily do go get -u to update all direct and indirect dependencies helps balance out that more conservative start.

However, once you do go get -u, you have stepped away to some degree from some of the benefits of "High-Fidelity Builds" at that point in time.

The same is true of go get -u=patch, though the step away is smaller.

Suggestion

Consider some form of providing for an easy upgrade just to direct dependencies.

Setting aside the actual mechanics (e.g., new flag vs. some other mechanism), if you could easily ask for to get the latest versions of your direct dependencies, e.g., via something like:

$ go get direct@latest

...or less likely, perhaps that same sentiment could be written something like:

$ go get -u -directonly

...or some other form that would ask to upgrade only your direct dependencies to the latest version available. That would mean in the common case the resulting versions for your indirect dependencies would be the ones listed in a require directive by at least one of your other dependencies, which would preserve many of the benefits of 'High-Fidelity Builds'.

In contrast, a simple go get -u often moves your indirect dependencies to versions beyond the versions listed in any require directive, and hence you might be using versions of modules that the module's importer in your build has never used or tested, or you otherwise might find yourself in a rare combination of versions involving indirect dependencies.

The author of the top-level build:

• often has the most insight into their direct dependencies (e.g., the author made a decision to use them, the top-level code is directly interacting with their direct dependencies).
• often has progressively less insight into indirect dependencies further and further down the chain
• often is not well positioned to chase down pair-wise incompatibility bugs that show up in their build deep in their dependency chain (e.g., in some indirect dependency 7 levels down).

For the rest of this write-up, we'll use go get -u -directonly as the strawman form (rather than go get direct@latest).

go get -u -directonly or similar could be viewed as a 'high-fidelity upgrade', though that might not be not good terminology.

If some form of mechanics for more easily upgrading direct dependencies was adopted, it could apply for patch upgrades as well, such as something like:

go get -u=patch -directonly

...which would update your direct dependencies to their latest patch releases.

In addition, it might make sense to also allow for easily specifying how you want to upgrade your indirect dependencies. If that was allowed, then for some projects a natural strategy for dependency upgrades might be:

$ go get -u -directonly
$ go get -u=patch -indirectonly

That would more conservative than go get -u, and slightly more aggressive than the hypothetical go get -u -directonly, but there is some risk mitigation for indirect dependencies by picking up the latest patch versions for indirect dependencies (go get -u=patch -indirectonly, which admittedly is not a great flag name).

Backing up:

• Different projects will almost certainly adopt different dependency upgrade strategies (due to different risk tolerances, different depth and width of their dependency trees, different levels of stability observed over time within their dependencies, etc.).
• There is a general question repeated within the community regarding best practice for library maintainers for dependency updates, including if libraries should:
  • "Ride the top", such as by always shipping with requirements for the latest version of all dependencies
  • "Ride the bottom", where require statements for a module have the true minimum supported version of dependencies
• Options along the lines of go get -u -directonly, go get -u=patch -directonly, go get -u=patch -indirectonly could provide simpler and better answers to those questions at least for some projects.

Alternatives

I think even today in 1.11 you can emulate the suggested behavior above, though it is not always natural or obvious.

Perhaps there is a simpler way today, but for example given the flexibility of go list, I suspect in 1.11 the following gives you the go get -u=patch -indirectonly behavior described above:

go get -u=patch $(go list -f '{{if not (or .Main .Indirect)}}{{.Path}}@{{.Version}}{{end}}' -m all)

To upgrade your direct dependencies to their latest release (the go get -u -directonly or go get direct@latest behavior described above), this should work in 1.11:

go get $(go list -f '{{if not (or .Main .Indirect)}}{{.Path}}{{end}}' -m all)

go mod edit -json and similar open up even more doors for greater control today.

[philippgille]

Great proposal! I just asked a related question in the Gopher Slack channel regarding how to update only my direct dependencies and use the indirect dependencies they require, because of fear of subtle changed behavior in latest indirect v0.x dependencies and got pointed to this issue.

I would love to see something like this in Go 1.14!

In case the Slack discussion is of any interest: https://gophers.slack.com/archives/C9BMAAFFB/p1570822899096300

[zikaeroh]

For what it's worth, I haven't found the original issue's go list command to be sufficient in the case of dependencies which aren't module aware. In order to emulate something like "remove go.mod and recreate it", leading to everything at latest (including // indirect dependencies), but also preserve existing replacements/manual versions, I have a script like this:

module=$(go list -f '{{.Module.Path}}' .)

go mod tidy
go get -d -t $(go mod graph | grep "^$module" | cut -d ' ' -f 2 | sed 's/@.*/@upgrade/g')
go mod tidy
go mod download

Which uses go mod graph's output to extract exactly the requirements listed in go.mod, then upgrade them with @upgrade. It's worked quite well compared to what I used to do (go get -d -t -u ./..., which upgraded a pretty difficult to reason about set of packages).

Just to show what the difference is on one of my projects:

$ diff <(go list -f '{{if not (or .Main .Indirect)}}{{.Path}}{{end}}' -m all) <(go mod graph | grep "^$(go list -f '{{.Module.Path}}' .)" | cut -d ' ' -f 2 | sed 's/@.*//g')
2a3
> github.com/DATA-DOG/go-sqlmock
3a5
> github.com/PuerkitoBio/goquery
7a10,11
> github.com/cenkalti/backoff
> github.com/containerd/continuity
8a13
> github.com/ericlagergren/decimal
17a23
> github.com/gotestyourself/gotestyourself
28a35
> github.com/opencontainers/runc
31a39
> github.com/pmylund/go-cache
36a45,46
> github.com/spf13/cobra
> github.com/spf13/viper
37a48
> github.com/volatiletech/inflect

All of which are things that my non-module-aware dependencies require themselves but are not satisfied elsewhere.

[cristian-aldea]

I would love to see this implemented!

Sometimes, when updating dependencies for a project with go get -u, my project is left in a broken state due to incompatibilities in library code, since everything was getting updated to their latest version, which wasn't always compatible.

this snippet:

go get $(go list -f '{{if not (or .Main .Indirect)}}{{.Path}}{{end}}' -m all)

has been really handy, and made the process of keeping projects up to date very easy. It would be nice for this to be a feature integrated in the CLI itself.

[josharian]

An experience report on this front.

I have a dep on tailscale.com in my code. tailscale.com has a dep on package gvisor.dev/gvisor/pkg/tcpip/buffer. However, the gvisor project has apparently eliminated or moved that package (grrrr). The tailscale.com go.mod file uses a version of gvisor in which that package exists. When I run go get -u ./... to upgrade my deps, cmd/go attempts to upgrade gvisor, which fails:

$ go get -u ./...
righthandgreen.com/www imports
	tailscale.com/tsnet imports
	tailscale.com/wgengine/netstack imports
	gvisor.dev/gvisor/pkg/tcpip/buffer: cannot find module providing package gvisor.dev/gvisor/pkg/tcpip/buffer

In order to upgrade my deps (not knowing about the other workaround in this issue), I have been using this script to do them one at a time:

$ go mod edit -json | jq ".Require" | jq "map(.Path)" | jq -r ".[]" | xargs go get -u

And then I manually copy over the known good gvisor package version from the tailscale.com go.mod. And then run go get again to get the missing go.sum entries.

Being able to easily upgrade only the direct dependencies would be a big help in this situation.