cmd/go: lazy modules: separate section for indirect imports

[rogpeppe]

go version devel go1.17-9e0facd26e Wed May 5 04:48:30 2021 +0000 linux/amd64

With the new lazy module loading introduced recently, a go.mod file will contain all indirect dependencies, not just the ones which lack go.mod files. This can make it hard to see the direct dependencies because they're obscured by all the indirect dependencies. As an example, in the first repository I tried, the number of indirect dependencies listed rose from 52 to 157, against 123 direct dependencies.

Although indirect dependencies are important, we should mostly be considering the direct dependencies when adjusting version numbering. It also makes it more likely that people resolving conflicts might get things wrong by choosing the wrong indirect versions.

As a way of avoiding this issue, it might be a good idea to format indirect dependencies in their own require section.

For example:

module m

go 1.17

require (
    github.com/foo/bar v1.2.3
)

require (
    github.com/baz v1.0.0    // indirect
)

Perhaps even consider supporting a whole-directive indirect comment rather than line-by-line:

// indirect
require (
    github.com/baz v1.0.0
)

[bcmills]

Interesting suggestion. I think I would rather keep per-line annotations, both because they're easier to spot in a very long file (which block is this screenful in again‽) and because they make erroneous edits more obvious.

[mvdan]

This would also help with code review. When one is developing a module it's easy to use go list -m to get the information as needed, but when reviewing changes, one is often looking at a plain go.mod file.

[bcmills]

That's true, although in the code-review setting it's not obvious to me that adding an indirect dependency is substantially different from adding a direct one — it's essential to audit the dependency for security either way.

And if an indirect dependency is promoted to a direct one, the diff is probably easier to review if it appears as the removal of an // indirect comment rather than the removal of an indirect line and the addition of a direct one with the same version.

[rsc]

I feel like vgo or xgo did this, or at least I considered it, but I didn't keep it (or do it) for exactly the reason @bcmills gives in the comment just above this one: something changing from // indirect to not-indirect doesn't feel like it should be zapped to an entirely different part of the file.

[mvdan]

I guess it's a tradeoff between the plain file being easier to read and understand by humans, and diffs in some cases being more concise. I personally run into the former more often, as it's not particularly often that indirect dependencies get promoted to direct dependencies. Updating dependencies happens significantly more often than adding or removing direct dependencies, at least in my experience.

[rogpeppe]

For me, the view of direct dependencies that you currently see in a go.mod file is very useful, as there you easily see the version knobs that you might wish to be tweaking directly, and the dependencies that you might be able to do something about easily.

When you've imported three modules and those are swamped in a sea of indirect dependencies, the go.mod file becomes less useful to view directly. I think that would be a shame.

FWIW I think that the transition between indirect and direct dependency is an important one and worth a bigger diff.

[peebs]

I like this idea. I personally don't think looking at the diff would be confusing. The concerns about losing your place in large go.mod files could be avoided by keeping a // indirect comment per line.

[bcmills]

There are conceptually three different categories of dependencies that a user might want to think about:

1. Dependencies that are directly imported by something in the main module.
2. Indirect-only dependencies that are upgraded above the maximum version otherwise required by transitive dependencies, including “upgraded above none”.
3. Indirect-only dependencies that are redundant with the maximum version required by transitive dependencies.

go mod tidy in a go 1.16 module explicitly lists the dependencies in (1) and (2) in one block, and omits the dependencies in (3).

go mod tidy in a go 1.17 module adds in the dependencies in (3).

So, one question is: should we continue to mix (1) and (2) together (as “express” and “implied”), or keep (1) separate and mix together (2) and (3) (as “direct” and “indirect”), or split them into three separate sections (“direct”, “upgraded indirect”, “implied indirect”)?

@jayconrod pointed out that the “direct” category may also include upgrade to modules that would otherwise be constrained by implied transitive dependencies, so at the moment we're leaning toward the “direct” / “indirect” split.

[rogpeppe]

Given that we seem to be leaning towards including an "indirect" qualifier on every line, how about including category 2 dependencies in the first section but still including the indirect qualifier comment?

That way the first section includes all the dependencies that the module directly interacts with in any way, and the manually upgraded deps are still clear.