cmd/go: add a flag to ignore build constraints when listing packages

[mvdan]

I realise that the point of go list is to list packages, and I realise that one must generally obey build constraints when loading packages. For example, it makes no sense to try to type-check all the Go files in a package while ignoring build constraints, because there will likely be duplicate definition errors if declarations are split by GOOS or GOARCH.

Having said that, it can be very useful to merely list or traverse a set of packages in a way that build constraints are completely ignored. The most common use case is: what packages are potentially imported by the current package, directly or indirectly, across all build configurations?

This question is valid, for example, if one wants to copy a package and all of its transitive package dependencies. go list -deps <package> does not work in general, because if I run the command on Linux, I would not be including imports which are only used on Windows.

The closest thing we have right now is go mod vendor, which copies all transitively imported packages by all the packages in the current module into the vendor folder, across all build constraints. So cmd/go already has the machinery to traverse a package dependency graph while ignoring build constraints, it seems.

--

Here ends the problem statement, and begins my initial suggestion for a solution: a go list flag to ignore build constraints. Here are some example use cases:

• List all the packages potentially depended on by a given package: go list -deps -newflag <package>

• List all the packages under the current directory tree, not just for the current platform: go list -newflag ./...

I'm not sure what this new flag could be called. Perhaps -nobuild or -anybuild.

The flag would also restrict what go list can do. For example, using -newflag along with -export or -compiled would be an error, because it does not make sense to load/build packages when we're ignoring build constraints. Similarly, -newflag -json would always omit some fields like IgnoredGoFiles, since they make sense only when following build constraints.

This idea has been discussed briefly before, but as far as I know no problem statement or solution has been raised as an issue yet. I'm not using a proposal title and label; as far as I know that's only necessary if we need the proposal review committee to intervene.

cc @bcmills @jayconrod @matloob @rsc for cmd/go
cc @ianthehat @dominikh @myitcv @rogpeppe @kardianos for golang-tools and some previous issues

[mvdan]

The main counter-argument I see to this flag solution is that it could make go list harder to understand. Right now, it always obeys build constraints when looking at packages, and it only has a second mode: -m for instead working with modules.

However, I think this third mode would be fairly niche, and we could explain it rather simply: list packages while ignoring build constraints, hence not loading packages as if they were to be built. This helps us limit go list -newflag to only support basic operations.

[rogpeppe]

I support the idea here. I've often wanted to enumerate dependencies across all build configurations and it's awkward to do currently (I've built custom solutions to do it in the past).

As for the name of the flag, how about -all-builds ?

[mvdan]

The only reason I tried to avoid "all builds" as a flag name is because, to me, that implies "load and possibly build all build constraint combinations". As if go list -export -all-builds would then build an export file for each GOOS/GOARCH platform, for example.

[kardianos]

For govendor, years ago, I had to re-build go list functionality because of this very issue, but that was specific for vendoring that as you note go mod handles correctly. Initially go code completion tools also (unfortunately) followed build constraints, but that seems to have been fixed now in recent gopls. So I agree there is strong data points support this need in general. I cannot comment on the addition to go list itself. But I could easily believe this would be useful to have.

[mvdan]

I hadn't thought about gopls. Perhaps they have some code that could be eventually replaced with this new flag. cc @stamblerre

[jayconrod]

This would be useful. We already have some of the code for this in cmd/go for module commands.

In go mod tidy, the go command loads packages in all and their tests, ignoring build constraints with one exception: the tag ignore is still considered false. That's useful for examples and small one-file programs that are in the same directory as another package. If there's a flag to ignore build constraints, I think it should have the same semantics.

As for the CLI, maybe overload -tags? Something like -tags=all or -tags=*?

This should be restricted to go list, too, and I agree about restricting -export and -compiled.

[mvdan]

the tag ignore is still considered false. [...] If there's a flag to ignore build constraints, I think it should have the same semantics.

That would be fine for my use case. The packages I want to list will eventually be loaded in some build configuration, so the ignore tag will never really play a part in that. I guess that, if someone has a very niche use case where they want all files including those with +build ignore, they could use go list -f {{.Dir}} -newflag and then grab/copy the entire directory.

As for the CLI, maybe overload -tags?

Overloading -tags could be a nice idea. I only worry that it's a generic build flag, so what would go build -tags=all do? Do we define that go list's -tags flag is actually different than all the other occurrences of the same flag?

[mvdan]

they could use go list -f {{.Dir}} -newflag and then grab/copy the entire directory.

To clarify in case anyone wonders - this kind of workaround about copying entire directories doesn't help solve the issue in this thread. For example, think packages which contain zero files matching the current build constraints, which wouldn't show up in the go list output at all.

[bcmills]

We will need a bit of care with import cycles, though. When ignoring build tags, it is possible to end up with import cycles that are otherwise rejected.
(See https://github.com/golang/go/blob/master/src/cmd/go/testdata/script/mod_tagged_import_cycle.txt.)

[ianthehat]

It transforms the ouptut of go list from a DAG to a true graph, and there are real world cases of this. It would never be feasible to have a version of the go/packages API that sat on top of this for instance (which if it was just a special tag would end up happening)
I do think that it would be a valuable feature, but it really does need to be called out as a special case that you have to hold carefully, because its the kind of thing that is easy to use in a way that works on your testing and dies in the real world in hard to comprehend ways.

[bcmills]

@ianthehat, FWIW I suspect that the existing -e flag also transforms the output from a DAG to a graph, or at least would if it worked as often as we want it to. (Cycles cause an error, but -e suppresses errors.) So tools built on top of go list -e probably already need to know how to handle cycles, and I'm not sure how many tools build on go list today don't want -e.