Proposal: improve UX for major module versions

[adg]

DEPRECATED

Please see the new proposals #40357 and #40323 that supersede this one.

---

Proposal: improve UX for major module versions

Peter Bourgon (@peterbourgon), Andrew Gerrand (@adg)

Problem statement

When a user wants to use a module, it is the v0/v1 version of that module which is most prominent, as it is selected by the base repository path: github.com/user/repo is in effect a constraint to v0.x.y/v1.x.y.

To use v2 or above, Semantic Import Versioning requires that the major version number is a suffix of the module path: github.com/user/repo/v2 (constrained to v2.x.y), github.com/user/repo/v3 (constrained to v3.x.y), and so on.

It’s easy for module consumers to default to v0/v1, even if that version is obsoleted by a more recent major version. Module consumers may even be totally unaware of later major versions.

Discoverability is a key issue. The mechanisms for module authors to advertise recent major versions are inconsistent, and can be low-visibility (documentation? README.md?) or highly disruptive (printing deprecation warnings in init, broken builds to force an investigation).

Abstract

We propose two improvements: one targeted at module consumers, and the other at producers.

For consumers, we propose a mechanism that notifies users of the latest major version of a module dependency when that dependency is first added to a project.

For producers, we propose adding a deprecated directive to go.mod files to signify the end-of-life of a major version.

These are just preliminary ideas which we hope to refine and improve in response to feedback gathered here.

Proposal 1: Notification of latest major version

We propose notifying users of new major versions when:

• They first add a requirement for an old major version of a module
• They update a requirement for an old major version of a module

There are a few ways users add requirements to their modules:

• By running go get github.com/user/repo
• By adding a require github.com/user/repo line to their go.mod file manually
• By adding an import line to a Go source file, and having goimports or gopls editor integration add the import line on save

For the latter two, the module isn't fetched until the go command is invoked within the module.

There are a few ways users update requirements:

• By running go get [-u] github.com/user/repo
• By running go get -u ./... from the module root
• Or to find out about available updates, they may run go list -m -u all

In each of these cases, the go tool plays a key role, and so we propose to make the go tool print a note if a requirement is being added when there is a more recent major version of the module available.

Examples

Consider a user fetching peterbourgon/ff with go get. We propose adding a notification to the output, alerting the user to a new major version:

$ go get github.com/peterbourgon/ff
go: finding github.com/pelletier/go-toml v1.6.0
go: finding gopkg.in/yaml.v2 v2.2.4
go: finding github.com/davecgh/go-spew v1.1.1
go: downloading github.com/peterbourgon/ff v1.7.0
go: extracting github.com/peterbourgon/ff v1.7.0
go: note: more recent major versions of github.com/peterbourgon/ff are available     👈
go: note: to install the most recent one, run `go get github.com/peterbourgon/ff/v3` 👈

Consider a user listing all of the most recent versions of their dependencies. We propose adding the latest major version alongside any new minor or patch versions:

$ go list -m -u all
example.com/my-module
github.com/BurntSushi/toml v0.3.1
github.com/mitchellh/go-wordwrap v1.0.0
github.com/peterbourgon/ff v1.6.0 [v1.7.0] <v3.0.1>                        👈 ONE OF
github.com/peterbourgon/ff v1.6.0 [v1.7.0] <github.com/peterbourgon/ff/v3> 👈 THESE
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7 [v0.0.0-20191204190536-9bdfabe68543]
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 [v1.0.0-20200227125254-8fa46927fb4f]
gopkg.in/yaml.v2 v2.2.2 [v2.2.8]

If a new requirement is added to a go.mod file manually, the go tool would print the notification when it first fetches the new module, as part of a go build, go test, etc. run.

Proposal 2: A new deprecated directive

Producer-side deprecation feature: add a deprecated line, which will cause the module to fail to fetch at that version, printing an error with the most recent non-deprecated version of the same major version.

module github.com/user/repo

deprecated

require (...)

If the go tool were asked to fetch this deprecated module, it would fail:

error: module github.com/user/repo@v1.7.1 is deprecated

If the Proposal 1 were adopted, the error message could be more useful:

error: module github.com/user/repo@v1.7.1 is deprecated
error: try the latest major version: go get github.com/user/repo/v2

The deprecated directive may optionally include a successor module identifier. If specified, the error printed when fetching this module version would also include a reference to the successor module. This can be used to point to the next major version:

deprecated github.com/user/repo/v2

Or to a new import path altogether:

deprecated gitlab.com/newuser/newrepo

If the go tool were asked to fetch this deprecated module, it would fail with a more explicit suggestion to the user:

error: module github.com/user/repo@v1.7.1 is deprecated
error: try its successor: go get github.com/user/repo/v2

If the successor module is also marked as deprecated and includes a successor module, the go tool might follow those links, and print the final non-deprecated module in the error message.

When package producers decide that a major version is deprecated, the intent is for them to create a new minor or patch release of that major version with the deprecated directive in the go.mod. This version will only be selected by the go tool when the corresponding major version is first added to a project, or if someone tries to update from an earlier version of that major version. In both cases, the go tool fails to fetch the module and provides a useful and actionable message: the user is instructed to pick a non-deprecated version.

Users can always use earlier versions of that major version, and MVS should ensure that the deprecated version will not be selected. If module A has a requirement for B at v1.7.0, and B v1.7.1 is later tagged as deprecated, A can continue to use B at v1.7.0, and the maintainer of A will only become aware of the deprecation when they try to update B.

It should not be possible for the go tool to mechanically add a deprecated version to a go.mod file.

A major version may be "un-deprecated" by publishing a subsequent minor or patch version without a deprecated directive in its go.mod. The specific deprecated version remains unusable, but the earlier and later versions still work normally.

Examples

A producer deprecating v1 of their module:

• Producer tags github.com/user/repo v1.0.0, v1.1.0, ... up to v1.7.0
• Producer tags github.com/user/repo/v2 v2.0.0, and soon afterward v2.0.1
• Producer adds a deprecated or deprecated github.com/user/repo/v2 line to the go.mod of the v1 branch and tags it v1.7.1

A consumer using a module that is then deprecated:

• Consumer has a module requirement require github.com/user/repo v1.7.0
• Producer tags the deprecated v1.7.1 version
• Consumer continues to use v1.7.0 without issue
• If the user tries to upgrade the dependency, the deprecation prevents it
• The user can keep their existing v1.7.0 or migrate to the new v2.x.y

A consumer tries to fetch a module at a deprecated major version:

• Consumer runs go get github.com/user/repo as a new dependency
• go get selects the highest version in the v0 or v1 trees, which is v1.7.1
• go get prints an error message:

error: module github.com/user/repo@v1.7.1 is deprecated
error: try the latest major version: go get github.com/user/repo/v2

• Consumer runs the suggested command, installing v2 of the module

Composition

One of Go’s strengths is the orthogonality of its features. We believe the proposed features compose nicely and strengthen each other.

Taken separately, the proposals can stand on their own: P1 provides package consumers with useful information without direct action from package producers; P2 allows package producers to give their consumers more specific guidance on an opt-in basis.

Taken together, the proposals enrich each other: P1 improves the error messages that accompany P2; P2 funnels users into the upgrade paths created by P1.

Integrations

pkg.go.dev

The Go package discovery website at pkg.go.dev shows modules and their versions. However, it obscures successive major versions when they exist, apparently treating major module versions as completely distinct. For example, the landing page for peterbourgon/ff shows v1.7.0 with a "Latest" bubble beside it. The versions tab does list other major versions, but under the heading "Other modules containing this package", which is confusing.

Instead, pkg.go.dev could feature a prominent indicator on the landing page for a v0/v1 module that there are two successive major versions (v2 and v3), to funnel the user toward the latter.

Editor integration (gopls, goimports)

Go text editor integrations typically include a feature that automatically adds import statements to source files based on the mentioned identifiers. Because of Semantic Import Versioning, this also gives those tools the responsibility of choosing the major version of the imported module. In the case where there is no suitable existing requirement in the project’s go.mod file, these editor integrations could alert the user to the availability of newer major module versions. How this works is outside the scope of this proposal.

Appendix

• Related proposals:
  • retract directive
  • go.dev: don't imply that vN.x.y is the latest version when vN+1 exists
  • go/list adds another flag to support showing major upgrade available for module

[mvdan]

Instead, pkg.go.dev could feature a prominent indicator on the landing page for a v0/v1 module that there are two successive major versions (v2 and v3), to funnel the user toward the latter.

I think this feedback is good, and it has been given before: #37765

Assuming that pkg.go.dev will become more prominent, and eventually replace godoc.org, I wonder if this change would be enough to nudge users towards the latest major version of a module.

I get that it would not be as comprehensive as the other two changes proposed here, but it might be best to attempt fixing this UX problem in incremental steps, only considering the more aggressive/intrusive changes when the smaller steps prove to not be enough.

[adg]

@mvdan thanks for the pointer to that proposal; missed that in our review of related issues. Added to the appendix.

[kokes]

Regarding the second subproposal: What I'm wondering is whether the hard error that disallows installing of deprecated trees is future proof.

1. There is a tutorial on using library@v1. This library is at some point deprecated in favour of v2, but this is a breaking change, the API is different and the tutorial may no longer apply. When the user reads the tutorial and tries installing library@v1, it gives them "don't install this, it's deprecated, try v2 instead" - which might confuse them as v2 is not really usable due to the breaking change, but they cannot install v1 at all - unless they dig out the last non-deprecated v1.x.y version (from where? pkg.go.dev? do they know it can be done?). I don't have a clear solution in mind, but perhaps some override or more verbose errors might be helpful here.

2. Say I install v1.0.0 and it has a security issue, so v1.0.1 is released, fixing it (I haven't noticed, I'm still on v1.0.0). Even later on, v1.0.2 is released with a deprecated tag, because there's a v2 now. At this point, I have v1.0.0 locked in my go.(mod|sum), but there is no way for me to automatically update to a safe version with the same API (I can't use v2).

Both of these cases could use a "give me the last non-deprecated version within this major version" - because erring out does not help (in installing/in fixing a security issue) and suggestion to use a new major version doesn't either, because of the potential API changes.

Also this is sort of breaking the semver contract of "patch versions don't mess things up" - since the newest patch version cannot be installed at all - this would break tools that automate dependency security updates (think dependabot, snyk) - when a new patch version gets released (with just a deprecation tag), the tool that tries to suggest an update fails in a weird way. Maybe I'm misunderstanding what happens when go get -u pkg gets run.

I hope I understood the flows correctly. In any case, thanks for this proposal, I love UX improvements.

[adg]

@kokes one thing we discussed, but didn't include in this proposal, was including in the deprecation error message a mention of the previous minor/patch version in addition to the pointer to the next available major version. Something like:

error: module github.com/user/repo@v1.7.1 is deprecated
error: try the latest major version: go get github.com/user/repo/v2
error: or the previous patch version: go get github.com/user/repo@v1.7.0

(This is just off the top of my head - such a message should be finessed to emphasise the major version, while still offering a path to the previous working version.) I think that would address the cases you describe.

With regard to external tools, they'd need to understand deprecation as much as it affects their operation, the same way they need to understand various other aspects of Go modules.

[Merovius]

To suggest an alternative (haven't thought it fully through, though - you might have considered this and rejected it for reasons I can't think of right now):

• If a tool initially is asked to add example.com/foo (without any version-signifier) to go.mod, it automatically adds the latest major version instead (e.g. go get example.com/foo might add require "example.com/foo/v3" to go.mod).
• If an explicit version-signifier is given, the latest version in that major line is used, even if it is not the latest (e.g. go get example.com/foo/v2 would add require "example.com/foo/v2 to go.mod, even though v3 also exists).
• Even though v0/v1 as version-signifiers are normally omitted, they can also be used for this purpose.

I think this would solve the same issues, in that users doing the obvious thing without checking which versions exist would still default to the latest. But in this more common case no extra step of mentally parsing the output and re-entering/changing is needed. The less common case (for some reason I specifically want to use an older version) is still allowed.

It should not be possible for the go tool to mechanically add a deprecated version to a go.mod file.

I'm not super against this, but I would prefer if a satisfying solution can be found that doesn't include this. I would just really like to avoid having to manually touch go.mod if at all possible. And for better or for worse, there can be reasons to specifically use a deprecated version, even in a new project. For example, major versions might be using different wire- or disk-encodings and I'm bound by existing services not having migrated yet. That reason IMO also prohibits any hard failures when using/upgrading to deprecated versions in general: I might want to deprecate v1 (to make clear new users shouldn't use it) but still maintain it for users that do need it.

[peterbourgon]

@Merovius

If a tool initially is asked to add example.com/foo (without any version-signifier) to go.mod, it automatically adds the latest major version instead

SIV dictates that example.com/foo is semantically equivalent to the larger of extant example.com/foo/v{01}. This might be fudge-able, but folks still have unqualified import statements in their code that must continue to work. I think for these reasons this is a non-starter. (Maybe other reasons, too.)

I would just really like to avoid having to manually touch go.mod if at all possible.

I don't think any use case described in these proposals requires manually editing go.mod?

And for better or for worse, there can be reasons to specifically use a deprecated version, even in a new project.

Deprecation would apply only to specific, fully-qualified versions. The behavior described by the proposal as "deprecating an entire major version tree" is emergent from the go tool's selection of the highest qualifying specific version of a major version tree upon first import by a consumer.

Concretely, if you have (say) v1.7.1 which is not deprecated, you would need to tag (say) v1.7.2 as deprecated, you couldn't retroactively apply the deprecated bit to v1.7.1. Consumers currently using v1.7.1 would continue to function without impact; new consumers could explicitly request v1.7.1 if they really needed to use the v1 major version tree.

Which is all to say: it's still perfectly possible to use a previous specific version of a module, even if the most recent version in the major version tree has been marked as deprecated. (It's also possible to un-deprecate a major version tree, by tagging a new specific version with the deprecated directive removed from the go.mod.)

[earthboundkid]

If a tool initially is asked to add example.com/foo (without any version-signifier) to go.mod, it automatically adds the latest major version instead

I often work by just typing an import path into a file and then later running go mod tidy. In that scenario, I don't know when there would be a good time for the tool to move me automatically to v2+, as opposed to just suggesting it when I run go mod tidy or as feedback to gopls. If I actually did want to stay on v1, I would not like the tool to move automatically to a newer version behind my back.

[Merovius]

@peterbourgon

SIV dictates that example.com/foo is semantically equivalent to the larger of extant example.com/foo/v{01}. This might be fudge-able, but folks still have unqualified import statements in their code that must continue to work.

Not to nitpick, but by name, it dictates that for imports. But I'm not talking about imports, I'm talking about arguments passed to go get as "I would like to use this thing". That's at least my workflow, when I want to include a new dependency: I use go get to download the module and add it to go.mod, then I add an import via goimports when using packages. I'm not suggesting that an import-statement of example.com/foo could trigger the addition of example.com/foo/v2 to go.mod.

I agree that the nomenclature isn't entirely precise and might very well contradict the possibility of this, if taken literally. Personally, I don't really care about nomenclature though, but mostly whether the semantics make sense :) And I don't see where what I imagine breaks existing users.

To clarify: Say, I'd write a tool called "add-go-module", which, when given the name of a module (with or without version-specifier respectively) behaves as I described to insert a require directive into go.mod, would you consider that tool broken (or breaking code)? If yes, how? If not, what specifically would break if that tool is called go get instead?

I don't think any use case described in these proposals requires manually editing go.mod?

I don't understand this. The proposal seems to clearly state that the go tool should be prohibited from mechanically adding a deprecated version. So ISTM that if I want to add a deprecated version, I would have to add that manually.

Consumers currently using v1.7.1 would continue to function without impact; new consumers could explicitly request v1.7.1 if they really needed to use the v1 major version tree.

But not v1.7.2, correct? Otherwise I don't understand this sentence:

If the go tool were asked to fetch this deprecated module, it would fail with a more explicit suggestion to the user:

This seems to strongly imply that (part of?) the build fails if I want to use a deprecated version of a module? At least the first time I add it?

The use-case I was talking about is specifically to maintain a v1 branch for users who have not yet been able to migrate to v2 - including realeasing bugfixes and the like on that branch (say, as v1.7.3 etc), while still deprecating v1, so new users don't use it unconsciously. I understand if that use-case is not something we might want to jump through extra hoops to make possible. But I do feel we shouldn't add new code to specifically make it impossible, unless there's a good reason.

[peterbourgon]

@Merovius

The proposal seems to clearly state that the go tool should be prohibited from mechanically adding a deprecated version. So ISTM that if I want to add a deprecated version, I would have to add that manually.

Consumers should never be allowed to add explicitly deprecated versions. The go tool should refuse to add them to the go.mod, and if they are manually placed in the go.mod, the go tool should refuse to compile them.

The use-case I was talking about is specifically to maintain a v1 branch for users who have not yet been able to migrate to v2 - including realeasing bugfixes and the like on that branch (say, as v1.7.3 etc), while still deprecating v1, so new users don't use it unconsciously.

Ah. This use case is not a match for the semantics of deprecated as we have laid them out. According to this proposal, if you are actively maintaining a v1 branch with bugfixes, it is necessarily not deprecated.

[sylr]

The use-case I was talking about is specifically to maintain a v1 branch for users who have not yet been able to migrate to v2 - including realeasing bugfixes and the like on that branch (say, as v1.7.3 etc), while still deprecating v1, so new users don't use it unconsciously.

You want to maintain a deprecated branch ? that seems paradoxical but if you really want to then:

v1.7.0 (not deprecated) -> v1.7.1 (deprecated) -> v1.7.2 (patch release not deprecated) -> v1.7.3 (deprecated).

each time you patch your deprecated branch you release a tag which is not deprecated then you release a deprecated one just after.

[fatih]

When package producers decide that a major version is deprecated, the intent is for them to create a new minor or patch release of that major version with the deprecated directive in the go.mod.

Is this intent a SHOULD or MUST? Can a producer deprecate a package without bumping the patch or minor version?

[peterbourgon]

@fatih

Can a producer deprecate a package without bumping the patch or minor version?

No, deprecation is defined as the go.mod including the deprecated directive, and that change, like any other semantic change to a module, can't be applied to existing versions, only new ones.