extended backwards compatibility for Go

[rsc]

Go's emphasis on backwards compatibility is one of its key strengths.
There are, however, times when we cannot maintain strict compatibility,
such as when changing sort algorithms or fixing clear bugs,
when existing code depends on the old algorithm or the buggy behavior.
This proposal aims to address many such situations by keeping older Go programs
executing the same way even when built with newer Go distributions.

See my talk on this topic at GopherCon for background.

See the design document for details.

[gopherbot]

Change https://go.dev/cl/453605 mentions this issue: cmd/go: set default GODEBUG for main packages

[gopherbot]

Change https://go.dev/cl/453604 mentions this issue: cmd/go: parse and report directives in file headers

[gopherbot]

Change https://go.dev/cl/453603 mentions this issue: go/build: parse and report directives in file headers

[rsc]

Design doc at https://go.dev/design/56986-godebug.

[Merovius]

An unrecognized //go:debug setting is a build error. It is also an error

Syntax error: Unterminated sentence (I assume).

[rittneje]

It would also be nice if runtime/debug.BuildInfo reported all the //go:debug directives that got compiled in.

[mateusz834]

@rittneje 
Note that there is still a possibility that the compiled //go:debug lines might change at runtime (GODEBUG env vars). 

@rsc 
We should probably also assume that someone in the wild is also parsing the GODEBUG, to detect additional capabilities (like sha1 in certs). Maybe we could add an interface to access them:

1. compiled ones like proposed by @rittneje (through BuildInfo) and
2. merge of compiled and GODEBUG env var (somewhere inside runtime package)

Or (And?) the runtime might re-set the GODEBUG env variable to the merged one.

[rsc]

@rittneje, great idea, thanks, added to the proposal and the implementation sketch as a new Setting with key "DefaultGODEBUG".

@mateusz834 People in the wild should really not be parsing GODEBUG. But if they really want to, they can now use the BuildInfo setting to find the default and then consult os.Getenv("GODEBUG") for the overrides. Changing the environment variable to hold the entire unified setting is a mistake because it would affect subprocesses.

@Merovius, fixed, thanks.

[liggitt]

Thanks, the design lgtm. I just had a couple thoughts as I read through it:

Set the default GODEBUG settings based on the go line the main module's go.mod, so that updating to a new Go toolchain with an unmodified go.mod mimics the older release.

Can the design add details about what this will do when building a main module whose go.mod references a go version prior to this compatibility behavior? For example, if this capability arrives in go1.21, when building a main module whose go.mod contains go 1.17, would go1.21 attempt to default to 1.17- or 1.20-compatible godebug settings?

An unrecognized //go:debug setting is a build error.

This means that when eventually-removed godebug options are dropped (after 2+ years / 4+ releases), a program explicitly reverting a runtime behavior with this option will no longer build with new go versions. This seems reasonable to me (even preferable compared to silently ignoring an explicit go:debug directive) but should be explicitly documented:

• to clarify to a user toggling this on that this represents a future incompatibility for them, and the limits of the compatibility guarantee for programs using //go:debug
• to encourage go maintainers to keep a high bar for making breaking changes even with this formalized extended compatibility approach

[rsc]

@liggitt, if this lands in Go 1.21, then I think probably we would make any go line prior to Go 1.20 provide Go 1.20 semantics, so that updating from a Go 1.20 toolchain to a Go 1.21 toolchain does not downgrade behaviors. The current sketch puts the baseline at Go 1.19, in part just to have a GODEBUG (randautoseed) to infer. A case could be made for Go 1.19 for people updating from Go 1.19 to Go 1.21, but I think the 1.20 -> 1.21 transitions should probably be given priority.

I agree with your comments about the unrecognized //go:debug lines. It is worth pointing out that these can only appear in a main package, so at least there is no chance of dependencies causing problems.

[davecb]

Has there been a discussion of the extent to which person wishing to defeat security could use a GODEBUG environment variable to, for example, use SHA1 certificates?