all: merge dev.typeparams to master during Go 1.17

[mdempsky]

What

Proposal #43651 is to add generics to the Go language. The proposal currently has 1223 likes and only 89 dislikes. Demand for generics was a top requested feature in the 2016, 2017, 2018, and 2019 Go developer survey results. There's a working go2go Playground for trying out the proposal. The type parameters proposal was the subject of a keynote talk at GopherCon 2020. This is a widely demanded feature from the Go community; it's not a niche feature like persistent memory or field tracking.

This proposal then is about allowing prototype compiler development of that feature to happen on the master branch. The new language functionality would be kept behind a compiler flag until both (1) #43651 is accepted and (2) the functionality is ready for end users.

Of course, if #43651 is accepted before this proposal, then this one becomes moot. Alternatively, should #43651 be rejected after this one is accepted, then the implementation support code should be backed out of master.

Rationale

We've already begun development for #43651 on the dev.typeparams branch, but keeping it in sync with master and dev.regabi has been very tedious. Within the last 2 months of development, we've had 20+ merges between branches. We had a failed merge that required resetting the development branches (#43147). We've had compiler engineers attempt and give up merges, because the conflicts were too difficult to resolve without more familiarity/time.

This tedium will likely decrease somewhat after Go 1.16 is released and dev.regabi merges back into master, but then there will also be changes happening elsewhere in the repo to keep up with (e.g., #42637 caused a lot of trybot flakes on both dev.regabi and dev.typeparams). Also, support for generics will continue requiring more pervasive changes that involve touching the backend. There are also other accepted Go language changes slated for Go 1.17 (e.g., #395, #19367, #38248, #40481) that will overlap in frontend areas.

It would ease cmd/compile development significantly if the compiler functionality in dev.typeparams could be merged into master and developed behind a feature flag. Feature flags are how we developed binary export (ae2f54a), SSA (c0740fe), package syntax's new parser (2ff4639), indexed export (ca2f85f), new escape analysis (97c4ad4), etc. (I'm sure there are others; these are just ones I'm personally familiar with.)

Additionally, dev.typeparams introduces a new go/types-based typechecker "types2". Even if we reject #43651, I think we want to adopt this new typechecker to replace the compiler's legacy typechecker ("typecheck"). And because generic support in types2 is already conditional behind a flag, removing generics support if #43651 is rejected would be no different than when we removed the old code any of the above features replaced.

Flag

As for how to spell the compiler flag, dev.typeparams currently uses -G, but other spellings would be fine. The particular values assigned to it though are somewhat arbitrary, driven by development needs that no longer apply, so they should be revisited. I suggest -G=0 means use typecheck; -G=1 means use types2 w/o generics support; and -G=2 means use types2 w/ generics support. (I'm hopeful -G=1 can be the default for Go 1.17; then we can drop support for -G=0 for Go 1.18, which should make it easier to enable -G=2 by default. However, I'm not proposing either of these default changes at this time.)

It's been brought up that users might then use it and things could break. We currently provide users with compiler flags like -B (disable bounds checking), -wb=false (disable write barriers), and -d=disablenil (disable nil checking), so users have ample opportunities for self-harm today already.

But moreover, I think we want users trying out features before they're fully ready. It's been helpful having users report regressions on dev.regabi (#43479, #43480, #43701, #43818), and at the end of release cycles we're always begging for users to try the betas and release candidates.

If really desired, we could put it behind GOEXPERIMENT=typeparams with other features like field tracking and static lock ranking, which never went through the proposal process. But then I'd ask again for consideration of #42681.

[mdempsky]

This is a widely demanded feature from the Go community; it's not a niche feature like persistent memory or field tracking.

Just to clarify, this was not intended to deride #43810. Persistent memory was brought up as a counter-point in the Google compiler/runtime team meeting, which I understood at the time to be a hypothetical proposal. I vaguely remembered the topic of Go language support for persistent memory being discussed in the past over email, but I was not aware it had yet advanced to a proposal.

I'm not presently familiar enough with persistent memory to know whether #43810 is a good idea, and I don't mean to imply either way about it here. But I do think its current thumbs up/down ratio demonstrates a different level of broad community support than #43651, which was the point I was arguing for why I think it's justified as in the interest of the broad Go community to allow development for #43651 to continue on master behind a feature flag, when we probably wouldn't extend the same offer to #43810.

[rsc]

I don't think we can land code on master before a proposal is accepted. That goes against the spirit of the proposal process and the rules we ask others to follow for proposed changes. I think the right way to view this proposal is as simply to start working on master once #43651 is accepted, changing our previous plan to stay on dev.typeparams until Go 1.18.

My bigger concern is how to properly firewall off the generics work from the Go 1.17 work so that changes being made to help set up for generics don't cause undue instability in the Go 1.17 release.

How many code changes do you contemplate (for example in the back ends) that would not be gated by the flag?

[komuw]

That goes against the spirit of the proposal process...

English isn't my first language, but that line sounds circular to me.

No proposal can be against the spirit of the proposal process, it is a proposal.

[mvdan]

No proposal can be against the spirit of the proposal process, it is a proposal.

I think Russ is being taken slightly out of context there. I understood his comment to say: we've historically not merged code before a proposal is accepted, so we should not make an exception here. It might set a bad precedent, which is something I can understand.

That said, I think this proposal is special, given how much it's refactoring and improving the compiler and typechecker. I assume we want those merged in the future, even if generics end up being rejected. So I understand Matthew's point about the large amount of extra tedious work involved, which could be avoided.

[andybons-stripe]

Landing code on master implementing a proposal before it is accepted is at odds with what the proposal process exists for. There is no implication that the filing of this proposal goes against that. I can perhaps see how the confusion arose, so please consider this to be a clarifying point, as I was in attendance when @rsc’s response was written.

[DeedleFake]

No proposal can be against the spirit of the proposal process, it is a proposal.

That's not what he's saying. He's saying that the point of the proposal process is to have a standardized way to get new features into the language. If a proposal, in this case #43651, is given special treatment and allowed to circumvent part of that process, then it is unfair to other proposals and sets a bad precedent for later ones because people will want to know why their popular but still unaccepted proposal can't get that treatment, too.

[hherman1]

Few proposals have the scope of the generics work. I think it’s ok to make exceptions, and if people need to understand why exceptions were made it’s easy to explain.

[komuw]

If a proposal,..., is given special treatment and allowed to circumvent ...

Not all proposals will have the same scope.
For some proposals, no design document is required. For others, a design document is required.
Other proposals come with a proto-type implementation(go-modules as an example).

I think what is important is for an explanation to be given as to why a certain proposal requires an exemption to the rules. For you to obtain such an exemption, what do you do? You write a proposal. That is what Mdempsky has done.

It is up to others, then to figure out if the reasons given are enough to warrant the exemption.

[bcmills]

To my knowledge #43651 still doesn't even have a specific proposal for the changes to the language spec (see #43651 (comment)), which seems like a necessary step for any serious proposal to change the language. But perhaps that step is waiting on detail to be informed by the implementation work.

#43651 sketches out a design that requires a lot of refactoring and implementation work. Some of the refactoring work seems outside the scope of the proposal process in the first place (since it has no user-facing effect on the toolchain). On top of that, the user-facing implementation work seems like a fine application for GOEXPERIMENT, which could be used to provide test coverage.

I would much rather we accept the implementation work as a GOEXPERIMENT than prematurely accept the language change (without the actual text of the language change!) in order to unblock implementation work. We shouldn't have to pass the bill to find out what's in it.

[bcmills]

My bigger concern is how to properly firewall off the generics work from the Go 1.17 work so that changes being made to help set up for generics don't cause undue instability in the Go 1.17 release.

As the de facto primary build-cop for the project over the past couple of years, I'm honestly more concerned with baking in regressions on the branch.

We already have what is in my opinion a severe problem with regressions being introduced into the tree and dismissed as “flakes”, or accepted as likely bugs but not triaged promptly because they occur with low frequency. With regressions introduced on a separate branch, it is even easier to (intentionally or accidentally) bury regressions that happen to manifest as intermittent failures.

I would rather we deal with those regressions in the main branch when they are introduced, rather than accumulating a backlog of regressions in the branch and spending a month or more dealing with an unstable tree after the branch is merged back in.

[ainar-g]

I really can't comment on the “spirit of the proposal process” line, but isn't the goal of the proposal process making sure that the changes are well designed? Features often get designed one way only for that original design to be improved in the process of implementing it and discovering its inconsistencies and omissions. The GO111MODULE experiment is one example. I don't have the precise data, but something tells me that Go modules also have been a feature touching a lot of code, and it has been developed (and improved) on the main branch for six releases.

Regarding the possible instability in Go 1.17. Type parameters are possibly the most anticipated feature since Go has been publicly released. I think it's reasonable to assume that many people will build Go from the main branch once the type parameters work is merged. They will want to try the new features, play with them, and report bugs. The bugs some of which will show flaws in the design. Especially if there is a call for help to the community, for example through the Go Blog.