proposal: Go 2: allow 3-index slices to elide the second (length) term

[bcmills]

I propose to allow the second index of 3-index slice expressions to be elided, with the default value “min(original length, new capacity)”.

This is based on the observation that the 3-index slice form is almost always used with the new length set to either 0 or the same expression as the new capacity, and the new capacity set to either exactly the original length or value known not to exceed it (such as in numerous fixes for #34972).

The specific default is chosen for the following properties:

• Unlike defaulting to the the original length alone, it is guaranteed not to panic for the expression a[::k] for any k ≤ cap(a).
• Unlike defaulting to the new capacity alone, it does not preclude a future change from establishing the intuitive identity x[::] ≡ x.

Template answers

• Would you consider yourself a novice, intermediate, or experienced Go programmer?

Experienced.

• What other languages do you have experience with?

C, C++, Standard ML, OCaml, Rust, JavaScript, Python, Bash, π-calculus, probably a few others I've missed.

• Would this change make Go easier or harder to learn, and why?

Could go either way: 3-index slices are already relatively obscure, but this proposal arguably makes them more closely match the intuitive behavior.

• Has this idea, or one like it, been proposed before?

This form was discussed in the original design doc, but explicitly deferred for later discussion. Now that we have released 12 intervening Go versions, I believe we have enough experience to continue that discussion.

It is not clear to me whether this point of the design has been revisited since Go 1.2. (If so, the original issue #1642 has no cross-references to that discussion.)

• Who does this proposal help, and why?

Anyone who uses append and doesn't enjoy debugging aliasing bugs.

This proposal makes it easier (less verbose) to append to a slice that is never otherwise mutated, without risking overlapping mutations to the portion of the slice beyond its length.

• What is the proposed change?

Allow the second (high) index of a 3-index “full slice expression” to be elided.

Specifically, for an expression of the form a[low : high : max], if the high term is elided, it should default to min(len(a), max).

This corresponds to cases 3 and 6 in the original design doc. However, this proposal does not allow the third element to be elided: we do not need to choose between those two cases at this time.

• What would change in the language spec?

- Only the first index may be omitted; it defaults to 0.
+ The first and/or second index may be omitted.
+ The first index defaults to 0.
+ The second index defaults to either `max` or the length of the sliced operand, whichever is smaller.

• Please also describe the change informally, as in a class teaching Go.

If the middle index of a 3-index slice expression is omitted, it defaults to either the length of the original slice or the new capacity, whichever is smaller.

• Is this change backward compatible?

Yes. It is explicitly called out as a possibility in the Go 1.2 design doc, and I don't believe any change in the interim has rendered it impossible.

• Show example code before and after the change.

This example is from real code (in #38077), fixing an aliasing bug.

Unfixed:

go/src/cmd/go/internal/generate/generate.go

Line 441 in 2975b27

cmd.Env = append(cfg.OrigEnv, g.env...)

Fixed with the language as defined in Go 1.14:

	cmd.Env = append(cfg.OrigEnv[:len(cfg.OrigEnv):len(cfg.OrigEnv)], g.env...)

Under this proposal:

	cmd.Env = append(cfg.OrigEnv[::len(cfg.OrigEnv)], g.env...)

• What is the cost of this proposal? (Every language change has a cost).
  • How many tools (such as vet, gopls, gofmt, goimports, etc.) would be affected?

The compiler and static analysis tools would need to be upgraded to accept the new form.
I believe the change to the compiler would be minimal.

go/ast.SliceExpr already allows a nil Expr in the High position, so other syntax-based tools would be unaffected unless they assume SliceExpr invariants beyond what is promised in the documentation.

• What is the compile time cost?

Negligible.

• What is the run time cost?

None: an equivalent 3-index slice can already be written today.

• Can you describe a possible implementation?

Yes: assign the 3rd index to a temporary, then use that temporary (and a trivial min calculation) for both the 2nd and 3rd indices.

• Do you have a prototype? (This is not required.)

No.

• How would the language spec change?

Objection: asked and answered.

• Orthogonality: how does this change interact or overlap with existing features?

It makes 3-index slices more consistent with 2-index slices, for which the high term may already be elided.

• Is the goal of this change a performance improvement?

No.

• Does this affect error handling?

No.

• Is this about generics?

No.

[ianlancetaylor]

I think question here is: what do people intuitively expect when the second operand is omitted? I think a decent argument could be made that people intuitively expect that if the second operand is omitted that the length remains unchanged. I think it's a little harder to argue for the min expression. But maybe that is OK.

(Another possible default would be that omitting the second expression means that the length is unchanged, and if the capacity becomes smaller than the length then the expression panics at run time. The most important goal is not "omit the second expression as often as possible." I think the most important goal is "don't surprise the reader.")

[bcmills]

I think a decent argument could be made that people intuitively expect that if the second operand is omitted that the length remains unchanged.

Indeed. You can think of the “min” part as meaning: “leave the length unchanged unless it would exceed the new capacity, in which case keep it as close to the original as possible”.

The alternative is to panic if the original length exceeds the new capacity, but panicking is arguably never the expected behavior — that's why it is a panic instead of an error return or a , ok — whereas slicing down to the new capacity is the natural and expected behavior for the (*[veryLarge]T)(p)[:actualLength:actualLength] expressions in #34972.

[zigo101]

A similar proposal by defaulting the max index as the high index: #25638

[bcmills]

In a cursory search of the x repos, I found only three examples for which the second element was not identical to the third.

One such example supports the “min” behavior over “default to the new cap”:

• https://github.com/golang/crypto/blob/5d647ca1575777a812e903a7e98177174d8c295a/sha3/sha3_s390x.go#L83-L87

func (s *asmState) clone() *asmState {
	c := *s
	c.buf = c.storage[:len(s.buf):cap(s.buf)]
	return &c
}

The other two use values that are not plausible defaults:

• https://github.com/golang/crypto/blob/a49355c7e3f8fe157a85be2f77e6e269a0f89602/internal/subtle/aliasing_test.go#L30-L31

	{a[:10], a[:10:20], true, false},
	{a[:10], a[5:10:20], true, true},

• https://github.com/golang/crypto/blob/5d647ca1575777a812e903a7e98177174d8c295a/sha3/sha3_s390x.go#L97-L102

// resetBuf points buf at storage, sets the length to 0 and sets cap to be a
// multiple of the rate.
func (s *asmState) resetBuf() {
	max := (cap(s.storage) / s.rate) * s.rate
	s.buf = s.storage[:0:max]
}

---

The vast majority of usage sites don't distinguish between “min” and “new cap” at all, because they use the same expression for both terms. However, they clearly support that “old length” alone is not a broadly-useful default.

There are a few that slice down to a capacity (and equal size) obtained elsewhere:

• https://github.com/golang/perf/blob/9c9101da83161dff5e53fc89bf35ed49ea286db8/vendor/google.golang.org/grpc/transport/handler_server.go#L330-L334

			n, err := req.Body.Read(buf)
			if n > 0 {
				s.buf.put(&recvMsg{data: buf[:n:n]})
				buf = buf[n:]
			}

• https://github.com/golang/text/blob/f4905fbd45b6790792202848439271c74074bbfd/transform/transform.go#L578-L582

	// Allocate only once. Note that both dst and src escape when passed to
	// Transform.
	buf := [2 * initialBufSize]byte{}
	dst := buf[:initialBufSize:initialBufSize]
	src := buf[initialBufSize : 2*initialBufSize]

• https://github.com/golang/perf/blob/9c9101da83161dff5e53fc89bf35ed49ea286db8/vendor/github.com/mattn/go-sqlite3/callback.go#L44

	args := (*[(math.MaxInt32 - 1) / unsafe.Sizeof((*C.sqlite3_value)(nil))]*C.sqlite3_value)(unsafe.Pointer(argv))[:int(argc):int(argc)]

A few slice both the size and cap down to the original length, in which case the distinction between “new cap” and “original length” is irrelevant:

• https://github.com/golang/arch/blob/5de9028c2478e6cb4e1c1b1f4386f3f0a93e383a/arm/armasm/ext_test.go#L220-L226

// pad pads the code sequence with pops.
func pad(enc []byte) []byte {
	if len(enc) < 4 {
		enc = append(enc[:len(enc):len(enc)], zeros[:4-len(enc)]...)
	}
	return enc
}

• https://github.com/golang/crypto/blob/a49355c7e3f8fe157a85be2f77e6e269a0f89602/bcrypt/bcrypt.go#L217

	ckey := append(key[:len(key):len(key)], 0)

And many slice an arbitrarily large unsafe slice down to a specific capacity, although #19367 (comment) is arguably even better for these cases:

• https://github.com/golang/sys/blob/85ca7c5b95cdf1e557abb38a283d1e61a5959c31/windows/security_windows.go#L600-L613

// AllGroups returns a slice that can be used to iterate over the groups in g.
func (g *Tokengroups) AllGroups() []SIDAndAttributes {
	return (*[(1 << 28) - 1]SIDAndAttributes)(unsafe.Pointer(&g.Groups[0]))[:g.GroupCount:g.GroupCount]
}

• https://github.com/golang/sys/blob/85ca7c5b95cdf1e557abb38a283d1e61a5959c31/windows/registry/value.go#L111

	u := (*[1 << 29]uint16)(unsafe.Pointer(&data[0]))[: len(data)/2 : len(data)/2]

• https://github.com/golang/sys/blob/85ca7c5b95cdf1e557abb38a283d1e61a5959c31/windows/registry/value.go#L301

	buf := (*[1 << 29]byte)(unsafe.Pointer(&v[0]))[: len(v)*2 : len(v)*2]

• https://github.com/golang/crypto/blob/f7b00557c8c46a1ea4b035cae84f52028c2c0564/sha3/xor_unaligned.go#L31-L32

	n := len(buf)
	bw := (*[maxRate / 8]uint64)(unsafe.Pointer(&buf[0]))[: n/8 : n/8]

[ianlancetaylor]

To me the issue here is not what is the most useful or most commonly used default. It is: what will people intuitively assume if the length is missing?

[bcmills]

I think those two judgments are closely connected: people will tend to intuitively assume whatever behavior is most convenient to their use-case, because they are primed to think about that use-case and not other possible behaviors.

[ianlancetaylor]

I think that to some extent you may be talking about the writer of the code, but I'm trying to talk about the reader of the code.

[bcmills]

I'm talking about both: neither the reader nor the writer of the code will expect a slice expression to panic when all of the non-elided indices are valid.

They will not expect a panic in part because the corresponding 2-index slice does not panic when either index is elided, provided that the other index is valid.

And, they will not expect a panic in part because they will observe that the code does not panic when trivial tests are run. (For the “cut cap down to length” use-cases, the distinction between the old length and new cap is meaningless, and for the “cut a buffer down to a computed size” use-cases, the code would panic under even the most trivial usage if the “min” operation were not applied.)

[bradfitz]

What is the run time cost?
None: an equivalent 3-index slice can already be written today.

There's some, right? We can't always statically prove which is min so that would need to happen at runtime? But perhaps that no different than the panicindex checks today.

[ianlancetaylor]

If in most examples the second operand of the slice expression is identical to the third operand of the slice expression, then perhaps the appropriate default for a missing second operand is simply the third operand. That is slightly simpler than the suggested min expression.

A separate consideration is how easy it would be to miss the extra colon.

    // These mean different things:
    a = s[:10]
    s = s[::10]