proposal: encoding/json: `nilasempty` to encode nil-slices as `[]`

[lukebarton]

https://go-review.googlesource.com/c/go/+/205897

This is an improvement over @pjebs PR/Proposal (which can be found here: #27589) for the following reasons:

• The previous PR had a naming issue which was unresolved. As such I spent a long time stewing over the name of the option, and ended up going with nilasempty which I think works very well to describe what it does, and even gives a clue as to why it exists
• The encoder functions are entirely responsible for the encoding of their nil versions resulting in a cleaner implementation
• There are separate tests

This is the cleanest, most straightforward, most complete change that could implement this feature in the existing encoding/json package. There are good, clean, readable tests too.

If the maintainers agree, I really think it'd be great to get this merged so that we can start benefiting from it's presence in the next suitable release! Let's put this shortcoming to bed 👍

The issue

I have an json API contract that says my response must look like this:

{
  "foo": ["some"],
  "bar": ["values", "for the consumer"], 
  "baz": []
}

(empty-array expected over null)

So naturally I create a type to help me satisfy the contract

type MyResponse struct {
  Foo: []string `json:"foo"`,
  Bar: []string `json:"bar"`,
  Baz: []string `json:"baz"`,
}

Then someone else comes along and creates a mock response for testing

myMock := MyResponse{
  Foo: []string{"blah"},
  Bar: []string{"blah"},
}

The type checker is happy with a nil slice for Baz

json.marshal(myMock)

for already understood reasons, this results in

{
  "foo": ["blah"],
  "bar": ["blah"],
  "baz": null,
}

which, of course does not satisfy the contract as intended.

In conclusion of that, the type system is not doing anything wrong, but it's not helping us satisfy contracts we have in json.

Moving forward

So what can we do? The obvious choice is a constructor to initialise MyResponse -- but Go doesn't have any way of enforcing constructors, so it has to be opt-in as far as usage goes, which is going to be forgotten and the type checker still isn't going to complain to save us from ourselves. It's not a satisfactory solution.

Digging deeper

I began thinking about where the fault lies - is it in Go's nil slices? is it in the json encoder? is it in our usage of types?

I concluded that the error does lie in the choice the json encoder makes in choosing how to encode a nil-slice and I'll do my best to explain why I think that's the case.

The author of the encoder chose to encode a nil-slice as null -- but why? The code that returns null is inside a function which knows it's encoding a slice.

go/src/encoding/json/encode.go

Lines 840 to 846 in 414c1d4

	func (se sliceEncoder) encode(e *encodeState, v reflect.Value, opts encOpts) {
	if v.IsNil() {
	e.WriteString("null")
	return
	}
	se.arrayEnc(e, v, opts)
	}

The reason that decision was made seems to be because it's nil under the hood, and nil == null.

So now I want to consider, why is the zero value of a slice nil in the first place? Correct me if I'm wrong (I'm relatively new to Go) but it's a memory allocation optimisation, meaning Go only needs to allocate memory when it gets it's first values -- which is totally smart when designing a memory-efficient language!

So, I'll say that nil-slices are just an implementation detail of Go in it's aim of being memory efficient out-of-the-box as a programming language -- It looks like a slice, it quacks like a slice, but aha! it's a nil under the hood -- but it's still a slice.

Aha!

Now let's look at the perspective of the encoder - the encoder appears to be encoding the underlying nil to null. But now we understand that the slice being nil is just an implementation detail of Go lang. The encoder should be encoding the values in the context of their type eg. []string to [], and definitely not converting the underlying zero-value-of-a-slice-nil to it's closest json equivelent of null. In wonderful irony, representing the underlying nil (which is a memory allocation optimisation for Go) as null in json actually costs more bytes over-the-wire than the empty array version []!

So how do we address this misstep? Ideally, mirroring omitempty and as the less common behaviour, nullempty would encode empty arrays as null. But to be backward compatible, something like nilasempty would be best.

Summary

In summary, nil-slices are an implementation detail of the Go language and nil-slices should be treated as plain-old slices by the json encoder - it's a slice and it's empty, and it should be represented in json as [] when specifying something like nilasempty to enable this new behaviour in a backwards compatible way.

[lukebarton]

@kortschak @slrz rather than just thumbsdowning, or confused emoji'ing could you please give some actual feedback?

This issue continues to prevent Go being a primary choice for building a JSON API.
JSON schemas that look like { foo: null | Thing[] } are horrible for consumers to work with, because they need to do null checks. Serialising nil-slices to null causes that. The only alternative is to initialise empty-slices where slices exist through extraneous constructor functions. And it's still not possible to guard against people from initialising an object of the type directly (which would result in the nil-slices). It's not a very nice developer experience at all, and thus why I would choose another language. It disappoints me that this issue hasn't been progressed when there's a perfectly good patch, complete with tests, that's well-aligned with the aproaches of the existing json/encoder.

[kortschak]

nil maps to null, and []T{} maps to []. This already just works out of the box, is simple and requires no explanation for users or addition to the API.

https://play.golang.org/p/5yJEv7hmrxf

If nil-safe deserialisation is required, then json.Unmarshaler can be implemented to take that into account.

[neilalexander]

Having been bitten by bugs caused by problems just like this in the past, I am very much in support of a nilasempty option, or some similar way of configuring such optional behaviour.

In a controlled environment, or in a context where you can specifically plan for nulls in the JSON, the default behaviour makes sense.

In an uncontrolled environment where you can’t necessarily prevent someone from passing in nil map or slices as inputs, or where interoperability with existing software is required, the only way to avoid this is to pepper the code with checks for nil maps or slices everywhere. This quickly gets verbose, it quickly gets tiresome and it is still all too easy to shoot yourself in the foot and land on a panic down the line as a result.

The solution proposed is a reasonable and pragmatic one. Being expected to write custom marshallers and unmarshallers to avoid this problem is yet another place to introduce verbosity, complexity and to, in all likeliness, shoot yourself in the foot some more.

[ToadKing]

The whole point of JSON is for exchanging data between two different systems. Having to do extra steps to get the data in a format for other systems just makes it harder to use JSON generated with Go with programs not written in Go.

A common usecase I use (and I'm sure other people use) is something like this:

var filtered []something

for _, item := range fullList {
	if isValid(item) {
		filtered = append(filtered, item)
	}
}

In this case, an empty filtered variable would be null if marshaled to JSON, when any place this is passed to is probably expecting an empty array. Since a nil slice behaves exactly like an empty slice in every other situation there shouldn't be extra steps that have to be done only for the JSON package.

Personally, if not for the 1.0 promise, I would want this new behavior to be the default. Implementing a data-exchange format with a language-specific quirk built into it is just bad design. null shouldn't show up except when a field is explicitly a pointer type.

[lukebarton]

@kortschak Your example might be just fine, if this were typical.

I think it appears to you to be less-of-an-issue because it's your example is only toy-size, scale this up to something more real world and the problem becomes more obvious:
https://play.golang.org/p/le_PN4NMw3u

Do you really want to have to initialise all those empty slices everywhere you use a T?

Now, considering this example, let's say I add J:
https://play.golang.org/p/0kQ_3L3YkQV

If I want to avoid an uninitialised J serializing to null I have to go through my entire codebase ensuring that I am initialising J as an []string{} in every single place that I am initialising a T. Not nice, especially when one gets missed. And remember here, we're not talking about just me remembering to do it, but everyone else on the team. Not gonna happen.

Constructor function are something to be forgotten about too, and they can't be enforced. Anyone can initialise a broken T{} with nil-slices that ends up serialising incorrectly -- a problem that doesn't manifest itself until runtime.

Honestly, given a type of T{ F []string }, a developer trying to build an API returning JSON would expect and bog-standard T{} to serialize to: { "F": [] } not { "F": null }. That's why this should be in the stdlib and as @ToadKing says -- if it weren't for never breaking backward compat, this behaviour should be the default.

There's a perfectly good patch sat here that would enable us to serialize in the way most developers would expect, without resorting to custom serializers or unenforcable constructors. We still have to litter our types with nilasempty property traits -- it's still not great, but it's better to have that available as part of the stdlib than all of the alternatives.

[kortschak]

Anything can be made to look bad given an appropriate strawman.

If I want to avoid an uninitialised J serializing to null I have to go through my entire codebase ensuring that I am initialising J as an []string{} in every single place that I am initialising a T. Not nice, especially when one gets missed. And remember here, we're not talking about just me remembering to do it, but everyone else on the team. Not gonna happen.

You're right, it's not going to happen. This is what construction helpers are for.

If I were forced to write something like that and were bound by a broken JSON ecosystem then I would say it this way, https://play.golang.org/p/_wbj8qAOAWz, or this way https://play.golang.org/p/noNNRLiO3TI, or this way https://play.golang.org/p/eehvczp12f6 (at a stretch).

I'd also like to address this comment:

So now I want to consider, why is the zero value of a slice nil in the first place? Correct me if I'm wrong (I'm relatively new to Go) but it's a memory allocation optimisation, meaning Go only needs to allocate memory when it gets it's first values -- which is totally smart when designing a memory-efficient language!

This is not the case. It is not merely a memory allocation optimisation, it is a reflection of a fundamental property of Go slice values, that []T{} is not the same as []T(nil). This difference is important in many situations. Just because other languages do not make this distinction does not mean that the difference is not important here.

[neilalexander]

@kortschak Your proposals are perfect illustrations of how difficult it can be to avoid this problem in the real world. You either need to be meticulous to ensure that you don’t secretly have nil slices somewhere, or you need to define custom marshalling, or you need to define custom types. None of these are good solutions.

The point here is that this behaviour feels unexpected, because it isn’t apparent to the programmer from looking at the field type alone that this should even be possible without tracing every use of that field.

If you start with an array type, then you would expect the JSON output to be an array. The fact that nil slices act like empty arrays in Go is an implementation detail that is pretty much irrelevant here. You asked for an array. The type states it should be an array. It should be an array.

[ToadKing]

The fact is that unless you explicitly check for nil, a nil slice and a slice of length 0 are functionally equivalent in Go code in any case I can think of. You can use both in range loops, call len on both, unpack with ..., and so on. There's no reason for this to appear as null in the one place where being null is detrimental for the sole purpose of the code: serializing data for exchanging between programs.

This is not the case. It is not merely a memory allocation optimisation, it is a reflection of a fundamental property of Go slice values, that []T{} is not the same as []T(nil). This difference is important in many situations. Just because other languages do not make this distinction does not mean that the difference is not important here.

I can't think of a single situation where a length 0 slice and a nil slice should behave differently. If it makes sense that they're different internally in the runtime, then that difference shouldn't be exposed in standard packages.

[kortschak]

The point here is that this behaviour feels unexpected, because it isn’t apparent to the programmer from looking at the field type alone that this should even be possible without tracing every use of that field.

This is a subjective statement dressed as fact. The behaviour that exists now is entirely unsurprising to me based on the behaviour of types and values in the Go language.

There's no reason for this to appear as null in the one place where being null is detrimental for the sole purpose of the code: serializing data for exchanging between programs.

It is perfectly reasonably to serialise the null state since that may be (and often is) important. That some programs in some languages make incorrect assumptions about the difference between null and [] should not be papered over here at the expense of clarity and complexity.

I can't think of a single situation where a length 0 slice and a nil slice should behave differently. If it makes sense that they're different internally in the runtime, then that difference shouldn't be exposed in standard packages.

It is exposed in the language, by the == nil check.

[neilalexander]

It is perfectly reasonably to serial the null state since that may (and often is) important.

Which is why nilasempty feels like a pragmatic solution - it’s optional. It doesn’t break the convention of the defaults, nor does it break anything that exists today.

That some programs in some languages make incorrect assumptions about the difference between null and [] should not be papered over here

Some languages simply don’t make these distinctions. JSON is a data interchange format. We have to expect that not everyone will have nil slices. Is it not OK to accommodate that?

at the expense of clarity and complexity.

Expecting custom marshallers to solve this problem is definitely adding extra complexity.

This is a subjective statement dressed as fact. The behaviour that exists now is entirely unsurprising to me based on the behaviour of types and values in the Go language.

Your expertise isn’t in question but rather the soul of the defaults. The problem is that this is a really easy hole to fall into and this PR is a sensible way to make it avoidable.