proposal: spec: option types

[lukescott]

User story:

I'm currently writing a REST API in Go, which uses the encoding/json package. The Update
part of CRUD gets tricky if you want to do partial updates because there is no way to
tell if something was left out of the JSON.

The structure, for example, could look like this:

type Message struct {
    Name     string
    Title       string
    Message string
    Date       time.Time
}

With an update I may only want to update Title. To do this I have to change the
structure to this:

type Message struct {
    Name *string
    Title *string
    Message *string
    Date *time.Time
}

This allows each value to have 3 states: nil, empty "", or non-empty.
Unfortunately this approach isn't very desirable because the values themselves don't
live in the struct, and this can create garbage that needs to be collected.

With structs that have an optional time.Time, changing it to *time.Time doesn't solve
the problem either. You can't tell the difference between {"Date":null} and {}
(remove expiration date vs don't touch it).

A possible solution to this is by wrapping the types like this:

type Int struct {
    int
    set bool
}

And this works very well. In fact, database/sql does this.

Full example of Int: http://play.golang.org/p/GbsOucDAPZ
Example of Time: http://play.golang.org/p/OpVVOVBIwc

Unfortunately there is no standardization for wrappers like this. I can use
sql.NullInt64 and keep my data store, model, and controller layers separate. Since my
data store needs to be interchangeable, what if I'm not using SQL?

A standard package to hold these wrapper types would be desirable. And they could
include methods like MarshalJSON and UnmarshalJSON. But what happens when a custom
package needs methods such as these?

Solutions:

Option 1 - A standard package for wrapper types could be created. Either there needs to
be a way to attach methods to a type declared in one package from another, or the
encoding packages need to be aware of these types just as they are the primitives.

Pros -- If the packages are aware of these types (and don't use Marshaler/Unmarshaler
methods) only an addition pkg written in Go needs to be added.

Cons -- This only works well for primitive types. Custom types such a time.Time rely on
Marshaler/Unmarshaler methods. These method are difficult to add for custom encoders /
database packages, etc.. (packages that use reflect).

Option 2 - A type extension, which could be expressed something like:

type Message struct {
    Name ?string
    Title ?string
    Message ?string
    Date ?time.Time
}

Internally it would be the same as the wrappers and it would work with a built-int
isset() method. Otherwise the interaction would be identical to the regular types, so
you could do something like msg.Name = "Luke" instead of something like
msg.Name.Set("Luke"). This would also be true with the reflect package, with
the addition of Value.IsSet.

This should be kept distinct from pointers. unset(msg.Name) instead of msg.Name = nil.
An unset() method would set a value to it's zero value and reset the internal bool.

Pros -- Use these types like normal types. Marshal/Unmarshal method would be inherited
from the types themselves.

Cons -- The spec needs to change to allow support for this. I cannot speak for the
challenges in doing this.

[rsc]

Comment 1:

Labels changed: added release-none.

Status changed to LongTerm.

[evillemez]

Having to deal with json and multiple database backends (not just sql), the lack of true optional types has been by far the biggest pain point in adopting Go. Not having it means the API surface increases in many places at the application level to compensate.

Providing real optional types at a language level would make it much more convenient, and safe, without requiring developers to increase the API surface by repeatedly having to write custom types for the same case.

I don't know about implementation, but something closer to Option 2 above would be incredibly beneficial. I don't use Swift, but the way that language expresses the idea of optionals seems consistent, convenient and safe. You declare any type as optional, and when you access it, it must be unpacked properly, so it's clear at compile time whether or not the operation is safe. The ability to chain those operations is icing on the cake.

[leighmcculloch]

I have in the past almost exclusively used pointers to skip fields on (de)serialization because they work well with omitempty; if they are nil, the field doesn't get written, if it has value it gets written even if it's the zero value for the type pointed at. But that gets really painful for fields that are *int.

type S struct {
        F0 *int `json:"f0,omitempty"`
        F1 *int `json:"f2,omitempty"`
        ...
}

When I have a struct like the struct like the above, I can't inline define the int values because it's not possible to get a pointer to a constant, or it may not be desirable to take the pointer of where the int value currently lives and doing so may actually introduce problems. So I repeatedly end up with many extra vars as a temporary place to copy the value, or a function to turn an int value into a pointer. e.g. https://play.golang.org/p/fb-9w9_vOU

Another problem I've found is when I use pointers in this way it feels like I'm misusing them and misscommunicating intent. I usually want a value type, and just want to indicate it's not set which is different to wanting to point to a value being stored somewhere else. Because the fields are pointers to other fields it's always possible I introduce bugs when using the pattern above by accidentally pointing to a value that changes underneath me. I think something like optionals could make it easier to express that intent and prevent errors with pointers when pointers are not needed.

To counter my own argument, I could also see myself overusing optionals when a default value would be simpler, and I've seen Java code get too interesting with it's use of Optionals and hope that wouldn't become the norm in Go code.

---

As an experiment I wrote an optional package that wraps builtin and user defined types (using gotemplate to go generate) in a compile safe way and not using interface{}. The optional type wraps the type, and when the optional is empty and tagged with omitempty, it's field is omitted. (It does this by internally wrapping the value in a slice, and when the slice is empty the JSON and XML encoding packages consider it empty.) Otherwise it behaves the same as the wrapped type for JSON and XML serialization. So it behaves like the pointer, without an explicit pointer and prevents misuse and pointer errors by requiring it to be unwrapped within a conditional.

https://github.com/leighmcculloch/go-optional

[dsnet]

Related to #19412. This proposal looks like a special-case of general sum types. Based on some proposed syntax in that issue, this may take the form:

type Message struct {
    Name    nil | string
    Title   nil | string
    Message nil | string
    Date    nil | time.Time
}

However, this would not be as nice since since a type assertion would still be required to switch on the exact type.

EDIT: Some of the proposals in that thread would allow this to be nicely represented where the "non-set" state is the zero value, but provides a way to distinguish whether it was actually set or not. I support a sum-type solution that solves this well rather than this being a standalone feature.

[evillemez]

That makes sense... how would accessing those properties work in practice, via type assertions maybe? Building on your example, something like:

m := Message{Name:"Foo"}
if name, ok := m.Name.(string); ok {
  fmt.Println(name)
}
if title, ok := m.Title.(string); ok {
  fmt.Println(title)
}

In that case, m.Name would be printed, but m.Title would not.

[dsnet]

@evillemez, it really depends on the exact specification of how sum types works. The proposal in #19412 has been active and there have been various proposed specifications. Some of which work better than others in acting as an "option" type. I suggest you participate in the discussion over there.

[ianlancetaylor]

There is a lot more discussion over on #19412 for a general approach to this problem. I'm going to close this issue in favor of that one.