The fate of Nullable

[nalimilan]

With the perspective of representing missing data as Union{T, Null} in Julia 0.7, we should decide what will happen to Nullable. I think the consensus is that being container-like, Nullable is appropriate to represent "the software engineer's null", as opposed to "the data analyst's null", a.k.a. missing values. In other words, Nullable offers three properties which Union{T, Null} does not:

1. It does not automatically propagate missingness.
2. It requires explicit handling of the possibility that a value may be null, even when it isn't (contrary to Union{T, Null}, where code may work when a value is of type T but not when it is of type Null, which might not have been properly anticipated/tested).
3. It allows distinguishing Nullable{Nullable{T}} from Nullable{T} (contrary to Union{Union{T, Null}, Null} == Union{T, Null}), which is useful when you need to make the difference between "no value" and "null value". Such situations arise e.g. when doing a dictionary lookup (tryget, cf. Get dict value as a nullable #13055 and Add get(coll, key) for Associative returning Nullable #18211); when parsing a string via tryparse to a value which could either be of type T, null, or invalid; or when wrapping a value which could either be of type T or null in a Nullable before returning it from a function.

The two first features are the ones which turned out to be annoying when working with missing data, but which can provide additional safety for general programming. A detailed discussion of the advantages and drawbacks of these approaches can be found in the Nullable Julep.

Given that, several paths can be taken for Nullable in Julia 0.7:

1. Make Nullable{T} a (deprecated) type alias for Union{T, Null}. This would have the advantage that Julia would have a single concept of null/missing values, but without the advantages of the three points above. Checks that code is correctly prepared to handle null values could still be done by a linter.
2. Make Nullable{T} a (deprecated) type alias for Union{Some{T}, Null}, with Some{T} a wrapper around a value of type T which would behave essentially like Nullable{T} currently. Applying a function on the value would require using f.(x), broadcast or pattern matching, so that missingness would never propagate without explicitly asking for it. The advantages would be those of the three points above, at the cost of two different representations of missingness (or almost two, since the Null type would be used in both cases).
3. Deprecate Nullable in Base and move it as-is to a package. A possible variant would be to rename it to Option in order to prevent confusion with Null and be more consistent with other languages like Scala, Rust or Swift (IIRC, Nullable was originally called Option and lived in the Options.jl package). The main advantage of this approach is to have a single representation of null values in Base, in particular to avoid setting the design of Nullable in stone in 1.0. The main issue is that no code would be able to use Nullable in Base, which implies in particular changing tryparse to return Union{T, Null}, and that no correct tryget method could be implemented for dicts (Get dict value as a nullable #13055). OTOH this could help increasing consistency with e.g. match, which returns Union{T, Void}, and uses of Nullable are not so widespread in Base.

EDIT: added mention of point 3. in the first series of bullets.

[ararslan]

In my opinion, we should do the following:

julia> Nullable{T} where T
WARNING: Nullable{T} is deprecated. Use T? for missing data or Maybe{T} for
a container similar to Nullable{T} in Julia versions 0.6 and lower.
Stacktrace:
 ...
while loading ...
Union{T, Null} where T

The current uses of Nullable in user and package code are likely to represent missing data, in which case the Union is what they'll actually want.

There's something to be said for having both representations in Base, as they serve fundamentally different purposes. The container-based approach is ideal for things like Keno's suggested revision to the iteration protocol, where a Union with a propagating null meaning "iteration is done" really isn't what you want, as the iterable itself may contain null values. It's also useful for cases such as tryparse(Int?, "null"): the Union approach would make that case unclear as to whether the parsing was successful.

[bkamins]

Regarding tryparse(Int?, "null"). Why would it be a valid code?
I understand that only tryparse(Int, "null") would be valid, and it would return null (as for any other invalid string) as in this case parsing would fail.

[ararslan]

Say you're parsing a text file, and you know that it contains data that is either an integer or null. Thus you want to parse something as Union{Int, Null}, i.e. Int?, since null is not a valid Int. So now if you split on the delimiter and do tryparse(Int?, x) on a value, you'll want to be able to distinguish between having parsed a null value and having gotten something you didn't expect. For example, we'd have tryparse(Int?, "george") == tryparse(Int?, "null") == null, which would be wrong.

[bkamins]

The use case you give is typical, but I am not sure if this should be handled by tryparse without any additional arguments. In the text file missing value could be represented differently (and most probably it would not be "null" but typically either "NA" or "").
However, I agree that I would like tryparse(Int, "any invalid integer string including null") return Maybe{Int}() and not null to distinguish "valid missing data" from "parse error". And when we would get parse error we can check if the string represents missing data.

Of course tryparse (and parse) could be extended to handle something like tryparse(Int?, "some string to parse", "string representing NA").

My feeling is that in general the discussion about leaving or removing Nullable (probably renamed) should include exceptions. This can be seen for instance in comparison between parse and tryparse. Ideally the design of Base should be consistent when exception are thrown, when Nullable{T} is produced and when Union{T, Null} is returned.

[bramtayl]

Am I correct in reading that the optimizations for unions (and Arrays of unions) will only be possible for isbits types? In which case Nullable would still be useful for more complex types.

[vtjnash]

No, Nullable{T} is already inefficient for non-isbits (where Union{T, Null} is already the more optimal approach), so making Union{T, Null} efficient for isbits is expected to now cover all cases optimally using one representation.

[malmaud]

Why does Nullable need to be formally deprecated? Can't the data libraries just update to use unions and people happily using Nullable as a Maybe type can continue to do so?

[ararslan]

The problem with keeping the Nullable name is that it's confusingly similar to Null, and we'd need to deprecate it in order to rename it.

[tkelman]

For the use cases where Nullable has been used as a result-or-flag that something went wrong (roughly as a deferred exception), I'm increasingly finding that it isn't only useful to know that something went wrong, but also a bit of detail about what specifically went wrong. So for the optional meaning we may get more mileage out of a result-or-error-code type than the current result-or-null type (assuming API's were migrated to start using it uniformly).

[martinholters]

Such a result-or-error-code type should probably be a sum type, so should we reconsider adding a facility for general sum types to Base and make the result-or-error-code type a special case of it?

(Sum types are not Unions, ref https://discourse.julialang.org/t/sum-types-in-julia/2795/13. Assume tryparse(T, ...) returned a ResultOrError{T} where T == Union{T, Error} where T. Now if you wanted to parse logged errors with a tryparse(Error, ...), you couldn't distinguished successfully parsed errors from parse errors. What one wants here would be an Either{T, Error} where T which holds a tag whether it contents are from the left or right. So Either{Error, Error} would hold just that extra bit of information needed in above example.)

[bkamins]

@tkelman This is exactly why I am asking Maybe vs throw an exception (or some other type like Either in Haskell).

See e.g. https://www.schoolofhaskell.com/school/starting-with-haskell/basics-of-haskell/10_Error_Handling

There are two things to consider in my opinion:

1. Performance of try-catch vs Nullable, as I have never done such a comparison?
2. Nullable forces user to think about an error (we know that function returns Nullable and we have to handle it); exception can be silently ignored

[ararslan]

IMO exception objects should stay out of control flow. If something goes wrong and emits an error, the error should be thrown. If you really need to deal with an exception object, catch it. The container-based null (value is or is not present) says to the user, "This worked fine; the code executed normally. Some valid logic in the code has told me that there should not be a value here."

What you're describing, @tkelman, sounds like result types. IMO they have their place, but generally speaking, if you have Result{T, Exception} as the result of a function call, you lose the stack trace information associated with the error. Only knowing the type of the error object not really all that useful, since you as a caller can't know what in the function caused it, or even if it was intentional.