Conversion.TryParse parses floats and doubles with empty/whitespace values as 0 instead of NaN

[Dmitry-A]

This is done for historical reasons since there are clients that depend on this behavior, so it's not a bug. Nevertheless, if we don't control the input dataset format this would produce signal loss since empty/whitespace values are now indistinguishable from value "0". We can fix that in AutoML by inserting a transform that addresses the issue but a much better fix would be to change the loader behavior.

[justinormont]

I agree. The dataset has missing values and the missing values are best represented as NA values.

As you mention, this is currently a loss of signal. When the missing value is important, we will not be able to tell 0 from NA, which causes a drop in accuracy.

The user can fill the NA values with the standard NAHandle / NAIndicator transforms.

If we want this as non-breaking change, we could add a TextLoader flag impute=true which by default keeps the current behavior. Then on the next release we can default to false.

[codemzs]

We use "?" to represent missing values in Text Loader, empty string is treated as default value for that type.

[justinormont]

Correct. The user can use "?" to represent missing values, but the majority of datasets represent missing values with either blank values 1,,2 (note the double ,, w/ a missing value between), or sometimes white space 1, ,2.

[mathias-brandewinder]

I assume this issue refers to LoadFromTextFile silently converting missing/blank values in a CSV file to 0. It really doesn't seem like a good default choice, I was very surprised to observe that behavior, and only caught it by chance.

Besides modifying the source data, replacing ,, with ,?, at the source, is there a clean way to address that issue and read ,, as NaN?

[antoniovs1029]

Hi, @mathias-brandewinder sorry for the late response. As of now, I don't think there's any way to fix this except by adding a ? into the missing values in your dataset.

On the other hand, I wonder if this behavior is done so that this matches the behavior of missing values on sparse datasets, where missing values are assumed to be 0 (the default).

In any case, I think that changing this behavior for Conversions.TryParse() wouldn't change the behavior of sparse datasets, so it might be safe to do this... although I wonder where else could Conversions.TryParse() from TX to R4/R8 be used aside from the TextLoader 😕

So yeah, perhaps adding an imputer option to the TextLoader, as suggested by @justinormont might be the best solution.