ONNX models dont run for Handler, Indicator & others

[ganik]

Repro script:
`
from nimbusml.datasets import get_dataset
from nimbusml.preprocessing import OnnxRunner
from nimbusml.preprocessing.missing_values import Handler

iris_df = get_dataset("iris").as_df()
nahandle = Handler(replace_with='Mean') << {'NewVals': 'Sepal_Length'}
nahandle.fit(iris_df)
nahandle.export_to_onnx("test.onnx", 'com.microsoft.ml')
onnx_runner = OnnxRunner(model_file="test.onnx")
onnx_runner.fit_transform(iris_df)
`
Error: *** System.ArgumentOutOfRangeException: 'Schema mismatch for input column 'Label': expected UInt32, got Key<UInt32, 0-2>

Same happens for Indicator, MeanVarianceScaler, MinMaxScaler, LpScaler, ToKey, etc.

[antoniovs1029]

For the record, this issue is originated when using a panda's DataFrame with a column of type category. When faced with this case, NimbusML converts the DataFrame into a IDataView, where a column of type KeyDataViewType is used in place of the category column.

For doing this, NimbusML doesn't actually add a ValueToKeyTransformer into the pipeline that is to be executed in ML.NET.

This is usually not a problem, except when saving the model as an ONNX model and using that model with the same input dataframe (as seen in the sample that @ganik provided). Which is where this exception appears.

Problem is that when ML.NET saves the pipeline as an ONNX model, the ONNX model doesn't contain the information to map the input KeyDataViewType into a UInt32. If the pipeline had a ValueToKeyTransformer, then the model will include such mapping. But since NimbusML doesn't include that in the pipeline, the Onnx model expects a UInt32 input, but the input dataview provides a KeyDataViewType column instead. The mismatch causes the exception.

After discussing this offline with @ganik , @pieths and @harishsk , the best (perhaps temporary) solution for this problem is to modify the OnnxTransformer in ML.NET to handle this special case. This has been done in dotnet/machinelearning#4824 .

Other alternative solutions might eventually include actually adding a ValueToKeyTransformer in the pipeline, either in NimbusML side or ML.NET side to avoid this kind of problems.

[harishsk]

This is fixed by dotnet/machinelearning#4824

[antoniovs1029]

PR dotnet/machinelearning#4824 only enabled the ONNX model to work with KeyDataViewType columns as if they were UInt32.

Problem was that the output of the ONNX model after that change wasn't the correct one when running the repro in here, so that PR didn't complety fix the problem. The reason for this is described in
dotnet/machinelearning#4841