ONNX model for OneHotVectorizer produces different result

[ganik]

Repro
`from nimbusml.datasets import get_dataset
from nimbusml.preprocessing import OnnxRunner
from nimbusml.feature_extraction.categorical import OneHotVectorizer

infert_df = get_dataset("infert").as_df()
infert_df.columns = [i.replace(': ', '') for i in infert_df.columns]
infert_df.rename(columns={'case': 'Label'}, inplace=True)

transform = OneHotVectorizer() << 'education_str'
print(transform.fit_transform(infert_df))
transform.export_to_onnx("test.onnx", 'com.microsoft.ml')
onnx_runner = OnnxRunner(model_file="test.onnx")
print(onnx_runner.fit_transform(infert_df))`

Output:
row_num education age ... education_str.0-5yrs education_str.6-11yrs education_str.12+ yrs
0 1 0.0 26.0 ... 1.0 0.0 0.0
1 2 0.0 42.0 ... 1.0 0.0 0.0
2 3 0.0 39.0 ... 1.0 0.0 0.0
3 4 0.0 34.0 ... 1.0 0.0 0.0
4 5 2.0 35.0 ... 0.0 1.0 0.0
.. ... ... ... ... ... ... ...
243 244 1.0 31.0 ... 0.0 0.0 1.0
244 245 1.0 34.0 ... 0.0 0.0 1.0
245 246 1.0 35.0 ... 0.0 0.0 1.0
246 247 1.0 29.0 ... 0.0 0.0 1.0
247 248 1.0 23.0 ... 0.0 0.0 1.0

[248 rows x 12 columns]
row_num education age ... education_str.onnx.0 education_str.onnx.1 education_str.onnx.2
0 1 0.0 26.0 ... 0.0 1.0 0.0
1 2 0.0 42.0 ... 0.0 1.0 0.0
2 3 0.0 39.0 ... 0.0 1.0 0.0
3 4 0.0 34.0 ... 0.0 1.0 0.0
4 5 2.0 35.0 ... 0.0 0.0 1.0
.. ... ... ... ... ... ... ...
243 244 1.0 31.0 ... 0.0 0.0 0.0
244 245 1.0 34.0 ... 0.0 0.0 0.0
245 246 1.0 35.0 ... 0.0 0.0 0.0
246 247 1.0 29.0 ... 0.0 0.0 0.0
247 248 1.0 23.0 ... 0.0 0.0 0.0

[248 rows x 22 columns]

[pieths]

Here is a second script which shows the issue with a smaller dataset.

import os
import tempfile
import numpy as np
import pandas as pd
from nimbusml.feature_extraction.categorical import OneHotVectorizer
from nimbusml.preprocessing import OnnxRunner

def get_tmp_file(suffix=None):
    fd, file_name = tempfile.mkstemp(suffix=suffix)
    fl = os.fdopen(fd, 'w')
    fl.close()
    return file_name

pd.set_option('display.max_columns', None)

data = {'a': [1, 2, 2, 3], 'b': [5, 5, 6, 7]}
data = pd.DataFrame(data).astype(np.float32)
print(data)

xf = OneHotVectorizer(columns=['a', 'b'])
xf.fit(data)

result_mlnet = xf.transform(data)
print(result_mlnet)

onnx_path = get_tmp_file('.onnx')
xf.export_to_onnx(onnx_path, 'com.microsoft.ml', onnx_version='Stable')
onnxrunner = OnnxRunner(model_file=onnx_path)
result_onnx = onnxrunner.fit_transform(data)
print(result_onnx)

#      a    b
# 0  1.0  5.0
# 1  2.0  5.0
# 2  2.0  6.0
# 3  3.0  7.0
#
#    a.1  a.2  a.3  b.5  b.6  b.7
# 0  1.0  0.0  0.0  1.0  0.0  0.0
# 1  0.0  1.0  0.0  1.0  0.0  0.0
# 2  0.0  1.0  0.0  0.0  1.0  0.0
# 3  0.0  0.0  1.0  0.0  0.0  1.0
#
#      a    b  a.onnx.0  a.onnx.1  a.onnx.2  b.onnx.0  b.onnx.1  b.onnx.2
# 0  1.0  5.0       0.0       1.0       0.0       0.0       1.0       0.0
# 1  2.0  5.0       0.0       0.0       1.0       0.0       1.0       0.0
# 2  2.0  6.0       0.0       0.0       1.0       0.0       0.0       1.0
# 3  3.0  7.0       0.0       0.0       0.0       0.0       0.0       0.0

[harishsk]

This should be fixed by #4827

[pieths]

Fixed by dotnet/machinelearning#4827