Binary Training Data

[jcapellman]

As shown during the .NET Overview Session today at BUILD and as discussed with the ML Team there is only a text based reader for training data. Having a binary reader for training data would be highly beneficial to my use case.

[markusweimer]

Which file format do you have in mind for us to support?

[codemzs]

We met him at BUILD and his data is in raw binary format, the reason being he has several thousand features, so may be to keep the dataset compact.

[glebuk]

Jarred,
(@jcapellman) that's an interesting suggestion. Are you looking at custom binary deserialization reader?
There are couple of options for you. One is erformant, but complex, and another is slower, but simpler.
If you are looking for ultimate performance, you will need to implement a IDataLoader interface. This will be a loader component that returns back a special datatype, IDataView. you can use the ParquetLoader as a template.

Alternatively, you can use your existing code that deserializes a binary file and make it return an IEnumerable< T > where T is the type that carries your features to the LearningPipeline. you can look at the HousePriceData example to see what T should look like. Let us know if you have any sparse data, as we have special methods to handle that as well

[abgoswam]

DRI RESPONSE: This is something what we want to have. But it doesn't look like a major priority for us right now, considering what we allow user to use enumeration of classes(List or Array) as data source.

[jcapellman]

Sorry for not looping back on this earlier - been crazy time for me since BUILD.

As we keep evaluating ML.NET vs XGBoost, one of the biggest things is our training data is now over 300 million files to do feature extraction and then train a model. As @codemzs mentioned these files are binary to begin with and keeping it in binary would save a ton of space over having 24k features written to a text file like we do today for those 300 million files.

[jwood803]

Would the addition of the binary saving and loading PR help with this?

[jcapellman]

@jwood803
Yes it would

[codemzs]

Closing this issue as we already have this functionality here: https://github.com/dotnet/machinelearning/tree/master/src/Microsoft.ML.Data/DataLoadSave/Binary

[jcapellman]

@codemzs
Thank you - on a side note, really appreciate you, the ML.NET and the community for making this an incredible framework to work with. Transformed my side projects and am going to use it for my DSc starting in August.