IFormatProvider in TextLoader

[ddobric]

Want to know how TextLoader deals with files in different cultures.
For example, if you are running on OS with culture like "de-de" such files might be loaded incorrectly:

CMT,1,1,1267,6.9,CRD,23
CMT,1,1,1267,6.8,CRD,24

With such cultures (none-invariant) would require files formatted in following way (separator = ';')

CMT;1;1;1267;6,9,CRD;23
CMT;1;1;1267;6,8,CRD;23

[veikkoeeva]

Cross-referencing since related: #74 and #109. And since this is a real problem, feeding thoughts: https://www.nuget.org/packages?q=CSV.

[veikkoeeva]

@ddobric You can see it at

machinelearning/src/Microsoft.ML/TextLoader.cs

Line 16 in 3780923

public class TextLoader<TInput> : ILearningPipelineLoader

.

I wonder if it were possible to do something like this

    /// <summary>
    /// A canonical interface for deserializer.
    /// </summary>
    public interface IDeserializer
    {
       /// <summary>
       /// Can this provider stream data.
       /// </summary>
       bool CanStream { get; }

       /// <summary>
       /// Deserializes the given data.
       /// </summary>
       /// <param name="data">The data to be serialized.</param>
       /// <param name="destination">The destination type.</param>
       /// <returns>The deserialized object.</returns>
       object Deserialize(object data, Type destination);

/// <summary>
/// Deserializes the given data from a stream.
/// </summary>
/// <param name="dataStream">The stream from which to serialize.</param>
/// <param name="destination">The destination type.</param>
/// <returns>The deserialized object.</returns>
object Deserialize(Stream dataStream, Type destination);

/// <summary>
/// Deserializes the given data from a text stream.
/// </summary>
/// <param name="reader">The text stream from which to serialize.</param>
/// <param name="destination">The destination type.</param>
/// <returns>The deserialized object.</returns>
object Deserialize(TextReader reader, Type destination);
}

/// <summary>
    /// A default JSON deserializer.
    /// </summary>
    [DebuggerDisplay("CanStream = {CanStream}")]
    public class DefaultJsonDeserializer: IDeserializer
    {
        public JsonSerializer Deserializer { get; }
        public JsonSerializerSettings Settings { get; }
        public bool CanStream { get; } = true;

        public DefaultJsonDeserializer(JsonSerializerSettings settings)
        {
            if(settings == null)
            {
                throw new ArgumentNullException(nameof(settings));
            }
            Deserializer = JsonSerializer.Create(settings);
            Settings = settings;
        }

        public object Deserialize(Stream dataStream, Type destination)
        {
            using(var streamReader = new StreamReader(dataStream))
            {
                using(var reader = new JsonTextReader(streamReader))
                {
                    return Deserializer.Deserialize(reader, destination);
                }
            }
        }

        public object Deserialize(object data, Type destination)
        {
            return JsonConvert.DeserializeObject((string)data, destination, Settings);
        }

        public object Deserialize(TextReader reader, Type destination)
        {
            using(var jsonReader = new JsonTextReader(reader))
            {
                return Deserializer.Deserialize(jsonReader, destination);
            }
        }
    }
}

The real case would be more complicated and the interface could support, say IAsyncEnumerable<T>. The intention would be to make canonical interfaces for input (and output) and make concrete implementations in arbitrary ways, users could bring their own (de)serializers. This might be possible in the current system. I don't know what is schema, for instance, in MachineLearning.NET. :)

[sfilipi]

@ddobric you can specify the column separator in the TextLoader, in ML.Net.
The TextLoader's default separator is ' \t'.

The culture does not play a factor.

[codemzs]

Please try with the new API in post 1.0 release and let me know if you still see this issue.