Add PreferFloats static param to CsvProvider

[ranigb]

There is a consistency issue between the default way decimals are printed and the way they are inferred. Values like 1.8, -2.22, 3.4 will be inferred to be Decimal. However if you printf "%a" these values, they will be represented as 1.8M, -2.22M, 3.4M. Reading them again will result in strings instead of decimals. It would be nice if there could be more consistency.

There are several solutions I can think of each one of them with pros and cons:

1. The type of numbers ending with M could be inferred to be Decimal
2. numbers like 1.8, -2.22, 3.4 can be inferred to be floats
3. we can allow the user to select what to do in case of ambiguity. For example a columns that contains 1.8, 2.3, 5.6 can be converted to strings, floats and decimals. The current default is to convert them to decimals but we can allow the user to specify that in such cases floats are preferred.
4. there are probably many other ways in which in can be resolved

[ovatsus]

%a is intended to print the internal F# representation, so csv and json files are not expected to have the M. Can you explain the scenario where you need to use %a to output data and then import it back with the providers?

[ranigb]

I can see this scenario happening in many cases in which you have a csv file, you manipulate it and would like to save the result for future analysis. In my case, you wanted to compute a join over two files and use the join for further processing. I loaded each of the files using the csv type provider, had a function that computes the join and saves it. At this point I wanted to load it again such that you can enjoy all the greatness of type providers over the joined file.

[ovatsus]

But why do you need to use %a? Any other format specifier will work. There's also the SaveToString method on CsvProvider that works well with roundtripping

[tpetricek]

I can see how one could use sprintf "%A" simply because it is the simplest option. I have not seen many CSV files that would use the 1.0M notation to explicitly denote decimals (that is probably F# specific), but I suppose one could find $1 in a CSV file...

That said, dealing with decimal values coming from the CSV provider is sometimes a bit painful, so I think having a parameter PreferFloats would be a useful thing (also, I think floats are generally faster).

So, perhaps we could turn this into up-for-grabs feature suggestion to add PreferFloats? (Although the number of parameters is growing in a slightly scary way :-) but I actually wanted this one recently)

[ovatsus]

Why do you say dealing with decimals is painful? If there are no perf issues, decimals are safer, and you can at least = them without having to worry about precision. We could add a parameter, but there are already so many... It's starting to look like R

[tpetricek]

There are definitely pros and cons. The fact that you can compare decimals is a big win.

The one concrete "painful" thing is that you cannot do sqrt 2.0M or sin 2.0M and generally, there is more .NET API that accepts floats than decimals. I was using decimals in some of my training materials and replaced them with floats for this reason (this is materials for beginners - so I wanted to avoid having to do sqrt (float num) even though this is easy enough in practice.

They seem to be a bit slower too (Array.average is about 4 times slower on 1k decimals than on 1k floats). Again, not a bit deal in many cases, but it could be a problem for some cases (e.g. if you load data into Deedle, it will not convert them to floats).

I also don't like the number of flags. But I think we could leave this as up-for-grabs and if there is enough interest in adding the feature, we could accept a PR adding it.

Incidentally, I think some of the parameters could be removed from the type provider and instead passed at runtime to the load etc. methods (well, at least one would be CacheRows).

[ranigb]

I wanted to use %a in a place where I waned to write something without assuming the types. My case was creating a join on a specific field. I knew the type of this field but I was trying to avoid assuming the types of the other fields.

Rani

Date: Sun, 7 Jun 2015 11:10:52 -0700
From: notifications@github.com
To: FSharp.Data@noreply.github.com
CC: rani_gb@hotmail.com
Subject: Re: [FSharp.Data] decimal type inferrence (#838)

But why do you need to use %a? Any other format specifier will work. There's also the SaveToString method on CsvProvider that works well with roundtripping

—
Reply to this email directly or view it on GitHub.

[ranigb]

Tomas,

If I understand correctly, if you do sqrt (float num) you will end up with a float. Therefore, if I have two fields num and factor and I would like to compute factor * (num^0.5) I will have to write factor * decimal (sqrt (float num)) instead of factor * sqrt num which can be annoying.

I understand the pain in adding more flags. Maybe, the way around it is to create a "type conversion" type provider. This type will take the type provided by the csv type provider and convert the column according to the specified preferences. For example, you can prefer floats (or singles) over decimals, you can tell it to convert binaries to ints and so on.

Rani

Date: Sun, 7 Jun 2015 13:49:55 -0700
From: notifications@github.com
To: FSharp.Data@noreply.github.com
CC: rani_gb@hotmail.com
Subject: Re: [FSharp.Data] decimal type inferrence (#838)

There are definitely pros and cons. The fact that you can compare decimals is a big win.

The one concrete "painful" thing is that you cannot do sqrt 2.0M or sin 2.0M and generally, there is more .NET API that accepts floats than decimals. I was using decimals in some of my training materials and replaced them with floats for this reason (this is materials for beginners - so I wanted to avoid having to do sqrt (float num) even though this is easy enough in practice.

They seem to be a bit slower too (Array.average is about 4 times slower on 1k decimals than on 1k floats). Again, not a bit deal in many cases, but it could be a problem for some cases (e.g. if you load data into Deedle, it will not convert them to floats).

I also don't like the number of flags. But I think we could leave this as up-for-grabs and if there is enough interest in adding the feature, we could accept a PR adding it.

Incidentally, I think some of the parameters could be removed from the type provider and instead passed at runtime to the load etc. methods (well, at least one would be CacheRows).

—
Reply to this email directly or view it on GitHub.

[ovatsus]

Updated the title. @ranigb feel free to send a PR adding the PreferFloats param