Future of the document type providers

[forki]

Hi,

since @tpetricek created https://github.com/tpetricek/FSharp.Data I wonder if we should still have the data type providers in fsharpx. It seems people like this new project and it has excellent documentation http://tpetricek.github.com/FSharp.Data/docs/FSharpData.html.

Should we remove them from fsharpx?

Cheers,
Steffen

@dsyme @mausch

[ovatsus]

I think in the long run we could probably remove them, but I would wait a little longer until FSharp.Data is more feature complete, there's still a couple of things the document type providers have that aren't present there

[forki]

@ovatsus which features are missing?

@tpetricek what do you think?

[tpetricek]

Sorry for the slow reply!

I'm not sure what is the best thing to do. I think FSharp.Data should not be home to all type providers, but it might be home to all type providers that work with data (document type providers, WorldBank, possibly Freebase, if we can sensibly include it without breaking too many things).

I think FSharp.Data now has better inference engine and I also reimplemented CSV to support the RFC (hopefuly). What is missing or needs to be done for FSharp.Data?

1. Support for portable libraries is the main missing thing, but there has been some initial work (see this discussion) to make this work and it is the top priority.
2. The project has tests to cover the tricky algorithms (CSV parsing, type inference), but it needs more tests for the end-user providers (these could be ported from FSharpX, which does this better)
3. The providers in FSharp.Data do not support constructing of documents. They only support reading.

I think (1) is the key thing that needs to be done before we can consider it "complete" and move it under the fsharp org on GitHub (and I want to do this before blogging about the library etc.)

FSharpX supports (3) and I think this is quite tricky thing. I'm not sure what the current status is in FSharpX, but there was a mutable version for some providers (XML?) and @forki did some pretty interesting work on zipper-based immutable version for JSON. This might not be needed for accessing and reading data, but I can definitely understand there are scenarios where this is useful. On the other hand, it makes the provider less "focused" and might complicate the generated API (though it could require a static parameter). It also complicates the runtime (it needs either zipper implementation or mutable types etc.)

So, I think figuring out what to do about (3) is the key thing. Do we want editing support in the document type providers (any ideas how often people use this?), if so, should this use mutable style or immutable style? Should this be in FSharp.Data (which tries to focus on simpler data access), in experimental branch, FSharpX or somewhere else?

Also, I think the community has already learned to contribute to FSharpX, and I think it would be good if this could continue - so it might make sense to build providers here and move them to FSharp.Data once they become more stable...

[ovatsus]

As for (1), FSharpx only supports portable in Freebase type provider yet, so it's not an issue.
I'm more worried about (3). It should be off by default and it shouldn't compromise the reading interface (the current work on the json branch has that problem), which should be the main focus, but I think it should be there (mutable or immutable, I would prefer immutable, though). At least on the json type provider when consuming web apis, many times there's the need to send data back

[tpetricek]

There is also a question whether people more often need to create a new JSON value (to be passed to a web API) from scratch or whether they need to clone and modify existing value they got from somewhere else?

(In the first case, it feels like we just need something to simplify construction of well-typed JSON values. In the second case, we really need some way of cloning or modifying values.)

[forki]

Regarding the Zipper. I think this should be configured via static param in the type provider and off by default.

[tpetricek]

Configuring whether to use zipper or not via a static param sounds like a good approach to me!

The read-only JSON provider uses a very simple runtime representation (just a JsonValue, which is a discriminated union returned by the JSON parser). The zipper version would need its own (more fancy) runtime objects, but I guess that can be built on top of the one that is already there.

I think adding that would probably mean implementing an alternative version of the generator (JsonGenerator.fs), which is quite manageable (~250 lines).

However, if we add this for JSON, do you think we will need similar thing for XML and CSV?

[forki]

The zipper uses JsonValue inside the Zipper. No problem there.

However, if we add this for JSON, do you think we will need similar thing for XML and CSV?

One step after the other ;-)

[tpetricek]

I agree that "one step after the other" makes a good sense, but I think we need to consider this in advance - in order to keep the API and type providers uniform (to a reasonable extent).

If JSON uses immutable style, XML uses mutable style and CSV is read-only, then it might be pretty confusing (and in that case, perhaps it might be better to clearly separate this from the more consistent part?)

[ovatsus]

I think it makes sense for the 3 to have immutable style write api's. Even today I used the csv provider, and then needed to generate another csv with some of the columns with the numbers changed, and I ended using the raw Columns string[] as a quick hack becuse that functionality wasn't there out of the box

[forki]

It's always the same. At one point you want write access...

[tpetricek]

@ovatsus Did you need immutable style editing for CSV files? On Twitter @Rickasaurus mentioned he is working with very large CSV files (tens of MB) and for that I could imagine that mutation would probably be preferable for performance reasons.

But perhaps mixing mutable style (for some) and immutable style (for other) is not such a big problem (especially if these features were enabled by static parameter)

[ovatsus]

Even if the CSV is very large, each row is not, and you typically change each row at a time, so I'd prefer immutable.

Currently I am doing this:

type csvType = CsvProvider<inputFile>
let input = csvType.Load inputFile
use output = new StreamWriter(File.OpenWrite(inputFile.Replace(".csv", ".output.csv")))
output.WriteLine(String.Join(",", input.Headers))
for row in input.Data do
    //do something with row.A row.B row.C
    let newB = (...)
    let newC = (...)
    row.Columns.[1] <- newB.ToString()
    row.Columns.[2] <- newC.ToString()
    output.WriteLine(String.Join(",", row.Columns))

and I would like to be able to do this instead:

type csvType = CsvProvider<inputFile>
let input = csvType.Load inputFile
use output = new StreamWriter(File.OpenWrite(inputFile.Replace(".csv", ".output.csv")))
output.WriteLine(String.Join(",", input.Headers))
for row in input.Data do
    //do something with row.A row.B row.C
    let newB = (...)
    let newC = (...)
    let newRow = row.With.B(newB).C(newC)
    output.WriteLine(newRow.Serialize())

With the added benefit of .Serialize handling the csv escaping for me.
Probably better than a Serialize method on the row would be to have the following functions:

CsvType.Save : filename:string -> RowType seq
CsvType.Save : Stream -> RowType seq

that way we could just use

input.Data |> Seq.map ( (...) row.With (....) ) |> CsvType.Save outputFile

Also, I think if we manage to get all the members of the write interface in a different type, accessible with a .With for example, there's really no big pollution of the main type with lot of members, so we could avoid adding another option to say we need the write functionality, and have it always present, both for CSV, XML, and JSON