Data 'diff' format

[rdpeng]

One thing I've always wanted is a 'diff' type output for datasets (let's say tabular datasets for now). When I use git to manage projects, changes to the datasets I use are difficult to visualize using the standard diff output, which is line based. That works when rows are changed but not when columns are added/deleted or transformations are made. Is there a way to categorize the types of changes that can be made to a dataset and then visualize them in a useful way?

[srvanderplas]

I would be most interested in this as well.

On Sun, Mar 1, 2015 at 11:10 AM, Roger D. Peng notifications@github.com
wrote:

One thing I've always wanted is a 'diff' type output for datasets (let's
say tabular datasets for now). When I use git to manage projects, changes
to the datasets I use are difficult to visualize using the standard diff
output, which is line based. That works when rows are changed but not when
columns are added/deleted or transformations are made. Is there a way to
categorize the types of changes that can be made to a dataset and then
visualize them in a useful way?

—
Reply to this email directly or view it on GitHub
#19.

[rdpeng]

Awesome! I'm not 100% sure how this would work, but I think for it to be useful it would have to sit on top of git and then maybe show how the dataset changes independent of git's own output. The issue there would then be efficiency....

[ledell]

@rdpeng, are you familar with dat? It's a version control system for data, which feels very similar to git. At the moment, I think it tracks modifications by row only, but I am hoping that column-based diffs will be part of a future release. I agree that something that tracks a variety of transformations/modifications would be very useful.

[jennybc]

👍 I was just talking to @gvwilson about this exact thing earlier this week….

[jeroen]

Also have a look at https://github.com/edwindj/daff

[karthik]

Thanks @ledell for mentioning Dat. I'm on my phone so this will be brief but I'll expand later. Dat can natively do diffs and ropensci has a rDat package in the works, waiting on Dat to come to beta (which is soon). I've invited the Dat project to join us and Karissa from their team will join us.

There are some issues with rDat that I'm hoping Jeroen will help resolve. But 💯 to pursuing this idea. It should be easy to complete at the event.

[rdpeng]

Daff looks quite good actually, and seems to implement most of what I was thinking about.

One thing I was hoping to do was implement was something a bit more "intelligent" (and likely more constraining). So for example, if I transform a column by squaring it, is there a way to show that rather than just indicating that every value in the column changed? Perhaps a diff could be expressed via R code rather than the something along the lines of the usual +/- diff format.

[gmbecker]

@rdpeng your latest comment sounds like a provenance-tracking problem. Are you thinking this will be applied in a system aware of what is done, or does it need to work like diff, I.e. given two datasets and no extra information, tell me the differences?

[jennybc]

Pardon if this is slightly off-topic but I want to park these links in a few relevant places, like this thread.

Re: weaning people off of Excel for data inspection and cleaning. OpenRefine comes up a lot and is generally popular with people expecting a GUI. I had always thought it was only mouse driven, but that it wrote some sort of log file. Did not realize these logs are perhaps re-executable. But a recent Twitter conversation intrigues me and also alerted me to Ruby and Python wrappers around the underlying Refine API. @ostephens says:

• OpenRefine captures actions which can be exported as JSON and replayed against same/other data
• the only things that aren’t captured for replay are edits at the level of individual cells
• so OpenRefine works well where the fix can be applied across rows - all transforms exportable

[rdpeng]

I'm not sure, to be honest. I think I would need a brief discussion of the
pros and cons of either one.

On Mon, Mar 9, 2015 at 5:22 PM, Gabe Becker notifications@github.com
wrote:

@rdpeng https://github.com/rdpeng your latest comment sounds like a
provenance-tracking problem. Are you thinking this will be applied in a
system aware of what is done, or does it need to work like diff, I.e. given
two datasets and no extra information, tell me the differences?

—
Reply to this email directly or view it on GitHub
#19 (comment).

Roger D. Peng | @rdpeng https://twitter.com/rdpeng |
http://www.biostat.jhsph.edu/~rpeng/

[sjackman]

I'm interested in this topic.

[vsbuffalo]

Me too! I really like this idea. In the Unix tradition, I think the best approach to an implementation might be a C or C++ library and command line tool (e.g. like curl). The, we could maybe write a simple R wrapper.

[gmbecker]

Maybe, though that requires us to write it in C/C++ instead of the much
easier R. There are benefits (though not that many to R users), but pretty
major downsides too.

I would argue that - for prototyping algorithms and features, at least -
implementing it initially in R is a more efficient use of our time.

Remember what Duncan always said: for every two lines of C you write, you
introduce 3 bugs.

~G

On Mon, Mar 23, 2015 at 10:18 PM, Vince Buffalo notifications@github.com
wrote:

Me too! I really like this idea. In the Unix tradition, I think the best
approach to an implementation might be a C or C++ library and command line
tool (e.g. like curl). The, we could maybe write a simple R wrapper.

—
Reply to this email directly or view it on GitHub
#19 (comment).

Gabriel Becker, PhD
Computational Biologist
Bioinformatics and Computational Biology
Genentech, Inc.

[bbest]

There's a slick little Chrome extension CSVHub that will visualize the daff like differencing of a CSV from within Github:

[sjackman]

@bbest Wow! That's fantastic! Oddly it doesn't work with TSV files. I've opened an issue Data-Liberation-Front/csvhub#8 to request this feature.