Layout of OpenRefine, Rows vs Records - separator example

[aadoek]

In https://librarycarpentry.org/lc-open-refine/03-working-with-data/index.html

“### Choosing a good separator
Now imagine that the document creator had chosen a comma as the separator instead of a pipe.

Jones, Andrew , Davis, S.

Can you spot the problem? Can you tell where one author stops and the next begins?”

It seems to me that a better example is possible, because a plausible answer to the question “Can you spot the problem?” would be: ‘No’. To the question “Can you tell where one author stops and the next begins?” my answer would be: ‘Yes, I can: the division between two author names is a blank, followed by a comma.’
My suggestion would be:

“Now imagine that the document creator had chosen a blank as the separator instead of a pipe.

Jones, Andrew Davis, S.

Can you spot the problem? Can you tell where one author stops and the next begins?”

[ostephens]

Thanks @aadoek

I understand the point - yes it is relatively easy for us to spot the separator and the two names.

However, I think the solution is to explain the problem in more detail, rather than to change the example.

The problems of using a comma as a spearator in this context as I would see it are:

• It creates more work for the data consumer - they have to split the data up based on two characters in combination rather than just one. While this is possible, it makes the process more difficult in many tools
• It's more ambiguous - though we can spot it, it isn't quite as clear as the pipe separator - I think this is clear by eye
• "space followed by a comma" could easily occur in an author name in this type of data based whoever is entering the data making a simple typo (accidentally adding a space before a comma in an authors name

I also think using a comma as a separator is something specific to warn against since it is commonly used as a separator, and while this may be appropriate sometimes, and certainly there are solutions to using it (as in a csv file), we have an opportunity to highlight here why it isn't always the best choice

My view is that putting more context into the lesson and explaining further the issues of using a comma as a separator is a better solution here. What do you think?

[aadoek]

You are absolutely right, @ostephens. It should be made very clear that commas are not always the ideal way to seperate fields (CSV notwithstanding). And I assume LC teachers do this in their training sessions. In formulating this comment I was concentrating too much on the literal text of the lesson. And being fussy :)

[ostephens]

No problem @aadoek - I think this could be improved either by adding some context to the lesson, or adding a note to the instructor notes. I'd be in favour of a pull request making a change for either or both of these

[jas58]

I would like to add guidance to craft a question welcoming and inclusive of learners (non-experts).

For example, "Suggest what is missing or misleading between some words or phrases which might confuse the software or a non-cataloger?"

Why? Because of course I couldn't spot the problem when I first sat this workshop. I needed scaffolding for what to even look for.

[ostephens]

@jas58 I completely agree about being welcoming and inclusive

Perhaps the solution here is to not phrase this as a question - as far as I can tell the use of a question here is completely rhetorical - this isn't part of an exercise for the user to complete and the explanation immediately follows in the lesson text. So we could just omit the question and jump to the explanation with something more like:

Now imagine that the document creator had chosen a comma as the separator instead of a pipe.

Jones, Andrew , Davis, S.

If we were to tell OpenRefine to split this cell using a comma, it would split on every comma it encounters, so we would end up with 4 authors, not two, because OpenRefine cannot tell that Jones, Andrew is supposed to be a single author. We will get the following four “authors” because there are 3 commas separating the name parts.

• Author 1: Jones
• Author 2: Andrew
• Author 3: Davis
• Author 4: S.

Splitting on a comma will not work with Authors because the names may include commas within them.

We could add some more context at the end about how an experience user might spot the trick and give OpenRefine more explicit instructions to make the split successfully - although I suspect this might be more detail than is required at this point.

[jas58]

@ostephens I'll verify I understand the requested edit to make in order to close this issue.

I am taking out the paragraphs (rows 98-102)
"Can you spot the problem? Can you tell where one author stops and the next begins?

OpenRefine will split on every comma it encounters, so we’ll end up with 4 authors, not two, because OpenRefine cannot tell that Jones, Andrew is supposed to be a single author. We will get the following four “authors” because there are 3 commas separating the name parts."

and putting in, instead:
"If we were to tell OpenRefine to split this cell using a comma, it would split on every comma it encounters, so we would end up with 4 authors, not two, because OpenRefine cannot tell that Jones, Andrew is supposed to be a single author. We will get the following four “authors” because there are 3 commas separating the name parts."

Then in instructor notes, adding "An experienced user might spot the ______ and give OpenRefine more explicit instructions to successfully split author names (such as: ____________)."

Is that correct? I will need help filling in the blanks with a term (not "trick" perhaps wisdom/experience/technique?) and the explicit instructions.

[ostephens]

@jas58 that looks good to me as the right approach. I'll take a look at what an alternative approach that does the split successfully in this scenario and make a suggestion to fill out your instructor notes

[ostephens]

@jas58 I'd suggest:

An experienced user might spot that because there is an extra space, as well as a comma, between the two author names it would be possible to successfully split the two names by instructing OpenRefine to split based on a multi-character separator string like , (space comma space)

Not sure if it's clearer or less clear to include inverted commas around the multi-character separator:
" , "
vs
,
(because the inverted commas aren't part of the separator string of course - just maybe highlights the presence of the space characters more clearly)

How does that wording sound to you - is it clear and understandable?

[jas58]

Third item for 11/18? I may need guidance to see where this went. Is it part of patch-4?

[jas58]

Did we need to revisit this one? @ostephens

[jas58]

There's an instructor notes sprint coming up in May. I'll tag @ErinBecker to let her know this is one I'm interested to work on.

[jas58]

Adding the episodes updated name and link: https://librarycarpentry.org/lc-open-refine/03-working-with-data.html

[jas58]

Additional thoughts:
Second sentence "row usually reps record"--in terms of tabular data.
Contrast to relational data multiple rows may be linked to one record.

Open Refine concept of record
General data concept of record

Bring in ODLIS definition: each record one row

In OR it's possible for multiple rows to connect to one record/idea