spelling variants in ott 2.0

[rhr]

Some names (seemingly from GBIF) have unique ids but are likely spelling variants based on things like umlauts. E.g.

Achelous, Acheloüs
Achnanthes plonensis, Achnanthes plönensis, Achnanthes ploenensis

In mysql, searching for 'Achelous' will return 'Acheloüs' as well. I would like to be able to put a unique index on the unique_name field, but can't if these are distinct names.

[jar398]

I'm not sure how to fix this one. It is a natural consequence of relying on
outside sources for taxonomies.

OTToL v1 suppressed Acheloüs, but v2 didn't, which is why it may look like
a new problem, but the problem existed before, for other names. So you are
imposing new requirements on v2 that you didn't expect of v1. That's OK, I
just want to note it.

NCBI and GBIF both have Achelous and they get unified in OTT 2.0. The name
finds its way into GBIF via Catalogue of Life, where it is spelled without
an umlaut. CoL says it comes from WoRMS, and if you go to WoRMS, it's also
spelled without the umlaut, and the umlaut spelling isn't given as a
synonym. If you look in GBIF to see where they got the umlaut spelling,
they say they got it from WoRMS! GBIF gives a link to a WoRMS Darwin Core
archive that supposedly is their source, but the link is read-protected so
I can't look at it.

Picked "Dieffenbachia bürgeri" as a spot check and it too occurs in both
forms. With umlaut it supposedly comes from CoL, without it comes from IPNI
(opposite of the other example). If you go to CoL the web site spells it
without umlaut... as does IPNI...

So the origin of the problem is obscure.

Here are some options:

1. Report the problem to GBIF and wait for them to fix it (could easily be
   1-2 years) (will file an issue now)
2. Suppress taxa whose names contain diacritics, when the undiacriticed
   variant exists. (Or vice versa)
3. Remove diacritics and let the taxonomy merging algorithm decide whether
   the two taxa constitute synonyms or homonyms.
4. Generate longer 'unique names' where the two variants exist - the
   problem then is that if they really are synonyms it may be a pain to find
   attributes that distinguish them
5. Manual curation. (We have 169 names containing u-umlaut alone...)

NCBI seems to do something like #3; there are no u-umlauts in names.dmp.

The difficulty is that we don't have any way to know whether Achelous and
Acheloüs are supposed to name the same taxon, or not, just as when we have
two entries in the taxonomy for Achelous, or one entry in NCBI and one in
GBIF, we don't know whether they are supposed to name the same taxon or
not. There are heuristics, such as checking the taxon's lineage and
subsumption, but they aren't very precise.

My code isn't set up to deal with synonyms within a single source (i.e.
GBIF) that aren't marked as such. Doing something like #3 requires a new
policy and is going to involve a fair bit of work and QC. I'd have to
decide what to do about weird cases such as Tripteroides in NCBI, which
currently I leave alone, but which would fall naturally under any synonym
detection regime.

Advice welcome.

On Wed, May 1, 2013 at 2:23 PM, Rick Ree notifications@github.com wrote:

Some names (seemingly from GBIF) have unique ids but are likely spelling
variants based on things like umlauts. E.g.

Achelous, Acheloüs
Achnanthes plonensis, Achnanthes plönensis, Achnanthes ploenensis

Is there a catalog of such variants somewhere? Or would a solution require
fuzzy matching or spelling correction? I would not look forward to
getting involved in sameness/differentness decisions based on fuzzy
matching - the probability of error is too great. e.g. Euphorbia bergeri is
(I think) not a spelling variant of Euphorbia bürgeri .

Jonathan

In mysql, searching for 'Achelous' will return 'Acheloüs' as well. I would
like to be able to put a unique index on the unique_name field, but can't
if these are distinct names.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/14
.

[mtholder]

I guess my gut instinct is that the most maintainable path would be to:

A. modify smasher to take an additional file (format TBD) that modifies
the taxonomy in an ad hoc way.

B. create an initial version of the ad hoc taxonomy edit file via JAR's
suggestion #2 ("Suppress taxa whose names contain diacritics, when the
undiacriticed variant exists.") and just note the provenance for each entry
as something like automated_deletion_of_lexical_variant. And

C. put that a version of that file in version control

At some point we're going to have to start dealing with corrections to the
taxonomy. It would be nice if there was a really transparent process so
that we could just point interested parties to the ad hoc taxonomy edit
file to know what we've done. We don't want to get bogged down in
maintaining a db of taxonomic names. So I'm not suggesting we start
encouraging people to modify the ad hoc edit file and send us pull requests.

If we do need some of the names with diacritics, we can just make not of
that revision in the file. Presumably phylografter could use some sort of
COLLATE syntax with mysql if we need to distinguish "u" from "u with
umlaut"

cheers,
Mark

On Thu, May 2, 2013 at 11:52 AM, Jonathan A Rees
notifications@github.comwrote:

I'm not sure how to fix this one. It is a natural consequence of relying
on
outside sources for taxonomies.

OTToL v1 suppressed Acheloüs, but v2 didn't, which is why it may look like
a new problem, but the problem existed before, for other names. So you are
imposing new requirements on v2 that you didn't expect of v1. That's OK, I
just want to note it.

NCBI and GBIF both have Achelous and they get unified in OTT 2.0. The name
finds its way into GBIF via Catalogue of Life, where it is spelled without
an umlaut. CoL says it comes from WoRMS, and if you go to WoRMS, it's also
spelled without the umlaut, and the umlaut spelling isn't given as a
synonym. If you look in GBIF to see where they got the umlaut spelling,
they say they got it from WoRMS! GBIF gives a link to a WoRMS Darwin Core
archive that supposedly is their source, but the link is read-protected so
I can't look at it.

Picked "Dieffenbachia bürgeri" as a spot check and it too occurs in both
forms. With umlaut it supposedly comes from CoL, without it comes from
IPNI
(opposite of the other example). If you go to CoL the web site spells it
without umlaut... as does IPNI...

So the origin of the problem is obscure.

Here are some options:

1. Report the problem to GBIF and wait for them to fix it (could easily be
   1-2 years) (will file an issue now)
2. Suppress taxa whose names contain diacritics, when the undiacriticed
   variant exists. (Or vice versa)
3. Remove diacritics and let the taxonomy merging algorithm decide whether
   the two taxa constitute synonyms or homonyms.
4. Generate longer 'unique names' where the two variants exist - the
   problem then is that if they really are synonyms it may be a pain to find
   attributes that distinguish them
5. Manual curation. (We have 169 names containing u-umlaut alone...)

NCBI seems to do something like #3; there are no u-umlauts in names.dmp.

The difficulty is that we don't have any way to know whether Achelous and
Acheloüs are supposed to name the same taxon, or not, just as when we have
two entries in the taxonomy for Achelous, or one entry in NCBI and one in
GBIF, we don't know whether they are supposed to name the same taxon or
not. There are heuristics, such as checking the taxon's lineage and
subsumption, but they aren't very precise.

My code isn't set up to deal with synonyms within a single source (i.e.
GBIF) that aren't marked as such. Doing something like #3 requires a new
policy and is going to involve a fair bit of work and QC. I'd have to
decide what to do about weird cases such as Tripteroides in NCBI, which
currently I leave alone, but which would fall naturally under any synonym
detection regime.

Advice welcome.

On Wed, May 1, 2013 at 2:23 PM, Rick Ree notifications@github.com
wrote:

Some names (seemingly from GBIF) have unique ids but are likely spelling
variants based on things like umlauts. E.g.

Achelous, Acheloüs
Achnanthes plonensis, Achnanthes plönensis, Achnanthes ploenensis

Is there a catalog of such variants somewhere? Or would a solution require
fuzzy matching or spelling correction? I would not look forward to
getting involved in sameness/differentness decisions based on fuzzy
matching - the probability of error is too great. e.g. Euphorbia bergeri
is
(I think) not a spelling variant of Euphorbia bürgeri .

Jonathan

In mysql, searching for 'Achelous' will return 'Acheloüs' as well. I
would
like to be able to put a unique index on the unique_name field, but
can't
if these are distinct names.

—
Reply to this email directly or view it on GitHub<
https://github.com/OpenTreeOfLife/opentree/issues/14>
.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/14#issuecomment-17350066
.

Mark Holder

mtholder@gmail.com
mtholder@ku.edu
http://phylo.bio.ku.edu/mark-holder

Department of Ecology and Evolutionary Biology
University of Kansas
6031 Haworth Hall
1200 Sunnyside Avenue
Lawrence, Kansas 66045

lab phone: 785.864.5789

fax (shared): 785.864.5860

[rhr]

I think #3 is probably the easiest and most effective. Attached is a file
with the variants as they exist in mysql.

On Thu, May 2, 2013 at 11:52 AM, Jonathan A Rees
notifications@github.comwrote:

I'm not sure how to fix this one. It is a natural consequence of relying
on
outside sources for taxonomies.

OTToL v1 suppressed Acheloüs, but v2 didn't, which is why it may look like
a new problem, but the problem existed before, for other names. So you are
imposing new requirements on v2 that you didn't expect of v1. That's OK, I
just want to note it.

NCBI and GBIF both have Achelous and they get unified in OTT 2.0. The name
finds its way into GBIF via Catalogue of Life, where it is spelled without
an umlaut. CoL says it comes from WoRMS, and if you go to WoRMS, it's also
spelled without the umlaut, and the umlaut spelling isn't given as a
synonym. If you look in GBIF to see where they got the umlaut spelling,
they say they got it from WoRMS! GBIF gives a link to a WoRMS Darwin Core
archive that supposedly is their source, but the link is read-protected so
I can't look at it.

Picked "Dieffenbachia bürgeri" as a spot check and it too occurs in both
forms. With umlaut it supposedly comes from CoL, without it comes from
IPNI
(opposite of the other example). If you go to CoL the web site spells it
without umlaut... as does IPNI...

So the origin of the problem is obscure.

Here are some options:

1. Report the problem to GBIF and wait for them to fix it (could easily be
   1-2 years) (will file an issue now)
2. Suppress taxa whose names contain diacritics, when the undiacriticed
   variant exists. (Or vice versa)
3. Remove diacritics and let the taxonomy merging algorithm decide whether
   the two taxa constitute synonyms or homonyms.
4. Generate longer 'unique names' where the two variants exist - the
   problem then is that if they really are synonyms it may be a pain to find
   attributes that distinguish them
5. Manual curation. (We have 169 names containing u-umlaut alone...)

NCBI seems to do something like #3; there are no u-umlauts in names.dmp.

The difficulty is that we don't have any way to know whether Achelous and
Acheloüs are supposed to name the same taxon, or not, just as when we have
two entries in the taxonomy for Achelous, or one entry in NCBI and one in
GBIF, we don't know whether they are supposed to name the same taxon or
not. There are heuristics, such as checking the taxon's lineage and
subsumption, but they aren't very precise.

My code isn't set up to deal with synonyms within a single source (i.e.
GBIF) that aren't marked as such. Doing something like #3 requires a new
policy and is going to involve a fair bit of work and QC. I'd have to
decide what to do about weird cases such as Tripteroides in NCBI, which
currently I leave alone, but which would fall naturally under any synonym
detection regime.

Advice welcome.

On Wed, May 1, 2013 at 2:23 PM, Rick Ree notifications@github.com
wrote:

Some names (seemingly from GBIF) have unique ids but are likely spelling
variants based on things like umlauts. E.g.

Achelous, Acheloüs
Achnanthes plonensis, Achnanthes plönensis, Achnanthes ploenensis

Is there a catalog of such variants somewhere? Or would a solution require
fuzzy matching or spelling correction? I would not look forward to
getting involved in sameness/differentness decisions based on fuzzy
matching - the probability of error is too great. e.g. Euphorbia bergeri
is
(I think) not a spelling variant of Euphorbia bürgeri .

Jonathan

In mysql, searching for 'Achelous' will return 'Acheloüs' as well. I
would
like to be able to put a unique index on the unique_name field, but
can't
if these are distinct names.

—
Reply to this email directly or view it on GitHub<
https://github.com/OpenTreeOfLife/opentree/issues/14>
.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/14#issuecomment-17350066
.

[jar398]

Rick, I don't see the attachment.

[jar398]

Reported to GBIF:
https://code.google.com/p/gbif-ecat/issues/detail?id=98

[jar398]

I agree that #3 is better in ways but I think Mark's solution is the one I prefer right now, given that (a) the ad-hoc edits feature now works and (b) #3 would require new logic. In addition #3 would only deal with diacritics, not with spelling variants - ad-hoc edits can deal with whatever variants we can dream up.
I need to implement a 'delete' command, but that will be easy. Will leave this issue open pending that.

[jar398]

The patch feature works so this is ready to be fixed. I have Rick's file in my email stream and will put it under version control. Thus: Mark's A + B + C plan. Sorry for the long wait.

[jar398]

Wrote code to deal with diacritics automagically, but single-letter diacritic/non-diacritic variants only. Now in reference-taxonomy repository. Should be active in OTT 2.4 when it comes out. OpenTreeOfLife/reference-taxonomy@d7fc779

[jar398]

I'm going to close the issue here but reopen it in the other repo. Please see OpenTreeOfLife/reference-taxonomy#1 .