Text processing and analysis

[jeroen]

Text Analysis in R

Text analysis is a scientific subspace that is not served well by general tidyverse / datascience tools. It is a focus area of rOpenSci and we have interesting tools and expertise around this topic.

• hunspell: High-Performance Stemmer, Tokenizer, and Spell Checker
• spelling: spell checker based on hunspell
• tokenizers (Lincoln Mullen): high level tokenizing toolkit
• textreuse: measuring similarity among documents
• pdftools: extract text or data from pdf files
• tabulizer: extract tables from pdf files
• rtika: extract text or metadata from any file format
• tesseract: ocr engine (text from images)
• antiword, unrtf: extract text from word/rtf files.
• cld2, cld3: language detection

Besides our own packages, we also have a lot of experts in our network doing interesting things. The tidytext package was conceived at unconf 2016 and Kenneth and Patrick organized the ropensci textworkshop 2017 and 2018.

Suggested speakers and topics

The call could be an intro/overview of text tools, or alternatively an expert could give an in-depth talk about a specific tool. We should not restrict this to official "ropensci packages", but reach out to our network about developments in the space that are interesting to our community.

Some cool recent work is done by the quanteda group from Kenneth Benoit. They are also building out a commercial product based on the quanteda R package. It would be interesting to hear what is new with quanteda, and their experience with building a product on top of R.

[spcanelon]

I would really enjoy a community call on this topic! My research leverages electronic health record data and so far my skills have limited me to analyzing structured data (e.g. billing codes, encounter dates). There is a wealth of clinical context that can be gleaned from unstructured data contained in clinical notes, but I find myself lacking a overview of the tools in R that can support text analysis. Most of my colleagues that do text analysis use only Python and I would find it helpful to get a better sense of what's out there in the R universe.