Add OpenNLP Analysis capabilities as a module [LUCENE-2899]

[asfimport]

Now that OpenNLP is an ASF project and has a nice license, it would be nice to have a submodule (under analysis) that exposed capabilities for it. Drew Farris, Tom Morton and I have code that does:

• Sentence Detection as a Tokenizer (could also be a TokenFilter, although it would have to change slightly to buffer tokens)
• NamedEntity recognition as a TokenFilter

We are also planning a Tokenizer/TokenFilter that can put parts of speech as either payloads (PartOfSpeechAttribute?) on a token or at the same position.

I'd propose it go under:
modules/analysis/opennlp

---

Migrated from LUCENE-2899 by Grant Ingersoll (@gsingers), 36 votes, resolved Dec 19 2017
Attachments: LUCENE-2899.patch (versions: 6), LUCENE-2899-6.1.0.patch, LUCENE-2899-RJN.patch, OpenNLPFilter.java, OpenNLPTokenizer.java
Linked issues:

• SOLR-4793
• SOLR-3623

[asfimport]

Jörn Kottmann (migrated from JIRA)

The first release is now out. I guess you will use maven for dependency management, you can find here how to add the released version as a dependency:
http://incubator.apache.org/opennlp/maven-dependency.html

[asfimport]

Lance Norskog (migrated from JIRA)

This is a patch for the trunk (as of a few days ago) that supplies the OpenNLP Sentence Detector, Tokenizer, Parts-of-Speech, Chunking and Named Entity Recognition tools.

This has nothing to do with the code mentioned above.

[asfimport]

Lance Norskog (migrated from JIRA)

Notes for a Wiki page:

OpenNLP Integration

What is the integration? The first integration is a Tokenizer and three Filters.

• The OpenNLPTokenizer uses the OpenNLP SentenceDetector and Tokenizer tools instead of the standard Lucene Tokenizers. This requires statistical model files. One quirk of these is that all punctuation is maintained.
• The OpenNLPFilter implements Parts-of-Speech tagging, Chunking (finding noun/verb phrases), and Named Entity Recognition (tagging people, place names etc.). This filter will add all tags as payload attributes to the tokens.
• The FilterPayloadsFilter removes tokens by checking the payloads. Given a list of payloads, it will either keep only tokens with one of those payloads, or remove only matching tokens and keep the rest. (This filter maintains position increments correctly.)
• The StripPayloadsFilter removes payloads from Tokens.

How do I get going?

• pull the latest trunk
• apply the patch
• download these models to contrib/opennlp/src/test-* files/opennlp/solr/conf/opennlp/
  • http://opennlp.sourceforge.net/models-1.5/
  • Everything that starts with 'en'
• download the OpenNLP distribution from http://opennlp.apache.org/cgi-bin/download.cgi
  • Currently it is apache-opennlp-1.5.2-incubating-bin.tar.gz
• unpack this and copy the jar files from lib/ to
  solr/contrib/opennlp/lib

Now, go to trunk-dir/solr and run 'ant test-contrib'. It compiles against the libraries and uses the model files.
Next, run 'ant example', cd to the example directory and run 'java -Dsolr.solr.home=opennlp -jar start.jar'
You now should start without any Exceptions. At this point, go to the Schema analyzer, pick the 'text_opennlp_pos' field type, and post a sentence or two to the analyzer. You should get text tokenized with payloads. Unfortunately, the analysis page shows them as bytes instead of text. If you would like this, then go vote on https://issues.apache.org/jira/browse/SOLR-3493.

[asfimport]

Lance Norskog (migrated from JIRA)

About the build-

1. This should be a Lucene module. I got lost trying to make the build work copying jars around, so it will ended up in Solr/contrib.
2. Downloading the jars. I don't know how to put together license validation with the OpenNLP Maven build. I think it takes some upgrading in the OpenNLP project.
3. Why download the models from a separate place? The models are not Apache licensed. They are binaries derived from GNU- and otherwise licensed training data. The OpenNLP people archived them on Sourceforge.

[asfimport]

Lance Norskog (migrated from JIRA)

I consider the code and feature set mostly cooked as a first release. The toolkit as is lets you do two things:

1. Do named entity recognition and filter out names for an autosuggest dictionary
2. Pick nouns and verbs out of text and only index those. This gives you a field with a smaller more focused set of terms. MoreLikeThis might work better.

Please review it for design, bugs, code nits, whatever.

[asfimport]

Lance Norskog (migrated from JIRA)

An explanation about the OpenNLPUtil factory class: the statistical models are several megabytes apiece. This class loads them and caches them by file name. It does not reload them across commits.

The models are immutable objects. The factory class creates another object that consults the model. There is one of these for each field analysis.

The models are large enough that if the different unit tests load them all at once, it needs more than the default ram. Therefore, the unit tests unload all models between tests, and only run single-threaded.

[asfimport]

Lance Norskog (migrated from JIRA)

License-ready.
Ivy-ready.
OpenNLP libraries available through Ivy.
You still have to download jwnl-1.3.3 from http://sourceforge.net/projects/jwordnet/files/

And of course download the model files. But this is committable to the Solr side.

[asfimport]

Grant Ingersoll (@gsingers) (migrated from JIRA)

Very cool Lance. The models are indeed tricky and I wonder how we can properly hook them into the tests, if at all. I wonder how hard it would be to create much smaller ones based on training just a few things.

[asfimport]

Tommaso Teofili (@tteofili) (migrated from JIRA)

I wonder how hard it would be to create much smaller ones based on training just a few things.

there was the idea of using the OpenNLP CorpusServer with some wikinews articles to train them (back to OPENNLP-385)

[asfimport]

Jörn Kottmann (migrated from JIRA)

I am using this mentioned Corpus Server together with the Apache UIMA Cas Editor for labeling projects. If someone wants to set something up to label data we (OpenNLP people) are happy to help with that!

[asfimport]

Grant Ingersoll (@gsingers) (migrated from JIRA)

Cool!

I think if we could just get a very small model that can be checked in and used for testing purposes, that is all that would be needed. We don't really need to test OpenNLP, we just need to test that the code properly interfaces with OpenNLP, so a really small model should be fine.

[asfimport]

Grant Ingersoll (@gsingers) (migrated from JIRA)

This really should just be a part of the analysis modules (with the exception of the Solr example parts). I don't know exactly how we are handling Solr examples anymore, but I seem to recall the general consensus was to not proliferate them. Can we just expose the functionality in the main one?

I'll update the patch to move this to the module for starters. Not sure on what to do w/ the example part.

[asfimport]

Jörn Kottmann (migrated from JIRA)

For a test you can run OpenNLP just over a piece of training data, even when trained on a tiny amount of data this will give good results. It does not test OpenNLP, but is sufficient for the desired interface testing.

[asfimport]

Lance Norskog (migrated from JIRA)

This really should just be a part of the analysis modules (with the exception of the Solr example parts). I don't know exactly how we are handling Solr examples anymore, but I seem to recall the general consensus was to not proliferate them. Can we just expose the functionality in the main one?

A lot of Solr/Lucene features are only demoed in solrconfig/schema unit test files (DIH for example). That is fine.

The models are indeed tricky and I wonder how we can properly hook them into the tests, if at all.

D'oh! Forgot about that. If we have tagged data in the project, it helps show the other parts of an NLP suite. It's hard to get a full picture of the jigsaw puzzle if you don't know NLP software.

[asfimport]

Lance Norskog (migrated from JIRA)

Wiki page is up! http://wiki.apache.org/solr/OpenNLP

Also, the Solr fancy toolkits had no links from the Solr front page, so I added 'Advanced Tools' with links to UIMA and this.