Neural Sequence Labeling

A TensorFlow implementation of Neural Sequence Labeling model, which is able to tackle Part-of-Speech (POS) Tagging, Chunking and Named Entity Recognition (NER) tasks. This repository is inspired after reading the following papers about sequence labeling:

• Named Entity Recognition with Bidirectional LSTM-CNNs
• Neural Models for Sequence Chunking
• End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF
• Neural Architectures for Named Entity Recognition
• Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks
• Part-of-Speech Tagging from 97% to 100%: Is It Time for Some Linguistics?
• Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Recurrent Neural Network

This model follows the structure of RNNs + CRF + Chars Embeddings (RNN/CNN), and several variant modules are available, like single bi-RNN, stacked bi-RNN and densely connected bi-RNN as well as experiments on introducing the attention mechanism (dot attention, Bahdanau attention and etc). RNN cell can use LSTM or GRU, and etc.

The performance (F1 Score) of this neural sequence labeling model on NER task is 90.00~91.00, which is near to the state-of-the-art performance (highest reported: F1 Score = 91.21, ref. link).

Task

Given a sentence, give a tag to each word. For example, a classical application is Part-of-Speech (POS) Tagging

EU  rejects German call to boycott British lamb . 
NNP VBZ     JJ     NN   TO VB      JJ      NN   . 

For Chunking task

EU   rejects German call to   boycott British lamb . 
B-NP B-VP    B-NP   I-NP B-VP I-VP    B-NP    I-NP O 

For Named Entity Recognition (NER) task

Stanford University located at California 
B-ORG    I-ORG      O       O  B-LOC      

Usage

Note: CoNLL 2003 English Dataset is obtained from anago/data/conll2003/en/, which is already placed in data/conll2003/en folder.

To download pre-trained word embeddings and pre-process data, run

# download embeddings (GloVe 6B and 840B word embeddings and GloVe 840B char embeddings)
$ ./download.sh
# pre-process data
$ python3 build_data.py 

Hyperparameters are stored in config.py (change to tasks here, train POS, Chunking or NER). To run the model, run

# training and testing model
$ python3 train_test.py
# if pretrained model exists in ckpt folder, restore and test
$ python3 restore_test.py

If training is processing normally, the log information will display as following

2018-01-10 13:51:26,225:INFO: Start training...
2018-01-10 13:51:27,771:INFO: Epoch  1/15:
2018-01-10 14:02:52,775:INFO: Testing model over DEVELOPMENT dataset
2018-01-10 14:03:57,189:INFO: accuracy: 97.76 -- f1 score: 88.30
2018-01-10 14:04:00,044:INFO:  -- new best score: 88.29796158040433
2018-01-10 14:04:00,045:INFO: Epoch  2/15:
2018-01-10 14:15:30,468:INFO: Testing model over DEVELOPMENT dataset
2018-01-10 14:16:36,593:INFO: accuracy: 98.24 -- f1 score: 90.86
2018-01-10 14:16:40,383:INFO:  -- new best score: 90.85699839892138
...
2018-01-10 16:42:36,101:INFO: Epoch 15/15:
2018-01-10 16:52:51,042:INFO: Testing model over DEVELOPMENT dataset
2018-01-10 16:53:43,379:INFO: accuracy: 98.86 -- f1 score: 94.76
2018-01-10 16:53:45,933:INFO:  -- new best score: 94.10280137965844
2018-01-10 16:53:45,933:INFO: Training process done...
2018-01-10 16:53:45,933:INFO: Testing model over TEST dataset
2018-01-10 16:54:34,322:INFO: accuracy: 97.93 -- f1 score: 90.66

After training and testing, a interact module is activated to allow user to manually test some input text

input> Stanford University located at California
Stanford University located at California 
B-ORG    I-ORG      O       O  B-LOC      
input> China is one of the biggest country in the World
China is one of the biggest country in the World 
B-LOC O  O   O  O   O       O       O  O   B-MISC

Resources

Embeddings

• GloVe Embeddings (6B, 42B, 840B)
• minimaxir/char-embeddings
• GloVe 840B 300 dimension Char Embeddings

GitHub Repositories of Sequence Labeling (Ref.)

• LopezGG/NN_NER_tensorFlow, implementation of End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF
• UKPLab/emnlp2017-bilstm-cnn-crf, implementation of Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging and Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks
• ThanhChinhBK/Ner-BiLSTM-CNNs
• clab/stack-lstm-ner, implementation of Neural Architectures for Named Entity Recognition
• Hironsan/anago
• guillaumegenthial/sequence_tagging, blog
• glample/tagger, implementation of Neural Architectures for Named Entity Recognition
• marekrei/sequence-labeler

Others

• models/tutorials/rnn/quickdraw/train_model.py
• 使用深度学习进行中文自然语言处理之序列标注
• Building a large annotated corpus of English: the Penn Treebank
• PennTree Bank POS tags
• randomrandom/deep-atrous-ner