which attention architecture is used in NER?

[omerarshad]

I want to understand how you used attention in NEr task, any paper or article which explains this? Thanks

[qq547276542]

According to the README, attention mechanism not suitable for NER task：
The variant modules include Stack Bidirectional RNN (multi-layers), Multi-RNN Cells (multi-layers), Lurong/Bahdanau Attention Mechanism, Self-attention Mechanism, Residual Connection, Layer Normalization and so on. However, these modifications did not improve (sometime even worse than basement model) the performance significantly (F1 score improves >= 1.5). It's easy to apply those variants and train by modifying the config settings in train_conll_ner_blstm_cnn_crf.py.

[qq547276542]

In my own experiment, attention mechanism really not work, but Layer Normalization can improve the robustness of the model

[omerarshad]

well in my experiments attention only models achieve comparable results to LSTM, even got better than LSTM with very less training time

[qq547276542]

well in my experiments attention only models achieve comparable results to LSTM, even got better than LSTM with very less training time

Maybe there are some problems with my experiment. I have tried BLSTM+SelfAttention+CRF，the effect is not as good as BLSTM+CRF. The structure of your model is SelfAttention+CRF, not LSTM? I want to give it a try.

[omerarshad]

yes, structure of my model is attention+crf only

[qq547276542]

yes, structure of my model is attention+crf only

do you have the relevant code, can I refer to it?

[VioletJKI]

I do not use crf, and get a best result.

@MingLunHan excuse me, what is your model's architecture? bilstm + attention only?