Bi-LSTM+CRF error

[ZhixiuYe]

hello, I try to run BiLSTM_CRF example. I add one example to training_data and get
"""
training_data = [(
"the wall reported apple corporation made money".split(),
"B I O B I O O".split()
), (
"georgia tech is a university in georgia".split(),
"B I O O O O B".split()
), ("China".split(), 'B'.split())
]
"""
And then, when I run
"""
precheck_sent = prepare_sequence(training_data[2][0], word_to_ix)
print model(precheck_sent)
"""
I get an output 3, which means Start_TAG.

And then I try to change the code of function _viterbi_decode
I change
"""
terminal_var = forward_var + self.transitions[self.tag_to_ix[STOP_TAG]]
best_tag_id = argmax(terminal_var)
path_score = terminal_var[0][best_tag_id]
best_path = [best_tag_id]
"""
to
"""
terminal_var = forward_var + self.transitions[self.tag_to_ix[STOP_TAG]]
temp = terminal_var.data
temp[0][self.tag_to_ix[STOP_TAG]] = -10000
temp[0][self.tag_to_ix[START_TAG]] = -10000
temp = torch.autograd.Variable(temp)
best_tag_id = argmax(temp)
path_score = temp[0][best_tag_id]
best_path = [best_tag_id]`
"""
then I get a correct output 1, which means 'B'.

I wonder to know why it produces a completely wrong answer, START_TAG. Thank you!

[fsonntag]

Hey Zhixiu,

I discovered similar behavior. If I understood the problem correctly, the Viterbi decoder basically still takes a transition to the start or stop state into account, which it shouldn't.
I think a little bit more elegant fix would look like this:

terminal_var = (forward_var + self.transitions[self.tag_to_ix[STOP_TAG]]).clone()
terminal_var[0, self.tag_to_ix[STOP_TAG]] = -10000.
terminal_var[0, self.tag_to_ix[START_TAG]] = -10000.
best_tag_id = argmax(terminal_var)

You can open a PR if you want, otherwise I'll open one.

I think another possible fix would be to change _get_lstm_features and set the features of of START_TAG and STOP_TAG to a low value:

lstm_feats = self.hidden2tag(lstm_out)
lstm_feats[:, self.tag_to_ix[START_TAG]] = -10000.
lstm_feats[:, self.tag_to_ix[STOP_TAG]] = -10000.
return lstm_feats

@rguthrie3 What do you think about this issue? I guess since it's also part in the official PyTorch tutorials, this should as people probably rely on that code.

[rguthrie3]

Hi, I think I see what you are saying. My code does not enforce in any way that START and STOP cannot be part of the best path. E.g START START B STOP would be a valid path for a sentence of length 2, if that makes sense. This is probably wrong to do, but with enough training such paths should never get decoded since transitioning to START should have a low weight, and transitioning from STOP should have a low weight. If you want to enforce this constraint plainly, you could intialize the transition matrix from the start to give those transitions very low scores.

This is what I think it is if I understand your question. I can't run this at work but I will look into it when I can. Are you training the model and still getting that output (I.e it decodes sequences with START and STOP) or is that the output before training?

[fsonntag]

Yeah, you are right! I ran into this issue as I while evaluating before training, so that completely matches with your guess.

Actually the first fix does not seem to prevent that, but my second one does.
So yeah, it's your code and it depends on you if you want to "fix" this (basically it would not be a fix, just different behavior).

[rguthrie3]

Here is the fix, which I will add later tonight. In init, after defining self.transitions, add

self.transitions.data[tag_to_ix[START_TAG], :] = -10000
self.transitions.data[:, tag_to_ix[STOP_TAG]] = -10000

The first line makes it never transition to START, the second makes it never transition from STOP. This will enforce those constraints a priori without having to depend on the model learning that those two states should never be transitioned to / from respectively.

Note like I said I can't run this right now, so I might have gotten the syntax slightly wrong above, but I hope the intent is clear (set the whole START row to -10000 and STOP column to -10000)

[fsonntag]

Great, that's actually more elegant! Thanks for the feedback and thanks in general for the whole work you did :)

[rguthrie3]

I just pushed the fix so I will close this issue. Also, I appreciate the nice comment and am happy the tutorial was helpful for you.