Incorporating Word Vectors rather than using an Embedding class

[vindiesel]

I am solving an NLP task and I am trying to model it directly as a sequence using different RNN flavors. How can I use my own Word Vectors rather than using an instance of layers.embeddings.Embedding?

[sergeyf]

UPDATED NOV 13, 2017

You have to pass a weight matrix to the Embedding layer. Here is an example:

Let's say index_dict is a dictionary that maps all the words in your dictionary to indices from 1 to n_symbols (0 is reserved for the masking).

So, an example index_dict is the following:

{
 'yellow': 1,
 'four': 2,
 'woods': 3,
 'ornate': 31,
 'woody': 5,
 'cyprus': 6,
 'marching': 7,
 'canes': 8,
 'caned': 9,
 'hermann': 10,
 'lord': 11,
 'meadows': 12,
 'shaving': 13,
 'swivel': 14
...
}

And you also have a dictionary called word_vectors that maps words to vectors like so:

{
 'yellow': array([0.1,0.5,...,0.7]),
 'four': array([0.2,1.2,...,0.9]),
...
}

The following code should do what you want

# assemble the embedding_weights in one numpy array
vocab_dim = 300 # dimensionality of your word vectors
n_symbols = len(index_dict) + 1 # adding 1 to account for 0th index (for masking)
embedding_weights = np.zeros((n_symbols, vocab_dim))
for word,index in index_dict.items():
    embedding_weights[index, :] = word_vectors[word]

# define inputs here
embedding_layer = Embedding(output_dim=vocab_dim, input_dim=n_symbols, trainable=True)
embedding_layer.build((None,)) # if you don't do this, the next step won't work
embedding_layer.set_weights([embedding_weights])

embedded = embedding_layer(input_layer)
# ... continue model definition here

Note that this kind of setup will result in your embeddings being trained from their initial point! If you want them fixed, then you have to set trainable=False.

[farizrahman4u]

No need of skipping the embedding layer.. setting word vectors as the initial weights of embedding layer is a valid approach. The word vectors will get fine tuned for the specific NLP task during training.

[dandxy89]

Has anybody else attempted to embedded the word vectors into a model?

I've managed to create the model however Im not able to achieve a worthwhile level of accuracy yet. I've used the "20 newsgroups dataset" from scikit-learn to test this model, with my own w2v vectors. The best accuracy I've achieved so far is 28%, over 5 epochs, which is not great (scikit script best - 85%). I intend to continue experimenting with the network configuration (inner dimensions and epochs initially). I do suspect that the number of dimentions is too high for such a small dataset (1000 samples).

Will update again if the results improve

[viksit]

From what I've seen, training your own vectors on top of a custom dataset has given me much better accuracy within that domain.

That said - any updates on this?

[farizrahman4u]

@viksit

There are 3 approaches:

• Learn embedding from scratch - simply add an Embedding layer to your model
• Fine tune learned embeddings - this involves setting word2vec / GloVe vectors as your Embedding layer's weights.
• Use word word2vec / Glove word vectors as inputs to your model, instead of one-hot encoding.

The third one is the best option(Assuming the word vectors were obtained from the same domain as the inputs to your models. For e.g, if you are doing sentiment analysis on tweets, you should use GloVe vectors trained on tweets).

In the first option, everything has to be learned from scratch. You dont need it unless you have a rare scenario. The second one is good, but, your model will be unnecessarily big with all the word vectors for words that are not frequently used.

[viksit]

@farizrahman4u agreed on those counts. The domains are a bit more specific and I have a lot more luck with option (2) than with (1) or (3) so far. An easy way to address the size problem with (2) is to prune out the vocabulary itself to the top k words.

[XuesongYang]

@farizrahman4u Thanks for sharing the ideas. I have a question on your first approach.

Learning embeddings from scratch: each word in the dictionary is represented as one-hot vector, and then this vector is embedded as a continuous vector after applying embedding layer. Is that right?

[farizrahman4u]

@MagicYoung Yes.

[dandxy89]

To follow up from @sergeyf suggestions... @viksit @MagicYoung @vindiesel

Take a look at my attempt (Script and Data) - it uses Gensim. This could easily all be done in Keras using the skipgram of course to remove dependencies...

[sergeyf]

Looks cool! Are the results to your liking?

[dandxy89]

Results after 2 epochs:
Validation Accuracy: 0.8485
Loss: 0.3442

[liyi193328]

@dandxy89
Wonderful examples for me.I'll try it in my context. Thanks.
And if I want to feed pre-trained word2vec to lstm directly, how to handle different sequence lengths?
My trying is: setting maxlen=200(every sequence) and word2vec dim=600, if some sequence's length is 100, then the first 100 rows([0:100)) has float numbers(every row is a word vector), then [100:200) rows is padding with zeors, which means the remaining rows is all zeros. But after doing that, I get loss NaN, which confuses me for a long time. like the issue #1360 , what I can do then?
Thanks.

[viksit]

@liyi193328 are you using keras' sequence.pad_sequences(myseq, maxlen=maxlen)?

Looks like your padding is to the right of the vectors where as it should be to the left.

from keras.preprocessing import sequence
In [69]: sequence.pad_sequences([[1,2], [1,2,3]], maxlen=10)
Out[69]:
array([[0, 0, 0, 0, 0, 0, 0, 0, 1, 2],
       [0, 0, 0, 0, 0, 0, 0, 1, 2, 3]], dtype=int32)

Secondly, you should be using categorical_crossentropy as your loss as opposed to mean_squared_error. See #321