How to use for word2vec training?

[Santosh-Gupta]

Sorry for the newbie question, but I'm having a bit of trouble in trying to use the library.

Specifically what I'm trying to do is embedding training, like word2vec. So I am trying to setup and embedding matrix, loss, and optimization using Adam.

Any pointers would be greatly appreciated.

[ddbourgin]

Don't be sorry - this question is very justified -- there is almost no usage documentation right now!

First, the bad news: at the moment, implementing word2vec will require a little bit of extra leg-work on your end. In particular, you'll need to implement either a negative sampler/noise contrastive estimation loss or a hierarchical softmax loss. For the latter, you could use the numpy_ml.preprocessing.nlp.HuffmanEncoder module.

Now, the good news: I'm actively working on writing an NCE loss object, and hope to push it ASAP. I'll also probably include a convenience Embedding layer to make embedding lookups a bit faster. I will update this thread when it has been pushed.

Ultimately, once these two components are in place, you should be able to write a relatively straightforward model. To see what a model object might look like, you can look at some examples in either the numpy_ml.neural_nets.modules or numpy_ml.neural_nets.models directories.

[ddbourgin]

Finally, one last caveat - if you're interested in training a non-toy word embedding model, I'd highly recommend using a library like keras, since it will make use of performance-optimized implementations for each model component. The code in this repo is meant to be clear and straightforward, but this often comes at the expense of efficiency!

[Santosh-Gupta]

Thanks, looking forward to it!

The reason why I am interested in doing it in Numpy is that Keras/Tensorflow isn't very great for sparse training. My use case is training 9 figures of embeddings. It eats up a lot of memory to have all those embeddings loaded into memory at the same time, and it's not necessary since only a small fraction is trained during each update step.

So I was thinking of using your library and altering it so that it saves the unused weights to disk, and only loads them when they are trained or about to be trained.

[ddbourgin]

So I was thinking of using your library and altering it so that it saves the unused weights to disk, and only loads them when they are trained or about to be trained.

That sounds like a good idea! If you end up implementing this, definitely consider submitting a PR :) I think this could be quite useful for a number of different model components, including the sparse evolutionary training layer (which currently uses dense matrices 😬).

In the meantime, you might look into the magnitude package (I haven't used it myself, but it seems potentially relevant).

[Santosh-Gupta]

It looks like this is exactly what I was looking for. I'm not familiar with a lot of computer science terms, but it uses sqlite as the datastore, so I'm guessing it does what I am looking for.

Edit: reading the paper

"Magnitude queries return almost instantly and
are memory efficient. It uses lazy loading directly from disk, instead of having to load the entire model into memory"

Wow! Thanks for this recommendation!!!

[Santosh-Gupta]

Looks like you can't use it for training, oh well

plasticityai/magnitude#32

[Santosh-Gupta]

But looking at the code gives me a strong idea of how I could implement it in this library, by using SQLite database as the vector store, and just copying the values back and forth between that and the numpy embedding array.

[ddbourgin]

Heya @Santosh-Gupta - I've just pushed a preliminary version of an NCELoss and word2vec model here and here.

Unfortunately, I suspect that if you're going to use the models for any sizeable dataset you'll have to do some performance modifications first. Let me know if you decide to try it out / have any questions in the meantime!

[Santosh-Gupta]

Sounds good!