Vocabulary size cannot be specified for word features

[noe]

When using word features, it is useful to be able to limit the vocabulary size, especially when using lemmas as word features, as the number of different lemmas can be very large.

However, preprocess.py does not support providing a list of integers, unlike the Lua version.

I understand that it should be possible to get the same effect by extracting the feature vocabulary externally (pruning the vocabulary by frequency) and then supplying the vocabulary to preprocess.py with the parameter --features_vocabs_prefix.

I just realized that --features_vocabs_prefix is completely ignored. Therefore, I understand that there is no way to control the feature vocabulary size; is that correct?

[vince62s]

Hi @noe
yes, this option was inserted in the opts.py file just to mimic the Lua version but it was never implemented.
Honestly, features are not on my priority list and @bpopeters seems to have taken some distance from the project at the moment.
If you feel comfortable to implement, PR are always welcome.

[eeric]

make OpenNMT project audio list and sentences text list, as following:
https://github.com/eeric/OpenNMT-py-make-tgt-train.txt/blob/master/openNMT_make_label.py

[noe]

Just for reference, issue #1534 (and associated PR #1564) is related to this, as it allows setting the embedding sizes. Nevertheless, it does not handle different vocabulary sizes.

[vince62s]

@vikrant97

[vikrant97]

@vince62s Can you point me out where does the Vocab pruning happens in the preprocessing script?
I remember earlier, there used to be function called makeVocabulary which contains that part.
Is it now shifted to using torchtext.vocab?
Can you please some light on this?

[vince62s]

is this what you are looking for ?
https://github.com/OpenNMT/OpenNMT-py/blob/master/onmt/inputters/inputter.py#L317