Preprocessing: faster build vocab + multiple weighted datasets

[francoishernandez]

The current preprocessing works in the following manner:

• build train (from a single src and tgt pair of files) and valid shards;
• build vocabulary from train shards.

This means train shards are re-loaded after having been dumped.
We have in mind to simplify this by creating the vocabulary along with the train shards.

For now I've just taken the logic from onmt.inputters.inputter.build_vocab and inserted the necessary bits in preprocess.build_save_dataset.
It might not be the cleanest way to do, but it's a start.

Tested on toy text / speech / image datasets, and it seems to work fine.

Glad to have some feedback on a cleaner way to refactor preprocessing codepath(s) @guillaumekln @flauted @bpopeters .

Next ideas would be, either in this PR or in a following one:

• allow several corpora with different weights (upsampling);
• allow multiple threads while creating shards.

PS: @flauted while testing stuff for this PR, I stumbled upon a maybe unwanted behaviour introduced here. This, in combination with some -src_words_min_frequency or tgt_words_min_frequency option may lead to removing some of the last tokens of the vocab.

[flauted]

About the PS: Good catch. It's older than that though, see here. I haven't really used the text file vocab feature, but I'm pretty sure it's just a list of (theoretically unique) words. So the counts/freqs are lost. The validation should probably check for non-default min freq/count type options if there's a source vocab path. I don't know if that's in the scope of this PR or should be done separately, though.

[francoishernandez]

I'm extending the scope of this PR, adding the possibility to pass several train corpora when preprocessing, and give them specific weights when training.

Preprocessing

I introduce -train_ids which is a list of IDs that will be given to the preprocessed shards.
E.g. we have two corpora : parallel.en and parallel.de + from_backtranslation.en from_backtranslation.de, we can pass

...
-train_src parallel.en from_backtranslation.en \
-train_tgt parallel.de from_backtranslation.de \
-train_ids A B \
-save_data my_data \
...

and it will dump my_data.train_A.X.pt based on parallel.en//parallel.de and my_data.train_B.X.pt based on from_backtranslation.en//from_backtranslation.de.

Training

I introduce -data_ids based on the same principle as above, as well as -data_weights, which is the list of the weight each corpus should have.
E.g.

...
-data my_data \
-data_ids A B \
-data_weights 1 7 \
...

will mean that we'll look for my_data.train_A.*.pt and my_data.train_B.*.pt, and that when building batches, we'll take 1 example from corpus A, then 7 examples from corpus B, and so on.

For this purpose, I created the MultipleDatasetIterator class in onmt.inputters.inputter which takes a list of DatasetLazyIter and the respective list of weights to yield the batch according to the rule described above.
The idea is to spawn an instance of DatasetLazyIter with batch_size 1, and new parameter yield_raw_example that will yield raw examples from each corpus one at a time.

[vince62s]

beyond the few comments, it does not work. I think the vocab built this way is wrong.
tried a quick training and does not work properly.

[vince62s]

Main issue was my fault, seems to work fine now.
Will merge.