Add phrase_table translation argument

[ymoslem]

If phrase_table is provided (with replace_unk), it will look up the identified source token and give the corresponding target token. If it is not provided (or the identified source token does not exist in the table), then it will copy the source token. Tested with both translate.py and server.py (with conf.json).

The default behaviour of the -replace_unk option is substituting <unk> (for an unknown word) with the source word that has the highest attention weight. Adding the option -phrase_table as well, it will look up in the phrase table file for a possible translation instead. If a valid replacement is not found, only then the source token will be copied.

The phrase table is a file with one translation per line in the format:
source|||target
Where source and target are case sensitive and single tokens.

Example with translate.py:
python3 OpenNMT-py/translate.py -model available_models/my.model_step_100000.pt -src source.txt -output prep.txt -replace_unk -phrase_table phrase-table.txt

Example with server.py:
python3 OpenNMT-py/server.py --ip "0.0.0.0" --port 5000 --url_root "/translator" --config available_models/conf.json

curl -i -X POST -H "Content-Type: application/json" -d '[{"src": "this is a test for model 100", "id": 100}]' http://127.0.0.1:5000/translator/translate

... where conf.json is:

{
    "models_root": "/home/available_models",
    "models": [
        {   
            "id": 100,
            "model": "my.model_step_100000.pt",
            "timeout": 600,
            "on_timeout": "to_cpu",
            "load": true,
            "opt": {
                "beam_size": 1,
                "replace_unk": true,
                "phrase_table": "/home/available_models/phrase-table.txt"
            }
        }  
    ]   
}

[ymoslem]

Fixed flake8 issues.

[ymoslem]

@vince62s @srush Could you please check. Thanks!

[vince62s]

Thanks for this contribution!
My first comments, but @francoishernandez will comment more when he finds the time.

This is single token replacement. Nowadays, most people use BPE/Subwords, so it would be even better to use multiple tokens replacement.

Also, I think (but @francoishernandez @guillaumekln will confirm) this would work for LSTM only. For the transformer it might be better to use a guided alignement methid as in openNMT-tf.

[ymoslem]

@vince62s Thanks, Vincent! I agree with you that it is much better to be multiple tokens replacement. I just wanted to start by imitating the current behaviour of phrase_table in the Lua version as I thought it is useful is some cases.

So how do you think I should proceed? Do you think it is worth adjusting the code to work on multiple tokens even if it will not work with the transformer for now? (I have not tested this either)

Thanks again for your time and insights!

[vince62s]

It's ok, we'll merge this one as is.
Just bringing attention to users it won't work with the Transformer.

[ymoslem]

@vince62s Thanks, Vincent, for merging it! If I am going to work on multiple tokens replacement:

1- Should it be a new argument like phrase_table_multiple or the same argument phrase_table?
2- How many maximum ngrams should it work on; i.e. what is the maximum sequential source words can be considered as a phrase?
3- Should it work on the unknowns only or on the source before machine translation?

Thanks!