Fine-tuning bug fix

[eu9ene]

• It causes fewer problems when teacher training on a parallel dataset happens in a separate directory. Model weights are initialized using --pretrained-model Marian parameter.
  fixes Teacher does not continue training if training on augmented data was early stopped #49

• Fixes a new bug with student fine-tuning, there was no weights initialization (it was lost during refactoring).

• Fixed usage of a pretrained backward model + vocab

• fixes Wrong tcol when cleaning with Bicleaner #56

[XapaJIaMnu]

As far as I understand this wouldn't affect the behaviour when training crashed and was resumed, correct? That would still continue with loading the optimiser parameters. (TBH I haven't tested if this case even works).

[XapaJIaMnu]

From reading the source, I don't understand what teacher_parallel_dir should contain. What is a parallel teacher model?

[eu9ene]

Teacher all - the model is trained on all available data.
Teacher parallel - optional model to fine-tune on parallel data only if the data was augmented with back translations.

Will it be easier to understand if I rename them to teacher and teacher-finetuned?

[XapaJIaMnu]

Yes that would be easier to understand, but this is a very minor point.

[eu9ene]

As far as I understand this wouldn't affect the behaviour when training crashed and was resumed, correct? That would still continue with loading the optimiser parameters. (TBH I haven't tested if this case even works).

This will work because I removed protection from the output file model.npz.best-chrf.npz. It means snakemake will delete it when the job is stopped or crushed, so it will rerun training next time to get this file. Since it will happen in the same directory and model.npz and optimizer progress files are there, it will continue training.

However, this is an irregular situation and not desirable. The pipeline is designed to work end to end without interruptions.