Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fine-tuning bug fix #51

Merged
merged 8 commits into from
Jan 24, 2022
Merged

Fine-tuning bug fix #51

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
b0e2e38
Continue teacher training in a separate directory
eu9ene Jan 14, 2022
5fbed78
Fix usage of pretrained backward model
eu9ene Jan 15, 2022
d822ccf
Make it possible to use pretrained vocab
eu9ene Jan 15, 2022
30d6271
Fix bicleaner
eu9ene Jan 21, 2022
ca2378b
Fix bicleaner
eu9ene Jan 22, 2022
55e94c2
Rename teacher models
eu9ene Jan 22, 2022
aea8645
Rename teacher models
eu9ene Jan 22, 2022
9abc3c5
Reduce bicleaner threshold for open subtitles
eu9ene Jan 22, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Rename teacher models
  • Loading branch information
@eu9ene
eu9ene committed Jan 22, 2022
commit 55e94c24cdf82701c0abc3f8ba38cd3e395144ef
12 changes: 6 additions & 6 deletions Snakefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,8 +92,8 @@ align_dir = f"{data_dir}/alignment"

# models
models_dir = f"{data_root_dir}/models/{src}-{trg}/{experiment}"
teacher_all_dir = f"{models_dir}/teacher-all"
teacher_parallel_dir = f"{models_dir}/teacher-parallel"
teacher_all_dir = f"{models_dir}/teacher"
teacher_parallel_dir = f"{models_dir}/teacher-finetuned"
student_dir = f"{models_dir}/student"
student_finetuned_dir = f"{models_dir}/student-finetuned"
speed_dir = f"{models_dir}/speed"
Expand Down Expand Up @@ -122,7 +122,7 @@ results = [f'{exported_dir}/model.{src}{trg}.intgemm.alphas.bin.gz',
f'{exported_dir}/lex.50.50.{src}{trg}.s2t.bin.gz',
f'{exported_dir}/vocab.{src}{trg}.spm.gz',
f'{experiment_dir}/config.yml',
*expand(f'{eval_res_dir}/teacher-all{{ens}}/{{dataset}}.metrics',ens=ensemble, dataset=eval_datasets),
*expand(f'{eval_res_dir}/teacher{{ens}}/{{dataset}}.metrics',ens=ensemble, dataset=eval_datasets),
*expand(f'{eval_student_dir}/{{dataset}}.metrics', dataset=eval_datasets),
*expand(f'{eval_student_finetuned_dir}/{{dataset}}.metrics', dataset=eval_datasets),
*expand(f'{eval_speed_dir}/{{dataset}}.metrics', dataset=eval_datasets)
Expand Down Expand Up @@ -166,7 +166,7 @@ if mono_trg_datasets:
teacher_corpus = f'{augmented}/corpus'
augment_corpus = True
final_teacher_dir = teacher_parallel_dir
results.extend(expand(f'{eval_res_dir}/teacher-parallel{{ens}}/{{dataset}}.metrics',ens=ensemble, dataset=eval_datasets))
results.extend(expand(f'{eval_res_dir}/teacher-finetuned{{ens}}/{{dataset}}.metrics',ens=ensemble, dataset=eval_datasets))
else:
augment_corpus = False
final_teacher_dir = teacher_all_dir
Expand Down Expand Up @@ -475,7 +475,7 @@ rule teacher_all:
bin=trainer, vocab=vocab_path
output: model=f'{teacher_all_dir}{{ens}}/{best_model}'
params: prefix_train=teacher_corpus, prefix_test=f"{original}/devset", dir=directory(f'{teacher_all_dir}{{ens}}'),
args=get_args("training-teacher-all")
args=get_args("training-teacher")
shell: '''bash pipeline/train/train.sh \
teacher train {src} {trg} "{params.prefix_train}" "{params.prefix_test}" "{params.dir}" \
"{input.vocab}" {params.args} >> {log} 2>&1'''
Expand All @@ -494,7 +494,7 @@ if augment_corpus:
output: model=f'{teacher_parallel_dir}{{ens}}/{best_model}'
params: prefix_train=clean_corpus_prefix, prefix_test=f"{original}/devset",
dir=directory(f'{teacher_parallel_dir}{{ens}}'),
args=get_args("training-teacher-parallel")
args=get_args("training-teacher-finetune")
shell: '''bash pipeline/train/train.sh \
teacher train {src} {trg} "{params.prefix_train}" "{params.prefix_test}" "{params.dir}" \
"{input.vocab}" --pretrained-model "{input.model}" {params.args} >> {log} 2>&1'''
Expand Down
2 changes: 1 addition & 1 deletion configs/config.prod.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ marian-args:
training-backward:
# change based on available training data
after: 10e
training-teacher-all:
training-teacher:
# remove for low resource languages or if training without augmentation
after: 2e
# these configs override pipeline/translate/decoder.yml
Expand Down
4 changes: 2 additions & 2 deletions configs/config.test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,9 +36,9 @@ experiment:
marian-args:
training-backward:
after: 1000u
training-teacher-all:
training-teacher:
after: 1000u
training-teacher-parallel:
training-teacher-finetune:
after: 2000u
training-student:
after: 1000u
Expand Down