[meta] General translation quality improvements

[eu9ene]

This is a meta issue for brainstorming ideas and tracking issues to improve the translation quality in general:

General improvements

Preview Give feedback

1. Made-up words in translations #843
   quality +1
2. Re-train existing models with larger datasets #178
   quality +1
3. Investigate distillation quality gap #231
   quality +1
   
4. max-length and max-length-crop considered harmful firefox-translations-training#89
   bug quality +2
5. Investigate performance, memory usage and translation quality without using a lexical shortlist #172
   quality +1
6. Investigate using larger student models firefox-translations-training#174
   experiment quality +2
   
7. Translation settings for backtranslation are suboptimal firefox-translations-training#92
   bug quality +2
8. Investigate the effect of back-translations firefox-translations-training#592
   quality +1
9. Consider using backward-forward translation for knowledge distillation firefox-translations-training#730
   quality +1
10. Use a more precise language identification model to clean datasets firefox-translations-training#248
    quality +1
11. Filter monolingual data based on fluency scores firefox-translations-training#789
    quality +1
    
12. Consider rebalancing datasets with clustering firefox-translations-training#844
    quality +1
13. Teacher model does not continue training on original corpus firefox-translations-training#472
    quality +1
    
14. Experiment with using more monolingual data firefox-translations-training#181
    quality +1
    
15. Evaluate translation capabilities of LLMs firefox-translations-training#186
    LLM +1
    
16. Ensure monolingual corpus is de-duplicated from the parallel corpus firefox-translations-training#390
    quality +1
    
17. Improve implementation of alignments firefox-translations-training#507
    cost & perf enhancement quality +3
    
18. Teacher training configuration improvements #987
    quality +1

Better cleaning

Preview Give feedback

1. Add full support of the custom filters for OpusCleaner firefox-translations-training#210
   quality +1
   
2. Investigate automatic generation of cleaning rules using OpusFilter firefox-translations-training#370
   language-coverage quality +2
   
3. Upgrade to BicleanerAI 3.0 firefox-translations-training#528
   quality +1
   
4. Investigate monolingual cleaning firefox-translations-training#476
   quality +1
   
5. Explore using genetic algorithms for data cleaning rules firefox-translations-training#581
   quality +1
6. Explore quality estimation methods for data filtering firefox-translations-training#559
   quality +1
7. In the cleaning task, output statistics about filtered out/kept sentences and maybe attach list of filtered out sentences as an artifact firefox-translations-training#274
   enhancement quality +2
8. Investigate Common Voice sentence cleaning rules firefox-translations-training#257
   quality +1
9. Consider using monocleaner for cleaning monolingual corpuses firefox-translations-training#247
   quality +1
   
10. Add a check for to verify datasets aren't already tokenized (similar to JW300 issue) firefox-translations-training#53
    quality +1
11. Handle soft hyphens with custom normalization tables firefox-translations-training#26
    good first issue quality +2
12. Add a cleaning rule for URL names, such as Amazon.com -> Amazon.it firefox-translations-training#758
    quality +1
13. Disable use of bicleaner-hardrules firefox-translations-training#888
    quality +1
    
14. More corpora specific fixes firefox-translations-training#910
    quality +1
15. Filter monolingual data based on fluency scores firefox-translations-training#789
    quality +1
    
16. Add fluency scoring + filtering #985
    quality +1
    

More data

Preview Give feedback

1. Consider integrating more data sources firefox-translations-training#235
   data sources quality +2
2. Add Hugging Face data importer firefox-translations-training#399
   data sources help wanted +2
3. Use SEACrowd datasets firefox-translations-training#324
   data sources good first issue +2
4. Use CommonVoice datasets as monolingual datasets firefox-translations-training#323
   data sources good first issue +2
5. Support monolingual data from OPUS #286
   data sources +1
   
6. Help get more datasets in OPUS firefox-translations-training#252
   data sources good first issue meta +3
7. Add bcp 47 code support in mtdata importer. firefox-translations-training#76
   bug data sources +2
8. tedx download via sacrebleu fails due to directionality of the data set firefox-translations-training#74
   bug data sources +2
9. Integrate HPLT Datasets v1.2 as a monolingual dataset firefox-translations-training#537
   data sources +1
   
10. Integrate datasets used for LLM training as monolingual datasets firefox-translations-training#766
    data sources +1

Evals

Preview Give feedback

1. Add COMET to the evaluation steps firefox-translations-training#503
   evals +1
   
2. Dig deeper in sentences we translate badly to try and identify common failure themes firefox-translations-training#228
   evals +1
3. Train a model to identify "good enough" translations firefox-translations-training#229
   evals +1

[marco-c]

Identify other classes of quality problems by comparing translations with a “good” known one and sort by BLEU

https://github.com/neulab/compare-mt could be another alternative to investigate classes of quality problems.

[marco-c]

Identify other classes of quality problems by comparing translations with a “good” known one and sort by BLEU

https://github.com/neulab/compare-mt could be another alternative to investigate classes of quality problems.

Filed #228.

[marco-c]

See also #238.

[marco-c]

• Using some sort of “fuzzing”/”genetic algorithm” to choose the rules (where the oracle is a LLM)

Interesting approach partially related to this: https://huggingface.co/papers/2309.08532.