Perform a comprehensive testing before the final reuploading

[eu9ene]

We've implemented a lot of fixes recently. Before the final reuploading, we should reupload several language pairs with different configurations and do a comprehensive check that everything is working as expected and that all the data is present and readable.
It should include:

• uploading old Snakemake experiments from GCP
• uploading recent experiments from Taskcluster
• uploading archived Taskcluster experiments from GCP
• running the pipeline on Taskcluster to test the online mode

The goal is not to just run all those but also to carefully verify the results.

[vrigal]

@eu9ene I cleaned the teklia workspace on W&B, and reuploaded experiments:

• From the recent GCP archive you provided (Taskcluster)
• From the old GCP archive you provided (Snakemake)
• From a recent task group Fv23lalyTfSfbmGx0YxypA, with the recursive option.

Everything is visible from https://wandb.ai/teklia/projects.

I also tested with a newer task group Pk0d9wH9Q96Q4U_lUwMnwQ but had this error:

[tracking WARNING] Task group ax-Cwm7TTDSXrQRPhC5tFA cannot be published to WandB: configuration missing @ extra/action/context/input

[eu9ene]

[tracking WARNING] Task group ax-Cwm7TTDSXrQRPhC5tFA cannot be published to WandB: configuration missing @ extra/action/context/input
@vrigal hm, I'm not sure what this is about. Where exactly does it happen? Is it an error from W&B client or Taskcluster?

[vrigal]

@eu9ene It comes from the parser, because the task corresponding to the group is missing the experiment entry in its task definition. Maybe it is an intended behavior ?

[eu9ene]

Ok, this task group is a pull request CI, so it doesn't have a train action. One of the newest manual runs is https://firefox-ci-tc.services.mozilla.com/tasks/groups/dzijiL-PQ4ScKBB3oIjGQg