WIP: Next steps

[antgonza]

This is based on #1481 so that should be merged first.

Here I'm fixing the last standing points in the joy list, so far:

• Fix Store EBI submission files with the preprocessed data #1479
• In the template documentation changing that sample description is a required field for Qiita deploy to a requirement for EBI, as this is required for submissions.
• Removing 2 cfg variables that are not needed anymore with the new EBI system. Today we got a notification that the old system will be out in 2 months.
• Changes so tests pass in the main branch.

[josenavas]

Why has this been moved from the sample template to the prep template? IIRC, there was a problem if we had duplicated columns between the sample template and the prep template, and the description was required already in the sample template. Also, I think the description applies to the "biological sample" (i.e. sample template) rather than to the preparation sample.

[antgonza]

My error, this was meant to be in the required fields for EBI in the sample template, not the prep template, will change.

[ElDeveloper]

I'm having a hard time figuring out what are the specifics that need review in this PR, maybe once #1490 is merged, it will be easier.

[antgonza]

This is ready for final review. Note that this solves all the open issues in the joy list: #1451. Once this is merged, we just need to fix conflicts between master and the ebi-improvements branch ...

[mortonjt]

Do we still want these tests? Or should they be dropped?

[ElDeveloper]

👍 done with my review!

[josenavas]

Is it worth using moi's moi.job.system_call function? It does the stdout and stderr parsing for you and it will reduce code in here.

[ElDeveloper]

👍 on my end, @josenavas, please merge once your review comments are addressed 👟

[josenavas]

I think it may be useful to store the stderr too, in case that something unexpected happens (not only in EBI end)

[josenavas]

This error message can be improved, as the error is that file is not a demux file.

[josenavas]

If system_call raises, xml_content is not defined, and stderr doesn't contain the value from such call (it will have the value from the above call to system_call in the loop.

[josenavas]

2 comments and tests are still failing. It looks like now you're installing ascp, so why is that test failing?

[antgonza]

@josenavas this is ready. After your comments I realized that we weren't doing the same for all possible failures in commands.py, which I fixed. Also, fixed some warnings that were left over cause missing instrument_model.

[josenavas]

Thanks @antgonza
awesome work!