TDF2MZML fails; running DIANN on .d files

[josenimo]

Description of the bug

I want to acknowledge that quantms is awesome, and I am looking forward to implementing it at our proteomics community with ~50 people. It will standardize our workflow and make our experiment much more replicable and reusable!
Also, I see @ypriverol plans to expand DIANN-related functionalities! These are great, and I am looking forward to seeing them. I will keep an eye out for any discussion, and perhaps I can give my somewhat naive opinion.

Issue:
Trying to run quantms 1.7.3 on .d files. Data was acquired with diaPASEF on timsTOF Ultra. I am running just 5 samples locally on my machine, and I also ran into on SLURM HPC, the errors were the same.

Command 1:
It runs into an issue with BIGBIO_QUANTMS:QUANTMS:FILE_PREPARATION:MZML_STATISTICS, because
[mzml_statistics] - Error processing file Olive_20241129_JoN_Evo2_80SPD_P26E17_991_C7_S2-A5_1_8397.d: Unsupported file type: .d

This is unexpected, it seems it fails at trying to convert .d into .mzml files for QC. I would either expect that the files are converted as default, or no QC is provided as default. This just fails without using a specific parameter. I write this without knowing the internal logic of quantms, so I might be unrealistic.

Command 2:
After some LLM help I figured I try to add --convert_dotd
What then happens is that BIGBIO_QUANTMS:QUANTMS:FILE_PREPARATION:TDF2MZML runs, but fails to deliver the .mzml file to the next step, and it gets stuck in retries.

Command used and terminal output

#command 1
❯ nextflow run bigbio/quantms -r 1.7.0 \
-profile docker,emulate_amd64 \
--input ./SDRF/QC_test1.sdrf.tsv \
--database ./FASTA/UP000005640_9606.fasta

#command 2
❯ nextflow run bigbio/quantms -r 1.7.0 \
-profile docker,emulate_amd64 \
--input ./SDRF/QC_test1.sdrf.tsv \
--database ./FASTA/UP000005640_9606.fasta \
--convert_dotd

Relevant files

test SDRF file (as txt, but loaded as .tsv)

QC_test1.sdrf.txt

nextflow.log (command 1)

nextflow_command1.log.txt

nextflow.log (command 2)

.nextflow.log

.command.log

.command.log

command.sh

.command.sh

command.err (as txt)

.command.txt

conversion.log

Olive_20241129_JoN_Evo2_80SPD_P26E17_991_C3_S2-A1_1_8393_conversion.log

ll -a of work directory

❯ ll -la
.rw-r--r--@    0 jnimoca 24 Feb 17:25  .command.begin
.rw-r--r--@  154 jnimoca 24 Feb 17:30  .command.err
.rw-r--r--@ 2.0k jnimoca 24 Feb 17:30  .command.log
.rw-r--r--@ 1.9k jnimoca 24 Feb 17:30  .command.out
.rw-r--r--@  11k jnimoca 24 Feb 17:25  .command.run
.rw-r--r--@  541 jnimoca 24 Feb 17:25  .command.sh
.rw-r--r--     0 jnimoca 24 Feb 17:25  .command.trace
.rw-r--r--@ 6.1k jnimoca 24 Feb 17:36  .DS_Store
.rw-r--r--@    1 jnimoca 24 Feb 17:30  .exitcode
lrwxr-xr-x@    - jnimoca 24 Feb 17:25  Olive_20241129_JoN_Evo2_80SPD_P26E17_991_C3_S2-A1_1_8393.d -> /Users/jnimoca/Jose_BI/2_Infrastructure/PY19_quantms_test/raws/Olive_20241129_JoN_Evo2_80SPD_P26E17_991_C3_S2-A1_1_8393.d
.rw-r--r--  3.1G jnimoca 24 Feb 17:30  Olive_20241129_JoN_Evo2_80SPD_P26E17_991_C3_S2-A1_1_8393.mzml
.rw-r--r--  1.9k jnimoca 24 Feb 17:30  Olive_20241129_JoN_Evo2_80SPD_P26E17_991_C3_S2-A1_1_8393_conversion.log

System information

Nextflow version: 25.10.4
Hardware: Macbook Pro M3 Max w 36gb RAM
Executor: local
Container engine: Docker
OS: MacOS 15.4
bigbio/quantms version: 1.7.3

[ypriverol]

@josenimo Thanks a lot for reporting this I will do right away a PR to remove the conversion to mzML for .d files; originally, we used the mzML to compute a lot of qc metrics but now pmultiqc supports most of the plots based on only the DIANN output; give me one day and I will fix this.

The support for conversion from .d -> mzML is something I don't have full control over and as far as I remember, it was buggy. While I use quantms daily for DIA, is mainly for thermo data; then stay with me if you find any other minor issues.

We should tackle this bug in bigbio/quantmsdiann#8

[josenimo]

Thank you @ypriverol!
I will keep at it, and report anything not working or unexpected

[jpfeuffer]

@ypriverol While I agree that the parts of QC that need mzml should be optional, this all actually looks solveable.
It very much looks like the conversion succeeds and the workflow just passes the wrong set of files.
In the case of convert_dotd (which by the way should only be used if you really want DiaNN to run on the mzml), it also looks like an obvious avoidable bug in the workflow to me:

mv: 'Olive_20241129_JoN_Evo2_80SPD_P26E17_991_C3_S2-A1_1_8393.mzml' and 'Olive_20241129_JoN_Evo2_80SPD_P26E17_991_C3_S2-A1_1_8393.mzML' are the same file

[jpfeuffer]

@ypriverol we should ideally add tests for all code paths to increase test coverage. At least with nextflow stub runs!
@josenimo what are the smallest .d file you have? 😅 the problem with those vendor files is that you never can filter them to produce test files

[josenimo]

I can look tomorrow for the shortest runs we have. I would still expect at least a gb 🙃.
I have a contact at Bruker here in Berlin. i can ask him for ideas 💡

[jpfeuffer]

Yeah I thought so 😀 Oh a contact is actually nice. Yes some test file or way of forcefully filtering the files would be nice. Also I always wanted to add reading support to pyopenms but it's just very messy with that mixture of SQL DB, metadata and translations (from frame to IM value etc.) using their proprietary dlls.

[josenimo]

@jpfeuffer could you elaborate on what you mean with forcefully filtering the files?
I must say I am rather naive in the open handling of .d or .raw files.

[jpfeuffer]

Ah I mean just truncating it by e.g. RT, which is obviously pretty easy with mzml but I dont know about .d, especially since it will be hard to "write it back" since I dont know any open ".d-writer" package.