IQ-TREE multicore version 2.3.6 for Linux x86 64-bit issue with core usage when using -T AUTO and --threads-max combination

[uee9]

Hello,

I have run into an issue with running IQTREE v2.3.6 on linux HPC which was installed via Conda.

When I ran the command:
$ iqtree -m GTR -s alignment.fasta -B 1000 -T AUTO --threads-max 12

The tree inference process would progress through the initial steps, and the log file would progress to the "creation of the initial fast parsimony tree." At this point progress would stop and no additional information would be printed to the log nor would any files be written to the disk.

Monitoring the job using the Linux top command showed a complete maxing out of CPU being used by iqtree for an extended period of time. The node running the job has 112 cores, all of which were maxed out via htop. However, iqtree was using no RAM during this time.

The program did not error out unless the job timed out.

The issue resolved when running a reduced command line:
$ iqtree -m GTR -s alignment.fasta -B 1000

I was able to recreate this issue on two machines with different installs of v2.3.6 but unable to recreate the issue on a third machine using IQTREE 2.2.2.3 COVID-edition for Linux 64-bit.

[thomaskf]

Hi @uee9
Thank you for reporting the issue. Unfortunately, I was unable to replicate the problem. I tested version 2.3.6 on two Linux machines, one with 256 cores and the other with 16 cores. IQ-TREE worked as expected using the -T AUTO and --threads-max options.
Did you use the binary from our release? Based on your log description, it seems IQ-TREE paused during the generation of 1000 samples for ultrafast bootstrapping, which follows the initial fast parsimony tree step. This process is unrelated to the -T AUTO or --threads-max options.
You may try deleting all temporary files from the previous run and rerunning IQ-TREE to see if the issue still appears.
Thank you again for reporting this.

[corneliusroemer]

I have also encountered this exact issue using bioconda installed iqtree 2.3.6 on macOS ARM - it is potentially quite serious as a command that should take seconds seems to run indefinitely for no reason and use more CPU.

In particular, I've noticed that --threads-max seems to be ignored for the parsimony part of tree building and can actually screw up the parsimony tree building step, it seems to run forever if I use -T auto instead of -T 1, where it quickly fails with ERROR: Initial tree is not compatible with constraint tree

I've attached files to allow you to reproduce the exact way I'm running this (ideally on mac):

This runs almost forever? And definitely uses more than 100% CPU, more like 220% in my experience:

iqtree2 --threads-max 1 -s builds/proteins/masked_masked_without_recombinants-delim.fasta -m GTR --ninit 2 -n 2 --epsilon 0.05 --redo -czb -g builds/proteins/pruned_constraint.nwk -ninit 1 -n 1 -T auto > builds/proteins/masked_masked_without_recombinants-delim.iqtree.log

When I explicitly set -T 1 all is good:

iqtree2 --threads-max 1 -s builds/proteins/masked_masked_without_recombinants-delim.fasta -m GTR --ninit 2 -n 2 --epsilon 0.05 --redo -czb -g builds/proteins/pruned_constraint.nwk -ninit 1 -n 1 -T 1 > builds/proteins/masked_masked_without_recombinants-delim.iqtree.log
ERROR: Initial tree is not compatible with constraint tree

Showing the bug, it keeps running, 1m30 and counting until I abort:

When I set -T 1 it finishes in a few seconds:

input_files.zip

Some logs:

IQ-TREE multicore version 2.3.6 for MacOS ARM 64-bit built Aug  4 2024
Developed by Bui Quang Minh, Nguyen Lam Tung, Olga Chernomor, Heiko Schmidt,
Dominik Schrempf, Michael Woodhams, Ly Trong Nhan, Thomas Wong

Host:    m1-mbp.local (SSE4.2, 32 GB RAM)
Command: iqtree2 --threads-max 1 -s builds/wuhan/masked_masked_without_recombinants-delim.fasta -m GTR --ninit 2 -n 2 --epsilon 0.05 -T AUTO --redo -czb -g builds/wuhan/>
Seed:    998322 (Using SPRNG - Scalable Parallel Random Number Generator)
Time:    Wed Nov 13 15:32:07 2024
Kernel:  SSE2 - auto-detect threads (10 CPU cores detected)

Reading alignment file builds/wuhan/masked_masked_without_recombinants-delim.fasta ... Fasta format detected
Reading fasta file: done in 0.906675 secs using 71.21% CPU
Alignment most likely contains DNA/RNA sequences
WARNING: 203 sites contain only gaps or ambiguous characters.
Alignment has 2266 sequences with 29903 columns, 3833 distinct patterns
1861 parsimony-informative, 2130 singleton sites, 25912 constant sites
            Gap/Ambiguity  Composition  p-value

[corneliusroemer]

Looking at the diff v2.3.5 to v2.3.6, one thing that might seem relevant is this line (there isn't much else going on that could cause this):
v2.3.5...v2.3.6#diff-e6cc54bd5b4208db7f24b3e671f9d431a884666bf330b712723e3123bb14b48dR2729

Location of this line changed:

boot_tree->setNumThreads(num_threads);

Could it be that the restore checkpoint changes the threads?

[corneliusroemer]

Something else from my bug: it only happens when there's enough CPU available to run at say 200% CPU.

When I have lots of things running in parallel and there's only 100% or so available then I don't hit the bug. So looks like a race condition of some sort? But there should be no parallelity in any case if I say --max-threads 1.