[ISSUE] Training stucked for longer time for larger gennome

[rohitlb-1999]

Hi, I am trying to run fun2 for Papaver_somniferum with below size, but it's getting longer time to run, not sure if it has got stucked or it takes longer time?
Papaver_somniferum.ASM357369v1.dna.toplevel.fa : 2.6 GB
Papaver_somniferum.ASM357369v1.61.gff3 : 79M

Even clean command could not finish, stucked at below shown step:

Command used:
funannotate2 train -f Papaver_somniferum.ASM357369v1.dna.toplevel.fa -s "papaver somniferum" -o train_results --cpus 16 -t Papaver_somniferum.ASM357369v1.61.gff3

OUTPUT:
[Jul 16 07:28 PM] Python v3.10.14; funannotate2 v25.7.1; gfftk v25.6.10; buscolite v25.4.24
[Jul 16 07:28 PM] Loading genome assembly and running QC checks
[Jul 16 07:29 PM] Genome stats:
{
"n_contigs": 34380,
"size": 2715377404,
"n50": 204470928,
"n90": 9903009,
"l50": 6,
"l90": 24,
"avg_length": 78981
}
[Jul 16 07:29 PM] Getting taxonomy information
{
"superkingdom": "Eukaryota",
"kingdom": "Viridiplantae",
"phylum": "Streptophyta",
"class": "Magnoliopsida",
"order": "Ranunculales",
"family": "Papaveraceae",
"genus": "Papaver",
"species": "Papaver somniferum"
}
[Jul 16 07:29 PM] Choosing best augustus species based on taxonomy: arabidopsis
[Jul 16 07:34 PM] Training set [/mnt/nsa4/projects/bioit/custom_bioit/projects/IC-A5334_Bindics_cDNA_clones_RNA-Seq/analysis/fun2_annotation/train_results/train_misc/training-set.temp.gff3] loaded with 41770 gene models
[Jul 16 07:35 PM] 26,587 of 41,770 models pass training parameters
[Jul 16 07:37 PM] 26587 gene models selected for training, now splitting into test [n=200] and train [n=26387]
[Jul 16 07:37 PM] Training augustus using training set
[Jul 16 07:39 PM] Initial training completed in 00:02:40s
{
"tool": "augustus",
"model": "67a71ccd-a776-4303-9b85-8591c348cab6",
"n_test_genes": 200,
"ref_genes_found": 174,
"ref_genes_missed": 26,
"extra_query_genes": 26,
"average_aed": 0.19032601432712637,
"nucleotide_sensitivity": 0.8285966236531693,
"nucleotide_precision": 0.8067907338916892,
"exon_sensitivity": 0.5292397660818714,
"exon_precision": 0.5797081182014101,
"gene_sensitivity": 0.59375,
"gene_precision": 0.59375
}
[Jul 16 07:39 PM] Training snap using training set
[Jul 16 07:43 PM] Initial training completed in 00:03:14s
{
"tool": "snap",
"model": "snap-trained.hmm",
"n_test_genes": 200,
"ref_genes_found": 193,
"ref_genes_missed": 7,
"extra_query_genes": 72,
"average_aed": 0.30908754793977833,
"nucleotide_sensitivity": 0.719315410532641,
"nucleotide_precision": 0.7337884105984789,
"exon_sensitivity": 0.3776595744680851,
"exon_precision": 0.33256602870097307,
"gene_sensitivity": 0.125,
"gene_precision": 0.0136986301369863
}
[Jul 16 07:43 PM] Training glimmerHMM using training set
STUCKED HERE FOR MORE THAN 12 hours.

[nextgenusfs]

glimmerHMM does take awhile because I added some iterative training. Although I've never run it on that large of genome, can you tell if the process is still running and how much cpu is being utilized? Perhaps there is a way to speed this up....

[rohitlb-1999]

I think it is getting silently killed, but faced this even after multiple try, not able to move ahead.
I tried with 16 cpus.

[rohitlb-1999]

Given fresh try let's see one more time, will update you if it still alt halt

COMMAND:
funannotate2 train -f /mnt/nsa4/projects/bioit/custom_bioit/projects/IC-A5334_Bindics_cDNA_clones_RNA-Seq/misc/ref_seq/ensembl/Papaver_somniferum.ASM357369v1.dna.toplevel.fa -s "papaver somniferum" -o fun2_train_results --cpus 16 -t /mnt/nsa4/projects/bioit/custom_bioit/projects/IC-A5334_Bindics_cDNA_clones_RNA-Seq/misc/ref_seq/ensembl/Papaver_somniferum.ASM357369v1.61.gff3

RUNNING LOG:
[18:05:41 rlb@srv210 fun2_annotation]$ cat fun2_train_results/logfiles/funannotate-train.log
[Jul 17 05:49 PM] Python v3.10.14; funannotate2 v25.7.1; gfftk v25.6.10; buscolite v25.4.24
[Jul 17 05:49 PM] Loading genome assembly and running QC checks
[Jul 17 05:49 PM] Genome stats:
{
"n_contigs": 34380,
"size": 2715377404,
"n50": 204470928,
"n90": 9903009,
"l50": 6,
"l90": 24,
"avg_length": 78981
}
[Jul 17 05:50 PM] Getting taxonomy information
{
"superkingdom": "Eukaryota",
"kingdom": "Viridiplantae",
"phylum": "Streptophyta",
"class": "Magnoliopsida",
"order": "Ranunculales",
"family": "Papaveraceae",
"genus": "Papaver",
"species": "Papaver somniferum"
}
[Jul 17 05:50 PM] Choosing best augustus species based on taxonomy: arabidopsis
[Jul 17 05:55 PM] Training set [/mnt/nsa4/projects/bioit/custom_bioit/projects/IC-A5334_Bindics_cDNA_clones_RNA-Seq/analysis/fun2_annotation/fun2_train_results/train_misc/training-set.temp.gff3] loaded with 41770 gene models
[Jul 17 05:55 PM] 41770 training set genes; 36508 have multi-CDS
[Jul 17 05:55 PM] diamond makedb --in /mnt/nsa4/projects/bioit/custom_bioit/projects/IC-A5334_Bindics_cDNA_clones_RNA-Seq/analysis/fun2_annotation/fun2_train_results/train_misc/training-set.proteins.fa --db /mnt/nsa4/projects/bioit/custom_bioit/projects/IC-A5334_Bindics_cDNA_clones_RNA-Seq/analysis/fun2_annotation/fun2_train_results/train_misc/training-set.dmnd
[Jul 17 05:55 PM] diamond blastp --query /mnt/nsa4/projects/bioit/custom_bioit/projects/IC-A5334_Bindics_cDNA_clones_RNA-Seq/analysis/fun2_annotation/fun2_train_results/train_misc/training-set.proteins.fa --db /mnt/nsa4/projects/bioit/custom_bioit/projects/IC-A5334_Bindics_cDNA_clones_RNA-Seq/analysis/fun2_annotation/fun2_train_results/train_misc/training-set.dmnd --more-sensitive -o /mnt/nsa4/projects/bioit/custom_bioit/projects/IC-A5334_Bindics_cDNA_clones_RNA-Seq/analysis/fun2_annotation/fun2_train_results/train_misc/training-set.self.blast.txt -f 6 qseqid sseqid pident --query-cover 80 --subject-cover 80 --id 80 --no-self-hits
[Jul 17 05:56 PM] 7,590 models fail blast identity threshold
[Jul 17 05:56 PM] 26,587 of 41,770 models pass training parameters
[Jul 17 05:57 PM] 26587 gene models selected for training, now splitting into test [n=200] and train [n=26387]
[Jul 17 05:58 PM] Training augustus using training set
[Jul 17 05:59 PM] /opt/conda/bin/new_species.pl --AUGUSTUS_CONFIG_PATH=/mnt/nsa4/projects/bioit/custom_bioit/projects/IC-A5334_Bindics_cDNA_clones_RNA-Seq/analysis/fun2_annotation/fun2_train_results/train_misc/augustus --species=16adaac8-789b-45a5-946e-636b24d263a3
[Jul 17 05:59 PM] etraining --species=16adaac8-789b-45a5-946e-636b24d263a3 --AUGUSTUS_CONFIG_PATH=/mnt/nsa4/projects/bioit/custom_bioit/projects/IC-A5334_Bindics_cDNA_clones_RNA-Seq/analysis/fun2_annotation/fun2_train_results/train_misc/augustus augustus.training.gb
[Jul 17 06:00 PM] Initial training completed in 00:02:46s
{
"tool": "augustus",
"model": "16adaac8-789b-45a5-946e-636b24d263a3",
"n_test_genes": 200,
"ref_genes_found": 174,
"ref_genes_missed": 26,
"extra_query_genes": 45,
"average_aed": 0.15983849582055246,
"nucleotide_sensitivity": 0.8687169743010864,
"nucleotide_precision": 0.8332573613568054,
"exon_sensitivity": 0.538235294117647,
"exon_precision": 0.6032369917664033,
"gene_sensitivity": 0.5873015873015873,
"gene_precision": 0.45121951219512196
}
[Jul 17 06:00 PM] Training snap using training set
[Jul 17 06:01 PM] 26387 gene models to train snap on 11 scaffolds
[Jul 17 06:02 PM] fathom snap.training.zff snap-training.scaffolds.fasta -categorize 1000 -min-intron 10 -max-intron 3000
[Jul 17 06:03 PM] fathom uni.ann uni.dna -export 1000 -plus
[Jul 17 06:03 PM] forge -min-intron 10 -lcmask export.ann export.dna
[Jul 17 06:04 PM] perl /opt/conda/bin/hmm-assembler.pl -r snap-trained .
[Jul 17 06:04 PM] Initial training completed in 00:03:10s
{
"tool": "snap",
"model": "snap-trained.hmm",
"n_test_genes": 200,
"ref_genes_found": 197,
"ref_genes_missed": 3,
"extra_query_genes": 154,
"average_aed": 0.2244832189097712,
"nucleotide_sensitivity": 0.791044686398361,
"nucleotide_precision": 0.7855457898103188,
"exon_sensitivity": 0.46938775510204084,
"exon_precision": 0.48251747918614685,
"gene_sensitivity": 0.875,
"gene_precision": 0.12
}
[Jul 17 06:04 PM] Training glimmerHMM using training set
[Jul 17 06:04 PM] trainGlimmerHMM /mnt/nsa4/projects/bioit/custom_bioit/projects/IC-A5334_Bindics_cDNA_clones_RNA-Seq/analysis/fun2_annotation/fun2_train_results/train_misc/glimmerhmm/glimmer.fasta /mnt/nsa4/projects/bioit/custom_bioit/projects/IC-A5334_Bindics_cDNA_clones_RNA-Seq/analysis/fun2_annotation/fun2_train_results/train_misc/glimmerhmm/glimmer.exons -d train
.....

[rohitlb-1999]

Still same issue!

[nextgenusfs]

So you tracked the PID and it has been killed?

[nextgenusfs]

I'd try to run this command separately and see if you get an error:

rainGlimmerHMM /mnt/nsa4/projects/bioit/custom_bioit/projects/IC-A5334_Bindics_cDNA_clones_RNA-Seq/analysis/fun2_annotation/fun2_train_results/train_misc/glimmerhmm/glimmer.fasta /mnt/nsa4/projects/bioit/custom_bioit/projects/IC-A5334_Bindics_cDNA_clones_RNA-Seq/analysis/fun2_annotation/fun2_train_results/train_misc/glimmerhmm/glimmer.exons -d train

[rohitlb-1999]

Yes Tried that, no error shown & no progress. Are you able to replicate the same behaviour, could you please try with below data set ?

Papaver_somniferum.ASM357369v1.dna.toplevel.fa : 2.6 GB
ENSEMBLE LINK TO DOWNLOAD FASTA REF:
wget https://ftp.ensemblgenomes.ebi.ac.uk/pub/plants/release-61/fasta/papaver_somniferum/dna/Papaver_somniferum.ASM357369v1.dna.toplevel.fa.gz

Papaver_somniferum.ASM357369v1.61.gff3 : 79M
ENSEMBLE LINK TO DOWNLOAD GFF3:
wget https://ftp.ensemblgenomes.ebi.ac.uk/pub/plants/release-61/gff3/papaver_somniferum/Papaver_somniferum.ASM357369v1.61.gff3.gz

[rohitlb-1999]

Is there any option to turn of glimmer hmm step?

[rohitlb-1999]

Hi,
I retried running train command this moring and finally got some error log:

COMMAND:
funannotate2 train -f /mnt/nsa4/projects/bioit/custom_bioit/projects/IC-A5334_Bindics_cDNA_clones_RNA-Seq/misc/ref_seq/ensembl/Papaver_somniferum.ASM357369v1.dna.toplevel.fa -s "papaver somniferum" -o fun2_train_results --cpus 16 -t /mnt/nsa4/projects/bioit/custom_bioit/projects/IC-A5334_Bindics_cDNA_clones_RNA-Seq/misc/ref_seq/ensembl/Papaver_somniferum.ASM357369v1.61.gff3

[09:50:30 rlb@srv210 annotation]$ cat fun2_train.log
.....
.....
[Jul 18 06:06 AM] Training glimmerHMM using training set
[Jul 18 09:38 AM] CMD ERROR: trainGlimmerHMM /mnt/nsa4/projects/bioit/custom_bioit/projects/IC-A5334_Bindics_cDNA_clones_RNA-Seq/analysis/annotation/fun2_train_results/train_misc/glimmerhmm/glimmer.fasta /mnt/nsa4/projects/bioit/custom_bioit/projects/IC-A5334_Bindics_cDNA_clones_RNA-Seq/analysis/annotation/fun2_train_results/train_misc/glimmerhmm/glimmer.exons -d train
[Jul 18 09:38 AM] Simple Consensus = ttttttttttttttttttttttgcaggtt
Markov Consensus = ttttttttttttttttttttttgcaggtt
******** Old Way = ttttttttttttttttttttttgcaggtt
Simple Consensus = ttttttttttttttttattttttaagata
Markov Consensus = ttttttttttttttttttttttgaagaaa
******** Old Way = ttttttttttttttttttttttttagaaa
Simple Consensus = taaaggtaagtttttt
Markov Consensus = ttcaggtaagtttttt
******** Old Way = ttcaggtaagtttttt
Simple Consensus = tttttgtttttttttt
Markov Consensus = tttttgtttttttttt
******** Old Way = tttttgtttttttttt
Consens 1 pe poz 23
Frunze de 83950 si 41541 componente
Consens 3 pe poz 21
Frunze de 54702 si 29248 componente
Consens 3 pe poz 20
Frunze de 29787 si 24915 componente
Consens 3 pe poz 14
Frunze de 15707 si 14080 componente
Consens 3 pe poz 15
Frunze de 8074 si 7633 componente
Consens 3 pe poz 18
Frunze de 4116 si 3958 componente
Consens 3 pe poz 12
Frunze de 2085 si 2031 componente
Consens 3 pe poz 9
Frunze de 1098 si 987 componente
Consens 3 pe poz 8
Frunze de 590 si 508 componente
Consens 3 pe poz 6
Frunze de 320 si 270 componente
Consens 3 pe poz 10
Frunze de 267 si 241 componente
Consens 2 pe poz 22
Frunze de 494 si 493 componente
Consens 3 pe poz 10
Frunze de 239 si 255 componente
Consens 3 pe poz 22
Frunze de 252 si 241 componente
Consens 3 pe poz 13
Frunze de 1051 si 980 componente
Consens 3 pe poz 10
Frunze de 550 si 501 componente
Consens 3 pe poz 9
Frunze de 294 si 256 componente
Consens 3 pe poz 19
Frunze de 244 si 257 componente
Consens 0 pe poz 13
Frunze de 497 si 483 componente
Consens 3 pe poz 16
Frunze de 250 si 247 componente
Consens 1 pe poz 13
Frunze de 283 si 200 componente
Consens 3 pe poz 19
Frunze de 1987 si 1971 componente
Consens 3 pe poz 10
Frunze de 1005 si 982 componente
Consens 3 pe poz 13
Frunze de 495 si 510 componente
......
......
......
Frunze de 901 si 335 componente
Consens 1 pe poz 7
Frunze de 482 si 0 componente
Simple Consensus = ttttttttttttttttttttttgcaggta
Markov Consensus = ttttttttttttttttttttttgcaggtg
******** Old Way = ttttttttttttttttttttttgcaggtg
Simple Consensus = tttttttttttttttttttttttcagttt
Markov Consensus = tttttttttttttttttttttttcagttt
******** Old Way = tttttttttttttttttttttttcagttt
Simple Consensus = ttttttttatttttttttttttgcaggta
Markov Consensus = ttttttttgtttttttttttttgcaggtt
******** Old Way = ttttttttgtttttttttttttgcaggtt
Simple Consensus = ttttttttattttttttttttttcagtat
Markov Consensus = ttttttttattttttttttttttcagttt
******** Old Way = ttttttttattttttttttttttcagttt
Simple Consensus = tttttttttattttttttttttgcaggtt
Markov Consensus = ttaatttttcttttttttttttgcaggtt
******** Old Way = tttttttttgttttttttttttgcaggtt
Simple Consensus = ttttttttaattttttgtttttgcagatg
Markov Consensus = tctgtaccaattttttttttttgcagaga
******** Old Way = tttgtacttattttttgtttttgcagaga
Simple Consensus = tttttttttatttttttttttttcaggtt
Markov Consensus = tttttttttatttttttttttttcaggtt
******** Old Way = tttttttttgtttttttttttttcaggtt
Simple Consensus = tttttttttatttttttttttttcagttt
Markov Consensus = tttttttttgtttttttttttttcagttt
******** Old Way = tttttttttgtttttttttttttcagttt
Simple Consensus = ttttttttttttatttttttttgcaggtt
Markov Consensus = ttttttttttttgtttttttttgcaggtt
......
......
......
******** Old Way = aagttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
Simple Consensus = tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
Markov Consensus = tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
******** Old Way = tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt

[Jul 18 09:38 AM] Segmentation fault
ERROR 69: /opt/conda/bin/../share/glimmerhmm/train/score exited funny: 35584 at /opt/conda/bin/trainGlimmerHMM line 445.

[nextgenusfs]

This genome has a ridiculous number of small contigs. I'm wondering if that is causing the problem with glimmerHMM training.

❯ seqkit stats Papaver_somniferum.ASM357369v1.dna.toplevel.fa.gz                             
file                                               format  type  num_seqs        sum_len  min_len   avg_len      max_len
Papaver_somniferum.ASM357369v1.dna.toplevel.fa.gz  FASTA   DNA     34,380  2,715,377,404      252  78,981.3  270,405,062

If we just filtered out those shorter than 10 kb, its only 240 contigs:

❯ seqkit seq --min-len 10000 Papaver_somniferum.ASM357369v1.dna.toplevel.fa.gz | seqkit stats
[WARN] you may switch on flag -g/--remove-gaps to remove spaces
file  format  type  num_seqs        sum_len  min_len       avg_len      max_len
-     FASTA   DNA        240  2,683,884,583   10,000  11,182,852.4  270,405,062

So I would try to remove all contigs less than 10 kb and then try to run funannotate2 train on that assembly and see if it still fails. If that still fails, then I guess it's on the longer contigs. But from your error above, it looks like there is just a bunch of short junk contigs.

If you can run this test, we can then add a filter to ignore short contigs for training -- really they won't be useful anyway for annotation/prediction.

[rohitlb-1999]

Hi, I tried this, but the issue still exist:

$ cat train_results/logfiles/funannotate-train.log
[Jul 19 07:56 PM] Python v3.10.14; funannotate2 v25.7.1; gfftk v25.6.10; buscolite v25.4.24
[Jul 19 07:56 PM] Loading genome assembly and running QC checks
[Jul 19 07:56 PM] Genome stats:
{
"n_contigs": 240,
"size": 2683884583,
"n50": 204470928,
"n90": 12190192,
"l50": 6,
"l90": 22,
"avg_length": 11182852
}
[Jul 19 07:57 PM] Getting taxonomy information
{
"superkingdom": "Eukaryota",
"kingdom": "Viridiplantae",
"phylum": "Streptophyta",
"class": "Magnoliopsida",
"order": "Ranunculales",
"family": "Papaveraceae",
"genus": "Papaver",
"species": "Papaver somniferum"
}
[Jul 19 07:57 PM] Choosing best augustus species based on taxonomy: rice
[Jul 19 08:01 PM] Training set [/mnt/nsa4/projects/bioit/custom_bioit/projects/IC-A5334_Bindics_cDNA_clones_RNA-Seq/analysis/annotation/fun2_train_results/train_results/train_misc/training-set.temp.gff3] loaded with 41770 gene models
[Jul 19 08:02 PM] 41770 training set genes; 36508 have multi-CDS
[Jul 19 08:02 PM] diamond makedb --in /mnt/nsa4/projects/bioit/custom_bioit/projects/IC-A5334_Bindics_cDNA_clones_RNA-Seq/analysis/annotation/fun2_train_results/train_results/train_misc/training-set.proteins.fa --db /mnt/nsa4/projects/bioit/custom_bioit/projects/IC-A5334_Bindics_cDNA_clones_RNA-Seq/analysis/annotation/fun2_train_results/train_results/train_misc/training-set.dmnd
[Jul 19 08:02 PM] diamond blastp --query /mnt/nsa4/projects/bioit/custom_bioit/projects/IC-A5334_Bindics_cDNA_clones_RNA-Seq/analysis/annotation/fun2_train_results/train_results/train_misc/training-set.proteins.fa --db /mnt/nsa4/projects/bioit/custom_bioit/projects/IC-A5334_Bindics_cDNA_clones_RNA-Seq/analysis/annotation/fun2_train_results/train_results/train_misc/training-set.dmnd --more-sensitive -o /mnt/nsa4/projects/bioit/custom_bioit/projects/IC-A5334_Bindics_cDNA_clones_RNA-Seq/analysis/annotation/fun2_train_results/train_results/train_misc/training-set.self.blast.txt -f 6 qseqid sseqid pident --query-cover 80 --subject-cover 80 --id 80 --no-self-hits
[Jul 19 08:02 PM] 7,590 models fail blast identity threshold
[Jul 19 08:02 PM] 26,587 of 41,770 models pass training parameters
[Jul 19 08:04 PM] 26587 gene models selected for training, now splitting into test [n=200] and train [n=26387]
[Jul 19 08:04 PM] Training augustus using training set
[Jul 19 08:05 PM] /opt/conda/bin/new_species.pl --AUGUSTUS_CONFIG_PATH=/mnt/nsa4/projects/bioit/custom_bioit/projects/IC-A5334_Bindics_cDNA_clones_RNA-Seq/analysis/annotation/fun2_train_results/train_results/train_misc/augustus --species=81a90a14-f6b2-4cb1-b2ec-591be6421179
[Jul 19 08:05 PM] etraining --species=81a90a14-f6b2-4cb1-b2ec-591be6421179 --AUGUSTUS_CONFIG_PATH=/mnt/nsa4/projects/bioit/custom_bioit/projects/IC-A5334_Bindics_cDNA_clones_RNA-Seq/analysis/annotation/fun2_train_results/train_results/train_misc/augustus augustus.training.gb
[Jul 19 08:06 PM] Initial training completed in 00:02:31s
{
"tool": "augustus",
"model": "81a90a14-f6b2-4cb1-b2ec-591be6421179",
"n_test_genes": 200,
"ref_genes_found": 181,
"ref_genes_missed": 19,
"extra_query_genes": 34,
"average_aed": 0.15213157896305537,
"nucleotide_sensitivity": 0.8681415255375792,
"nucleotide_precision": 0.8421619889350836,
"exon_sensitivity": 0.5875706214689266,
"exon_precision": 0.6716354045367006,
"gene_sensitivity": 0.7466666666666667,
"gene_precision": 0.6222222222222222
}
[Jul 19 08:06 PM] Training snap using training set
[Jul 19 08:07 PM] 26387 gene models to train snap on 11 scaffolds
[Jul 19 08:08 PM] fathom snap.training.zff snap-training.scaffolds.fasta -categorize 1000 -min-intron 10 -max-intron 3000
[Jul 19 08:09 PM] fathom uni.ann uni.dna -export 1000 -plus
[Jul 19 08:09 PM] forge -min-intron 10 -lcmask export.ann export.dna
[Jul 19 08:09 PM] perl /opt/conda/bin/hmm-assembler.pl -r snap-trained .
[Jul 19 08:10 PM] Initial training completed in 00:03:15s
{
"tool": "snap",
"model": "snap-trained.hmm",
"n_test_genes": 200,
"ref_genes_found": 160,
"ref_genes_missed": 40,
"extra_query_genes": 197,
"average_aed": 0.399817987494647,
"nucleotide_sensitivity": 0.3950886720589528,
"nucleotide_precision": 0.8377100525375112,
"exon_sensitivity": 0.1780821917808219,
"exon_precision": 0.1672374429223744,
"gene_sensitivity": 0.047619047619047616,
"gene_precision": 0.010050251256281407
}
[Jul 19 08:10 PM] Training glimmerHMM using training set
[Jul 19 08:10 PM] trainGlimmerHMM /mnt/nsa4/projects/bioit/custom_bioit/projects/IC-A5334_Bindics_cDNA_clones_RNA-Seq/analysis/annotation/fun2_train_results/train_results/train_misc/glimmerhmm/glimmer.fasta /mnt/nsa4/projects/bioit/custom_bioit/projects/IC-A5334_Bindics_cDNA_clones_RNA-Seq/analysis/annotation/fun2_train_results/train_results/train_misc/glimmerhmm/glimmer.exons -d train
[Jul 19 11:43 PM] CMD ERROR: trainGlimmerHMM /mnt/nsa4/projects/bioit/custom_bioit/projects/IC-A5334_Bindics_cDNA_clones_RNA-Seq/analysis/annotation/fun2_train_results/train_results/train_misc/glimmerhmm/glimmer.fasta /mnt/nsa4/projects/bioit/custom_bioit/projects/IC-A5334_Bindics_cDNA_clones_RNA-Seq/analysis/annotation/fun2_train_results/train_results/train_misc/glimmerhmm/glimmer.exons -d train
[Jul 19 11:43 PM] Simple Consensus = ttttttttttttttttttttttgcaggtt
Markov Consensus = ttttttttttttttttttttttgcaggtt
******** Old Way = ttttttttttttttttttttttgcaggtt
Simple Consensus = ttttttttttttttttattttttaagata
Markov Consensus = ttttttttttttttttttttttgaagaaa
******** Old Way = ttttttttttttttttttttttttagaaa
Simple Consensus = taaaggtaagtttttt
Markov Consensus = ttcaggtaagtttttt
******** Old Way = ttcaggtaagtttttt
Simple Consensus = tttttgtttttttttt
Markov Consensus = tttttgtttttttttt
******** Old Way = tttttgtttttttttt
Consens 1 pe poz 23
Frunze de 83917 si 41541 componente
Consens 3 pe poz 21
Frunze de 54662 si 29255 componente
Consens 3 pe poz 20
Frunze de 29760 si 24902 componente
Consens 3 pe poz 14
Frunze de 15692 si 14068 componente
Consens 3 pe poz 15
Frunze de 8064 si 7628 componente
Consens 3 pe poz 18
Frunze de 4107 si 3957 componente
Consens 3 pe poz 12
Frunze de 2085 si 2022 componente
Consens 3 pe poz 9
Frunze de 1097 si 988 componente
.....
Frunze de 1236 si 479 componente
Consens 0 pe poz 8
Frunze de 902 si 334 componente
Consens 1 pe poz 7
Frunze de 479 si 0 componente
Simple Consensus = ttttttttttttttttttttttgcaggta
Markov Consensus = ttttttttttttttttttttttgcaggtg
******** Old Way = ttttttttttttttttttttttgcaggtg
Simple Consensus = tttttttttttttttttttttttcagttt
Markov Consensus = tttttttttttttttttttttttcagttt
........
******** Old Way = tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
Simple Consensus = ttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttgc
Markov Consensus = ttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttgc
******** Old Way = ttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttgc
Simple Consensus = ttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttatttttta
Markov Consensus = ttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttga
******** Old Way = tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
Simple Consensus = aagttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
Markov Consensus = aagttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
******** Old Way = aagttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
Simple Consensus = tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
Markov Consensus = tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
******** Old Way = tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt

[Jul 19 11:43 PM] Segmentation fault
ERROR 69: /opt/conda/bin/../share/glimmerhmm/train/score exited funny: 35584 at /opt/conda/bin/trainGlimmerHMM line 445.

[10:45:50 rlb@srv197 fun2_train_results]$

[nextgenusfs]

Okay, and did you try to it the default BUSCOlite by chance? Perhaps it's a memory issue or something wrong with the gene models you are passing, or even that there are so many of them (ie 45,000 filtered down to 26,000) is probably way more than needed. Running some tests now to see if we should limit number of models for training to something like 5000.

[nextgenusfs]

Okay, limiting it now by default to 5000 models for training seems to have worked on my system:

❯ f2 train -s "Papaver somniferum" -f Papaver_somniferum.ASM357369v1.dna.toplevel.fa -t Papaver_somniferum.ASM357369v1.61.gff3 --cpus 8 -o papaver
[Jul 20 09:18 PM] Python v3.12.10; funannotate2 v25.7.19; gfftk v25.6.10; buscolite v25.4.24
[Jul 20 09:18 PM] Loading genome assembly and running QC checks
[Jul 20 09:18 PM] Genome stats:
{
  "n_contigs": 34380,
  "size": 2715377404,
  "n50": 204470928,
  "n90": 9903009,
  "l50": 6,
  "l90": 24,
  "avg_length": 78981
}
[Jul 20 09:20 PM] Filtered genome for training: kept 240 contigs (2,683,884,583 bp), filtered 34140 contigs (31,492,821 bp) shorter than 10,000 bp
[Jul 20 09:20 PM] Getting taxonomy information
{
  "superkingdom": "Eukaryota",
  "kingdom": "Viridiplantae",
  "phylum": "Streptophyta",
  "class": "Magnoliopsida",
  "order": "Ranunculales",
  "family": "Papaveraceae",
  "genus": "Papaver",
  "species": "Papaver somniferum"
}
[Jul 20 09:20 PM] Choosing best augustus species based on taxonomy: rice
[Jul 20 09:23 PM] Training set [/Users/jon/Downloads/papaver/train_misc/training-set.temp.gff3] loaded with 41770 gene models
[Jul 20 09:25 PM] 26,587 of 41,770 models pass training parameters
[Jul 20 09:26 PM] Large training set detected (26,587 models). Subsampling to 5,200 best models to prevent memory issues.
[Jul 20 09:26 PM] 5,200 gene models selected for training, now splitting into test [n=200] and train [n=5000]
[Jul 20 09:26 PM] Training augustus using filtered training set
[Jul 20 09:29 PM] Initial training completed in 00:03:06s
{
  "tool": "augustus",
  "model": "d87daa5d-773b-40a4-9681-6578d34f44d0",
  "n_test_genes": 200,
  "ref_genes_found": 175,
  "ref_genes_missed": 25,
  "extra_query_genes": 28,
  "average_aed": 0.1634908120523539,
  "nucleotide_sensitivity": 0.8591282407803729,
  "nucleotide_precision": 0.8259209029793138,
  "exon_sensitivity": 0.5294117647058824,
  "exon_precision": 0.5706340749916601,
  "gene_sensitivity": 0.5833333333333334,
  "gene_precision": 0.5555555555555556
}
[Jul 20 09:29 PM] Training snap using filtered training set
[Jul 20 09:31 PM] Initial training completed in 00:01:38s
{
  "tool": "snap",
  "model": "snap-trained.hmm",
  "n_test_genes": 200,
  "ref_genes_found": 194,
  "ref_genes_missed": 6,
  "extra_query_genes": 142,
  "average_aed": 0.22465130539470737,
  "nucleotide_sensitivity": 0.7844537099933458,
  "nucleotide_precision": 0.7892492717803977,
  "exon_sensitivity": 0.45526315789473687,
  "exon_precision": 0.4489925907772575,
  "gene_sensitivity": 0.7857142857142857,
  "gene_precision": 0.13414634146341464
}
[Jul 20 09:31 PM] Training glimmerHMM using filtered training set
[Jul 20 09:55 PM] Initial training completed in 00:15:06 and parameter optimization completed in 00:08:57s
{
  "tool": "glimmerhmm",
  "model": "train",
  "n_test_genes": 200,
  "ref_genes_found": 151,
  "ref_genes_missed": 49,
  "extra_query_genes": 11,
  "average_aed": 0.16390948750409415,
  "nucleotide_sensitivity": 0.8282959583677336,
  "nucleotide_precision": 0.8574151942286673,
  "exon_sensitivity": 0.5238095238095238,
  "exon_precision": 0.5956618378637712,
  "gene_sensitivity": 0.3717948717948718,
  "gene_precision": 0.725
}
[Jul 20 09:55 PM] Training GeneMark-ES using self-training