augustus: ERROR PP::Profile: Error parsing pattern file

[aersoares81]

I'm using the latest Funnanotate, but when I reach the "funannotate predict" step I get this:

augustus: ERROR
        PP::Profile: Error parsing pattern file"/sw/anaconda/2019.10/envs/funannotate-test/lib/python3.8/site-packages/funannotate/config/EOG092C0B3U.prfl", line 8.
[May 18 12:01 PM]: ERROR: augustus --proteinprofile test failed, likely a compilation error. This is required to run BUSCO, exiting.

(conda env is called funannotate-test)

which Augustus give me:
/sw/anaconda/2019.10/envs/funannotate-test/bin/augustus
So I think it's looking at the right Augustus. I tried updating Augustus, but

Updating augustus is constricted by
/ funannotate -> requires augustus==3.3.3=pl5262h5a9fe7b_6

"

So I did a "conda install augustus==3.3.3=pl5262h5a9fe7b_6 --force-reinstall" just in case, but the error persists. (curiously I have another instance of funannotate running and it worked despite some augustus errors. Also, maybe it's worth noting, that I re-installed it fresh on this new conda environment because it was the only way to go around a "Diamond repeat database was created with different version of diamond, please re-run funannotate setup" error message that never disappeared no matter what I did.)

Any suggestions?

[Midreancgle]

I have the same problem

[nextgenusfs]

This is a compilation issue with Augustus with your system. This method is required for BUSCO to work. What OS are you on? Probably need to try a different conda build I guess or compile/install manually.

[Midreancgle]

My system is Ubuntu. The funannotate had been working for a while, and then it didn't work with following problems:

(funannotate) + [ /home/yue/anaconda3/envs/funannotate/test] funannotate predict -i GCA_ 000002515. 1_ ASM251v1_genomic. fna -o . /
species "Kluyveromyces Iactis" - - cpus 45
. May 20 12: 03 PM] : OS: Ubuntu 18. 04, 56 cores, ~ 264 GB RAM. Python: 3. 8.12
[May2012:03PM]:Running funannotate v1.8.9
Traceback ( most recent call Iast) :
Fi I e " /home/yue/anaconda3/envs/funannotate/bin/funannotate", 1i ne 10, i n
sys. exit( main( ))
Fi I e " /home/yue/anaconda3/envs/funannotate/lib/python3. 8/site-packages/funannotate/funannotate. py", Iine 705, in main
mod. mai n( arguments)
File " /home/yue/anaconda3/envs/funannotate/lib/python3. 8/site-packages/funannotate/predict. py", Iine 472, in main
forfinos.listdir(os.path.join(LOCALAUGUSTUS,'species',aug_species)):
FiIeNotFoundError: [ Errno 2] No such fiIe or directory: '. /predict_ misc/ab_initio_parameters/augustus/species/kI uyveromyces_lactis'

I reinstalled it and I had this compile problem.

[zsdxgl]

I have the same problem

[hyphaltip]

I have the same problem
the prediction folder failing? -what folders are created in the directory? Does predict_misc get created.

@Midreancgle the spaces in your quoted errors are hard to read - can you put these in direct quote in github eg (in here)

[mgarl-10]

Same problem here

[katajar]

Same here. In my case test failed.

augustus: ERROR
	PP::Profile: Error parsing pattern file"/home/agazar/anaconda3/envs/funannotate/lib/python3.7/site-packages/funannotate/config/EOG092C0B3U.prfl", line 8.
[May 24 01:10 PM]: ERROR: augustus --proteinprofile test failed, likely a compilation error. This is required to run BUSCO, exiting.

[nextgenusfs]

In the docker image I can't reproduce (my dev machine is Mac so only linux test I have is docker), it installed this version of augustus (so same as yours @aersoares81):

 augustus                                3.3.3  pl5262h5a9fe7b_6     bioconda/linux-64         77 MB

The docker image is running Debian (so not Ubuntu, although ubuntu based on Debian). Here is the successful test:

$ funannotate_dev/funannotate-docker test -t busco --cpus 4
#########################################################
Running `funannotate predict` BUSCO-mediated training unit testing
CMD: funannotate predict -i test.softmasked.fa --protein_evidence protein.evidence.fasta -o annotate --cpus 4 --species Awesome busco
#########################################################
-------------------------------------------------------
[May 24 08:14 AM]: OS: Debian GNU/Linux 10, 4 cores, ~ 8 GB RAM. Python: 3.8.12
[May 24 08:14 AM]: Running funannotate v1.8.10
[May 24 08:14 AM]: GeneMark not found and $GENEMARK_PATH environmental variable missing. Will skip GeneMark ab-initio prediction.
[May 24 08:14 AM]: Skipping CodingQuarry as no --rna_bam passed
[May 24 08:14 AM]: Parsed training data, run ab-initio gene predictors as follows:
  Program      Training-Method
  augustus     busco          
  glimmerhmm   busco          
  snap         busco          
[May 24 08:14 AM]: Loading genome assembly and parsing soft-masked repetitive sequences
[May 24 08:14 AM]: Genome loaded: 6 scaffolds; 3,776,588 bp; 19.75% repeats masked
[May 24 08:14 AM]: Mapping 1,065 proteins to genome using diamond and exonerate
[May 24 08:14 AM]: Found 1,505 preliminary alignments with diamond in 0:00:02 --> generated FASTA files for exonerate in 0:00:00
[May 24 08:15 AM]: Exonerate finished in 0:00:39: found 1,270 alignments
[May 24 08:15 AM]: Running BUSCO to find conserved gene models for training ab-initio predictors
[May 24 08:30 AM]: 373 valid BUSCO predictions found, validating protein sequences
[May 24 08:31 AM]: 370 BUSCO predictions validated
[May 24 08:31 AM]: Training Augustus using BUSCO gene models
[May 24 08:31 AM]: Augustus initial training results:
  Feature       Specificity   Sensitivity
  nucleotides   99.5%         83.8%      
  exons         71.8%         59.7%      
  genes         86.5%         59.3%      
[May 24 08:31 AM]: Running Augustus gene prediction using awesome_busco parameters
[May 24 08:32 AM]: 1,301 predictions from Augustus
[May 24 08:32 AM]: Pulling out high quality Augustus predictions
[May 24 08:32 AM]: Found 313 high quality predictions from Augustus (>90% exon evidence)
[May 24 08:32 AM]: Running SNAP gene prediction, using training data: annotate/predict_misc/busco.final.gff3
[May 24 08:35 AM]: 1,529 predictions from SNAP
[May 24 08:35 AM]: Running GlimmerHMM gene prediction, using training data: annotate/predict_misc/busco.final.gff3
[May 24 08:39 AM]: 1,769 predictions from GlimmerHMM
[May 24 08:39 AM]: Summary of gene models passed to EVM (weights):
  Source         Weight   Count
  Augustus       1        988  
  Augustus HiQ   2        313  
  GlimmerHMM     1        1769 
  snap           1        1529 
  Total          -        4599 
[May 24 08:39 AM]: EVM: partitioning input to ~ 35 genes per partition using min 1500 bp interval
[May 24 08:50 AM]: Converting to GFF3 and collecting all EVM results
[May 24 08:50 AM]: 1,691 total gene models from EVM
[May 24 08:50 AM]: Generating protein fasta files from 1,691 EVM models
[May 24 08:50 AM]: now filtering out bad gene models (< 50 aa in length, transposable elements, etc).
[May 24 08:50 AM]: Found 143 gene models to remove: 0 too short; 0 span gaps; 143 transposable elements
[May 24 08:50 AM]: 1,548 gene models remaining
[May 24 08:50 AM]: Predicting tRNAs
[May 24 08:51 AM]: 112 tRNAscan models are valid (non-overlapping)
[May 24 08:51 AM]: Generating GenBank tbl annotation file
[May 24 08:51 AM]: Collecting final annotation files for 1,660 total gene models
[May 24 08:51 AM]: Converting to final Genbank format
[May 24 08:51 AM]: Funannotate predict is finished, output files are in the annotate/predict_results folder
[May 24 08:51 AM]: Your next step might be functional annotation, suggested commands:
-------------------------------------------------------
Run InterProScan (manual install): 
funannotate iprscan -i annotate -c 4

Run antiSMASH (optional): 
funannotate remote -i annotate -m antismash -e youremail@server.edu

Annotate Genome: 
funannotate annotate -i annotate --cpus 4 --sbt yourSBTfile.txt
-------------------------------------------------------
                
[May 24 08:51 AM]: Training parameters file saved: annotate/predict_results/awesome_busco.parameters.json
[May 24 08:51 AM]: Add species parameters to database:

  funannotate species -s awesome_busco -a annotate/predict_results/awesome_busco.parameters.json

#########################################################
SUCCESS: `funannotate predict` BUSCO-mediated training test complete.
#########################################################

So you can probably help each other by sharing the augustus builds and which are failing on which OS, etc -- and you can test your installation with funannotate test -t busco --cpus x. If you get the PP::Profile error above it means Augustus isn't compiled properly. This is an issue that goes back at least 4-5 years that I've raised but it seems to get fixed and then broken again depending on how augustus is compiled, on macOS the only way I could get it to work was to use a specific version (3.2.1) and modify the compilation. For linux it has generally worked from bioconda over the years, but sometimes builds are incompatible with some OS, so try a different build if possible otherwise just download source and compile/install manually and remove the bioconda installed augustus. Use of funannotate test can help you troubleshoot the install, but this is an augustus install issue and not something I can fix. I have on my list to make augustus >3.4 compatible, but currently that still does not work with funannotate.

[aersoares81]

Following your comment I decided to test run it via Docker on my local macOS machine. I installed Funannotate following the "Quickest start Docker" instructions. This local machine never had any version of augustus, funannotate, etc, before (macOS v.11.6.4). This is what I got:

funannotate-docker test -t predict --cpus 8
#########################################################
Running `funannotate predict` unit testing
CMD: funannotate predict -i test.softmasked.fa --protein_evidence protein.evidence.fasta -o annotate --augustus_species saccharomyces --cpus 8 --species Awesome testicus
#########################################################
-------------------------------------------------------
[May 25 02:54 PM]: OS: Debian GNU/Linux 10, 4 cores, ~ 2 GB RAM. Python: 3.8.12
[May 25 02:54 PM]: Running funannotate v1.8.10
[May 25 02:54 PM]: GeneMark not found and $GENEMARK_PATH environmental variable missing. Will skip GeneMark ab-initio prediction.
[May 25 02:54 PM]: Skipping CodingQuarry as no --rna_bam passed
[May 25 02:54 PM]: Parsed training data, run ab-initio gene predictors as follows:
  Program      Training-Method
  augustus     pretrained
  glimmerhmm   busco
  snap         busco
augustus: ERROR
	PP::Profile: Error parsing pattern file"/venv/lib/python3.8/site-packages/funannotate/config/EOG092C0B3U.prfl", line 8.
[May 25 02:54 PM]: ERROR: augustus --proteinprofile test failed, likely a compilation error. This is required to run BUSCO, exiting.
#########################################################
Traceback (most recent call last):
  File "/venv/bin/funannotate", line 8, in <module>
    sys.exit(main())
  File "/venv/lib/python3.8/site-packages/funannotate/funannotate.py", line 711, in main
    mod.main(arguments)
  File "/venv/lib/python3.8/site-packages/funannotate/test.py", line 405, in main
    runPredictTest(args)
  File "/venv/lib/python3.8/site-packages/funannotate/test.py", line 160, in runPredictTest
    assert 1500 <= countGFFgenes(os.path.join(
  File "/venv/lib/python3.8/site-packages/funannotate/test.py", line 45, in countGFFgenes
    with open(input, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: 'test-predict_02121c9a-e9ca-4739-9e1e-74dc80c19c83/annotate/predict_results/Awesome_testicus.gff3'

After that I went back to our cluster and ran funannotate test -t busco --cpus 8 like you suggested. This was installed via conda on our cluster. This is what happened:

funannotate test -t busco --cpus 8
#########################################################
Running `funannotate predict` BUSCO-mediated training unit testing
Downloading: https://osf.io/te2pf/download?version=1 Bytes: 1489808
CMD: funannotate predict -i test.softmasked.fa --protein_evidence protein.evidence.fasta -o annotate --cpus 8 --species Awesome busco
#########################################################
-------------------------------------------------------
[May 25 03:02 PM]: OS: Ubuntu 18.04, 32 cores, ~ 66 GB RAM. Python: 3.8.12
[May 25 03:02 PM]: Running funannotate v1.8.9
[May 25 03:03 PM]: Skipping CodingQuarry as no --rna_bam passed
[May 25 03:03 PM]: Parsed training data, run ab-initio gene predictors as follows:
  Program      Training-Method
  augustus     busco
  genemark     selftraining
  glimmerhmm   busco
  snap         busco
augustus: ERROR
	PP::Profile: Error parsing pattern file"/sw/anaconda/2019.10/envs/funannotate-test/lib/python3.8/site-packages/funannotate/config/EOG092C0B3U.prfl", line 8.
[May 25 03:03 PM]: ERROR: augustus --proteinprofile test failed, likely a compilation error. This is required to run BUSCO, exiting.
#########################################################
Traceback (most recent call last):
  File "/sw/anaconda/2019.10/envs/funannotate-test/bin/funannotate", line 10, in <module>
    sys.exit(main())
  File "/sw/anaconda/2019.10/envs/funannotate-test/lib/python3.8/site-packages/funannotate/funannotate.py", line 705, in main
    mod.main(arguments)
  File "/sw/anaconda/2019.10/envs/funannotate-test/lib/python3.8/site-packages/funannotate/test.py", line 407, in main
    runBuscoTest(args)
  File "/sw/anaconda/2019.10/envs/funannotate-test/lib/python3.8/site-packages/funannotate/test.py", line 200, in runBuscoTest
    assert 1500 <= countGFFgenes(os.path.join(
  File "/sw/anaconda/2019.10/envs/funannotate-test/lib/python3.8/site-packages/funannotate/test.py", line 45, in countGFFgenes
    with open(input, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: 'test-busco_a5a26b86-60e5-4ee6-8d0b-8a1a3f37b3ce/annotate/predict_results/Awesome_busco.gff3'

(to make sure, I ran which augustus and it pointed to the right binary)

What other steps do you suggest me to take to help troubleshooting this?

[nextgenusfs]

Okay, so something must have changed in last build after I pushed commits yesterday, I can finally reproduce. Not sure what the problem is yet, will update here when its fixed.

(py3-funannotate) jon@Jons-MacBook-Pro:~$ docker pull nextgenusfs/funannotate
Using default tag: latest
latest: Pulling from nextgenusfs/funannotate
b03a94565ecb: Pull complete 
fdaecf38e064: Pull complete 
e1538457407a: Pull complete 
fe881bf022f5: Pull complete 
Digest: sha256:6129b60cbe393471acbe8ea49b2df8dec5e6f5a566f6e78b640b98587266a21d
Status: Downloaded newer image for nextgenusfs/funannotate:latest
docker.io/nextgenusfs/funannotate:latest
(py3-funannotate) jon@Jons-MacBook-Pro:~$ ./funannotate_dev/funannotate-docker test -t predict --cpus 4
#########################################################
Running `funannotate predict` unit testing
CMD: funannotate predict -i test.softmasked.fa --protein_evidence protein.evidence.fasta -o annotate --augustus_species saccharomyces --cpus 4 --species Awesome testicus
#########################################################
-------------------------------------------------------
[May 25 10:01 AM]: OS: Debian GNU/Linux 10, 4 cores, ~ 8 GB RAM. Python: 3.8.12
[May 25 10:01 AM]: Running funannotate v1.8.10
[May 25 10:01 AM]: GeneMark not found and $GENEMARK_PATH environmental variable missing. Will skip GeneMark ab-initio prediction.
[May 25 10:01 AM]: Skipping CodingQuarry as no --rna_bam passed
[May 25 10:01 AM]: Parsed training data, run ab-initio gene predictors as follows:
  Program      Training-Method
  augustus     pretrained     
  glimmerhmm   busco          
  snap         busco          
augustus: ERROR
	PP::Profile: Error parsing pattern file"/venv/lib/python3.8/site-packages/funannotate/config/EOG092C0B3U.prfl", line 8.
[May 25 10:01 AM]: ERROR: augustus --proteinprofile test failed, likely a compilation error. This is required to run BUSCO, exiting.
#########################################################

[nextgenusfs]

Okay, the docker image is finally working again. The workaround was to use Debian apt-get to install augustus (be sure to change/update the AUGUSTUS_CONFIG_PATH from the conda version if you take this route -- you can see the Dockerfile on how I did it. So I don't know why the conda packages are failing, I tried multiple versions with the same proteinprofile error.