funannotate built in some singularity versions not correctly running programs on the PATH such as gmap and gsnap

[rbatorsky]

I'm running funannotate on an HPC system where we have neither root or docker. I'm using singularity and mostly having success, except in later versions of singularity where I'm gettin en error calling certain programs on the path like gmap and gsnap.

I'm not sure if it's an issue with the way I'm building funannotate or if it's a problem with singularity. I've also posted the issue there and am hoping to see if anyone here has ideas:
sylabs/singularity#2541

The singularity def file (like Dockerfile) is described in that issue, as is the expected and observed behavior.

I first caught the error when running funannotate check.

Building/running that image with singularity 2.4 it looks like this:

vagrant@vagrant:~$ singularity run funannotate_docker_v3.simg 
linuxbrew@vagrant:~$ funannotate check
-------------------------------------------------------
Checking dependencies for funannotate v1.5.0
-------------------------------------------------------
To print all dependencies and versions: funannotate check --show-versions

You are running Python v 2.7.14. Now checking python packages...
All 11 python packages installed


You are running Perl v 5.026002. Now checking perl modules...
All 27 Perl modules installed


Checking external dependencies...
	ERROR: emapper.py not installed
Checking Environmental Variables...
All 7 environmental variables are set
-------------------------------------------------------

Building/running the identical image with singularity >2.6 it looks like this:

vagrant@vagrant:~$ funannotate check
-------------------------------------------------------
Checking dependencies for funannotate v1.5.0
-------------------------------------------------------
To print all dependencies and versions: funannotate check --show-versions

You are running Python v 2.7.14. Now checking python packages...
All 11 python packages installed


You are running Perl v 5.026002. Now checking perl modules...
All 27 Perl modules installed


Checking external dependencies...
Traceback (most recent call last):
  File "/home/linuxbrew/funannotate/util/check_modules.py", line 260, in <module>
    ExtDeps[prog] = check_version2(prog)
  File "/home/linuxbrew/funannotate/util/check_modules.py", line 66, in check_version2
    vers = vers.split(' called')[0].replace('version', '')
AttributeError: 'list' object has no attribute 'split'

The error is caused by a line in the /home/linuxbrew/funannotate/util/check_modules.py", line 66 that expects output from 'gmap --version' but instead that command returns nothing even though the program is on the path, changing to that directory and running the command gives the expected result, and running the container as root gives the expected result. This is explained in detail in the singularity issue that I submitted.

Please let me know if I can provide any other information and thanks for any advice.
Thank you!

[nextgenusfs]

Hi @rbatorsky, I wish I could be of more help, but I'm not really a docker user and have not tried singularity. I provided the docker image as a request, so if there is a better way to build/optimize the base repository that could be flexible for both docker/singularity, I'm happy to make some changes/updates. On our HPC we also can't/don't use docker, but we do use anaconda/conda, which does simplify the installation. I would like to eventually have a conda recipe for installation, but there still a few problems with some dependencies and finding time to get it done have delayed that project.

[rbatorsky]

Thanks for your feedback. I'll keep trying and post here if I find a solution. I did try the conda installation and all the python packages were simple to install, but I got stuck on some of the other dependencies. I'm in a particularly tricky spot because I need to configure it for many users to run but without root.

[nextgenusfs]

Okay, if you have questions about some specific dependencies, I can probably help with that. Most of the dependencies are on bioconda. I use conda to manage a few of my different pipelines, i.e. then the user just types conda activate funannotate and then are ready to go - and you don't need root for conda.

[rbatorsky]

Hello and thanks again for the advice! I tried to setup the conda enviornment, but connecting all of the non-python dependencies without root was really complicated.

I found that the issue with the singularity image was that when running an image, it's not possible to switch to the linuxbrew user. I opened up the permissions manually while building the image and now the "funannotate check" now returns expected results when I run it on the cluster.

I'm now running into the same issue as #118. I can't write to the augustus config directory within the singularity image. Since funannotate is not passing the $AUGUSTUS_BIN_PATH or $AUGUSTUS_SCRIPT_PATH variable, I can't move the config directory to a writable location.

I'm trying to run with genome assembly only, no RNA-seq. Could you advise on how to get around this issue for my case?

Here's what I'm doing:

singularity exec ${CONTAINER} funannotate predict -i $out_mask -o $out_dir --species "Candida Glabrata" --strain test --busco_seed_species candida_albicans --cpus 20

-------------------------------------------------------
[08:41 PM]: OS: linux2, 40 cores, ~ 132 GB RAM. Python: 2.7.14
[08:41 PM]: Running funannotate v1.5.0
[08:42 PM]: AUGUSTUS (3.3.1) detected, version seems to be compatible with BRAKER and BUSCO
[08:42 PM]: Loading genome assembly and parsing soft-masked repetitive sequences
[08:42 PM]: Genome loaded: 14 scaffolds; 12,338,305 bp; 1.81% repeats masked
[08:42 PM]: Existing protein alignments found: /cluster/kappa/bio/tools/funannotate/1.5.0/example_run/C_glabrata_CBS138_current_chromosomes_funannotate/predict_misc/protein_alignments.gff3
[08:42 PM]: Existing GeneMark annotation found: /cluster/kappa/bio/tools/funannotate/1.5.0/example_run/C_glabrata_CBS138_current_chromosomes_funannotate/predict_misc/genemark.gff
[08:42 PM]: Found 4,918 gene models
[08:42 PM]: Running BUSCO to find conserved gene models for training Augustus
[08:42 PM]: BUSCO training of Augusus failed, check busco logs, exiting

Thank you again.

[nextgenusfs]

Can you copy the Augustus configuration directory to a location you can write to? It needs write permissions to save the training parameters.

[rbatorsky]

Hello and thanks again for your advice.

As I believe was explained in #118, we can't move the location of AUGUSTUS_CONFIG_PATH outside the augustus installation directory, because funannotate uses AUGUSTUS_CONFIG_PATH to infer AUGUSTUS_BIN_PATH and AUGUSTUS_SCRIPT_PATH. I'm trying to get aroung this by copying the /home/linuxbrew/augustus/config directory outside the container in a writable location and binding that location to the original config location:

singularity exec -B /cluster/home/rbator01/example_run/:/home/linuxbrew/data/ -B /cluster/home/rbator01/augustus/config/:/home/linuxbrew/augustus/config/ ${CONTAINER} funannotate predict -i $out_mask -o $out_dir --species "Candida Glabrata" --strain test --busco_seed_species candida_albicans --cpus 10

I'm running into another problem because funannotate writes a temp directory in my home dir and then moves it. Here’s the error:

[08:34 PM]: Running BUSCO to find conserved gene models for training Augustus
Traceback (most recent call last):
  File "/home/linuxbrew/funannotate/bin/funannotate-predict.py", line 894, in <module>
    os.rename('busco', os.path.join(args.out, 'predict_misc', 'busco'))
OSError: [Errno 18] Invalid cross-device link

The problem is that os.rename only works if source and target are on the same filesystem.
Do I understand correctly that it’s moving the folder “busco” to /predict_misc/busco?

Is there any way to set the temporary directory for funannotate files, so that I can make sure it's on the same filesystem as the augustus/config and also final output directory?

I apologize for all these issues that are likely specific to singularity, but I think I’m close to getting this working.

Thanks again.

[nextgenusfs]

You should be able to cp the entire Augustus directory to a write location and then give that location to command at runtime, i.e.

$ funannotate predict

Usage:       funannotate predict <arguments>
version:     1.5.2-b854ac2

Description: Script takes genome multi-fasta file and a variety of inputs to do a comprehensive whole
             genome gene prediction.  Uses AUGUSTUS, GeneMark, BUSCO, BRAKER, EVidence Modeler,
             tbl2asn, tRNAScan-SE, Exonerate, minimap2.
    
Required:  -i, --input              Genome multi-FASTA file (softmasked repeats)
           -o, --out                Output folder name
           -s, --species            Species name, use quotes for binomial, e.g. "Aspergillus fumigatus"

Optional:  --isolate                Isolate name, e.g. Af293
           --strain                 Strain name, e.g. FGSCA4           
           --name                   Locus tag name (assigned by NCBI?). Default: FUN_
           --numbering              Specify where gene numbering starts. Default: 1
           --maker_gff              MAKER2 GFF file. Parse results directly to EVM.
           --pasa_gff               PASA generated gene models. filename:weight
           --other_gff              Annotation pass-through to EVM. filename:weight
           --rna_bam                RNA-seq mapped to genome to train Augustus/GeneMark-ET 
           --stringtie              StringTie GTF result
           --augustus_species       Augustus species config. Default: uses species name
           --genemark_mode          GeneMark mode. Default: ES [ES,ET]
           --genemark_mod           GeneMark ini mod file
           --busco_seed_species     Augustus pre-trained species to start BUSCO. Default: anidulans
           --optimize_augustus      Run 'optimze_augustus.pl' to refine training (long runtime)
           --busco_db               BUSCO models. Default: dikarya. `funannotate outgroups --show_buscos`
           --organism               Fungal-specific options. Default: fungus. [fungus,other]
           --ploidy                 Ploidy of assembly. Default: 1
           -t, --tbl2asn            Assembly parameters for tbl2asn. Default: "-l paired-ends"
           -d, --database           Path to funannotate database. Default: $FUNANNOTATE_DB
           
           --protein_evidence       Proteins to map to genome (prot1.fa prot2.fa uniprot.fa). Default: uniprot.fa
           --protein_alignments     Pre-computed protein alignments in GFF3 format
           --transcript_evidence    mRNA/ESTs to align to genome (trans1.fa ests.fa trinity.fa). Default: none
           --transcript_alignments  Pre-computed transcript alignments in GFF3 format
           --augustus_gff           Pre-computed AUGUSTUS GFF3 results (must use --stopCodonExcludedFromCDS=False)
           --genemark_gtf           Pre-computed GeneMark GTF results
           
           --min_intronlen          Minimum intron length. Default: 10
           --max_intronlen          Maximum intron length. Default: 3000
           --soft_mask              Softmasked length threshold for GeneMark. Default: 2000
           --min_protlen            Minimum protein length. Default: 50
           --repeats2evm            Use repeats in EVM consensus model building
           --repeat_filter          Repetitive gene model filtering. Default: overlap blast [overlap,blast,none]
           --keep_no_stops          Keep gene models without valid stops
           --SeqCenter              Sequencing facilty for NCBI tbl file. Default: CFMR
           --SeqAccession           Sequence accession number for NCBI tbl file. Default: 12345
           --force                  Annotated unmasked genome
           --cpus                   Number of CPUs to use. Default: 2
             
ENV Vars:  If not specified at runtime, will be loaded from your $PATH 
           --EVM_HOME
           --AUGUSTUS_CONFIG_PATH
           --GENEMARK_PATH
           --BAMTOOLS_PATH

You can pass a custom --AUGUSTUS_CONFIG_PATH to the scripts that require Augustus.

Yes I think that it moves the BUSCO output directory -- this can probably be fixed to run it in the correct location, but I think at the time the busco script wasn't very flexible.

[rbatorsky]

Thanks, I thought that it should work, but when I pass --AUGUSTUS_CONFIG_PATH to funannotate_predict.py:

singularity exec ${CONTAINER} funannotate predict -i $out_mask -o $out_dir --species "Candida Glabrata" --strain test --busco_seed_species candida_albicans --cpus 12 --AUGUSTUS_CONFIG_PATH=/cluster/home/rbator01/augustus/config/

It quickly complains that it can't find augustus programs:

rbator01@login001:~/example_run$ cat singularity_funannotate.35641363.err
WARNING: Non existent bind point (directory) in container: '/opt/shared'
WARNING: Non existent bind point (directory) in container: '/cluster/kappa'
[09:43 PM]: OS: linux2, 40 cores, ~ 132 GB RAM. Python: 2.7.14
[09:43 PM]: Running funannotate v1.5.0
[09:43 PM]: Missing Dependencies: /cluster/home/rbator01/augustus/bin/bam2hints.  Please install missing dependencies and re-run script

Seems to infer the bin path from the config path. Let me know if I'm missing something. Thanks again!

[hyphaltip]

can't you just make ~/augustus/bin ~/augustus/scripts ~/augustus/config but ~/augustus/bin and scripts are symlinks to /opt/.../ installed augustus Jason Jason Stajich @hyphaltip http://lab.stajich.org

…

On Sun, Jan 20, 2019 at 1:55 PM rbatorsky ***@***.***> wrote: Thanks, I thought that it should work, but when I pass --AUGUSTUS_CONFIG_PATH to funannotate_predict.py: singularity exec ${CONTAINER} funannotate predict -i $out_mask -o $out_dir --species "Candida Glabrata" --strain test --busco_seed_species candida_albicans --cpus 12 --AUGUSTUS_CONFIG_PATH=/cluster/home/rbator01/augustus/config/ It quickly complains that it can't find augustus programs: ***@***.***:~/example_run$ cat singularity_funannotate.35641363.err WARNING: Non existent bind point (directory) in container: '/opt/shared' WARNING: Non existent bind point (directory) in container: '/cluster/kappa' [09:43 PM]: OS: linux2, 40 cores, ~ 132 GB RAM. Python: 2.7.14 [09:43 PM]: Running funannotate v1.5.0 [09:43 PM]: Missing Dependencies: /cluster/home/rbator01/augustus/bin/bam2hints. Please install missing dependencies and re-run script Seems to infer the bin path from the config path. Let me know if I'm missing something. Thanks again! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#257 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAC-u8AJUBwQP5B6bTiEv3okf8aGqD_rks5vFOXAgaJpZM4Z-kYx> .

[rbatorsky]

Thanks but my augustus is not installed in a system location like /opt/ since I'm running in a singularity container. It's installed in a location inside the container called /home/linuxbrew/augustus/.

I think I have two separate problems:

1. writable augustus/config directory
2. I need to have the final output directory in the same file system as the temp file (os.rename problem)

I'm testing it now to see if I can bind the writable augustus/config directory outside of the container to /home/linuxbrew/augustus/config and also place the final output directory passed to funannotate predict in my home directory so it's on the same file system as the temp files that funannotate creates.

I looked in funannotate_predict.py for a way to easily redirect temp files, but I don't see that available.

[rbatorsky]

Hi, The pipeline is running for me now in the singularity container with the changes I mentioned above.
It would be helpful but not necessary to be able to redirect temp files to a custom location.
If I get around to making a self contained singularity definition file I will let you know.
Thanks for all your work on funannotate!

[mictadlo]

Hi @rbatorsky ,
Could you please share your Singularity recipe with me?

Thank you in advance,

Michal

[jdhayes]

It looks like @rbatorsky started some recipes here:
https://github.com/rbatorsky/singularity-recipes

This one could also work, although it basically uses miniconda to install everything:
https://github.com/jdhayes/funbox
It's a work in progress :)