Large genome reassembly based on Hi-C data (continuation and partial rewrite of GRAAL; Marie-Nelly et al., 2013; 2014) and post-scaffolding polishing libraries.
This work is under continuous development/improvement - see GRAAL for information about the basic principles.
You can now easily install instaGRAAL using a docker container available below or you can try it on Galaxy Europe.
Install from PyPI:
sudo pip3 install -U instagraalor, if you want to get the very latest version:
sudo pip3 install -e git+https://github.com/koszullab/instagraal.git@master#egg=instagraalThis should automatically handle most dependencies.
The scaffolder and polishing libraries are written in Python 3 and CUDA. As such, an NVIDIA graphics card is required for the scaffolder to run. The Python 2 version is available at the python2 branch of this repository, but be aware that development will mainly focus on the Python 3 version. The software has been tested for Ubuntu 17.04 and later, and most dependencies can be downloaded with its package manager (or Python's pip).
You will need to download and install the NVIDIA CUDA toolkit. Manual installation is recommended - installing nvidia-cuda-toolkit from Ubuntu's package manager has been known to cause glitches. It is fairly straightforward on OS X thanks to the installation wizard. Here is how to quickly do it on Ubuntu 18.04:
wget https://developer.nvidia.com/compute/cuda/10.0/Prod/local_installers/cuda_10.0.130_410.48_linux
chmod +x cuda_10.0.130_410.48_linux
sudo ./cuda_10.0.130_410.48_linuxNote to Ubuntu users: Be aware that the installation script will fail if it isn't run as root, or if a graphical instance (e.g. X) is running as well. You may need to temporarily shut it down, for instance by switching to tty1 and running the following (prior to the installation script):
sudo service lightdm stop(Replace lightdm with mdm, gdm or whichever login manager is present on your machine if that fails; if all else fails as well, you may have to run something like sudo pkill Xorg instead.)
Note to OS X users: There is currently no CUDA support on Mojave (10.14) and it is unclear when it is going to be added, if it is to be added at all. This means instaGRAAL (or indeed any CUDA-based application) will not work on Mojave. If you wish to run it on OS X, the only solution for now is to downgrade to High Sierra (10.13).
Because some Python dependencies (such as pyopengl or h5py) require to be built against specific files, it is recommended that you install the following packages if you encounter errors.
libglu1-mesalibxi-devlibxmu-devlibglu1-mesa-devfreeglut3-dev
hdf5-tools(hdf5for OS X in brew)
libboost-all-dev(boostandboost-pythonfor OS X in brew)
Python package requirements should be handled automatically by pip, but should you
wish to install them manually, these are:
numpyscipymatplotlibcodepyh5pypyopengldocoptbiopython
They can also be handily installed using the supplied requirements file in the repo:
pip3 install -Ur requirements.txt
You will also need to build pycuda with OpenGL support and disable its use of custom Boost libraries. Installing it directly from PyPI will cause errors at runtime. Here is how to do it manually with Git on Ubuntu or OS X:
git clone --recurse-submodules https://github.com/inducer/pycuda.git
cd pycuda
python3 configure.py --cuda-enable-gl --no-use-shipped-boost
sudo python3 setup.py installYou may run (as root) instagraal-setup, an all-in-one script to handle all the above dependencies on Ubuntu 17+.
There is experimental Docker support for instaGRAAL. You may fetch the corresponding image by running the following:
docker pull koszullab/instagraalAnd run it with
docker run --gpus all koszullab/instagraalNote: Running the container requires the dependency nvidia-docker2 [installation]
Unlike GRAAL, this is meant to be run from the command line.
instagraal <hic_folder> <reference.fa> [<output_folder>]
[--level=4] [--cycles=100] [--coverage-std=1]
[--neighborhood=5] [--device=0] [--circular] [--bomb]
[--save-matrix] [--pyramid-only] [--save-pickle] [--simple]
[--quiet] [--debug]
-h, --help Display this help message.
--version Display the program's current version.
-l 4, --level 4 Level (resolution) of the contact map.
Increasing level by one means a threefold smaller
resolution but also a threefold faster computation
time. [default: 4]
-n 100, --cycles 100 Number of iterations to perform for each bin.
(row/column of the contact map). A high number of
cycles has diminishing returns but there is a
necessary minimum for assembly convergence.
[default: 100]
-c 1, --coverage-std 1 Number of standard deviations below the mean.
coverage, below which fragments should be filtered
out prior to binning. [default: 1]
-N 5, --neighborhood 5 Number of neighbors to sample for potential
mutations for each bin. [default: 5]
--device 0 If multiple graphic cards are available, select
a specific device (numbered from 0). [default: 0]
-C, --circular Indicates genome is circular. [default: False]
-b, --bomb Explode the genome prior to scaffolding.
[default: False]
--pyramid-only Only build multi-resolution contact maps (pyramids)
and don't do any scaffolding. [default: False]
--save-pickle Dump all info from the instaGRAAL run into a
pickle. Primarily for development purposes, but
also for advanced post hoc introspection.
[default: False]
--save-matrix Saves a preview of the contact map after each
cycle, in csv format. [default: False]
--simple Only perform operations at the edge of the contigs.
[default: False]
--quiet Only display warnings and errors as outputs.
[default: False]
--debug Display debug information. For development purposes
only. Mutually exclusive with --quiet, and will
override it. [default: False]
The above <hic_folder> passed as an argument to instaGRAAL needs three files:
- A file named
abs_fragments_contacts_weighted.txt, containing the (sparse) Hi-C map itself. The first line must beid_frag_a id_frag_b n_contact. All subsequent lines must represent the map's contacts in coordinate format (id_frag_abeing the row indices,id_frag_bbeing the column indices,n_contactbeing the number of contacts between each locus or index pair, e.g. if 5 contacts are found between fragments #2 and #3, there should be a line reading2 3 5in the file).n_contactmust be an integer. The list should be sorted according toid_frag_afirst, thenid_frag_b. Fragment ids start at 0. - A file named
fragments_list.txtcontaining information related to each fragment of the genome. The first line must beid chrom start_pos end_pos size gc_content, and subsequent lines (representing the fragments themselves) should follow that template. The fields should be self-explanatory; notably,chromcan be any string representing the chromosome's name to which the fragment at a given line belongs, and fragment ids should start over at 1 when the chromosome name changes. Aside from thechromfield and thegcfield which is currently unused in this version and can be filled with any value, all fields should be integers. Note thatstart_posstarts at 0. - A file named
info_contigs.txtcontaining information related to each contig/scaffold/chromosome in the genome. The first line must becontig length_kb n_frags cumul_length. Field names should be again self-explanatory; naturally the contig field must contain names that are consistent with those found infragments_list.txt. Alsolength_kbshould be an integer (rounded up or down if need be), andn_fragsandcumul_lengthare supposed to be consistent with each other in that the cumulated length (in fragments) of contig N should be equal to the sum of the fields found inn_fragsfor the N-1 preceding lines. Note thatcumul_lengthstarts at 0.
All fields (including those in the files' headers) must be separated by tabs.
Minimal working templates are provided in the example folder.
If you want to generate instaGRAAL-compatible matrices from scratch (i.e. from reads and a reference genome, as opposed to existing Hi-C data in one of the numerous existing formats), you may do so with hicstuff, which acts as both a Python library and a pipeline. Instructions, parameters and optional arguments are detailed in the repo's readme. We strongly recommend using hicstuff with the parameter -m iterative or -m cutsite to improve mapping.
After the scaffolder is done running, whatever path you specified as output will contain a test_mcmc_X directory, where X is the level (resolution) at which scaffolding was performed. This directory, in turn, will contain the following:
genome.fasta: the scaffolded genome. Scaffolds will be ordered by increasing size in fragments, which roughly (but not always) translates into increasing size in bp.info_frags.txt: a file that contains, for each newly formed scaffold, the original coordinates of every single bin in that scaffold, in the formatchromosome, id, orientation, start, end. Each bin has a unique ID that provides a convenient way of tracking consecutive stretches. Orientations are relative to one another, and when "-1" is supplied, it is understood that the reverse complement should be taken.
Other files are mostly for developmental purposes and keep track of the evolution of various metrics and model parameters.
This step is strongly recommended to improve the quality of your scaffolds, unless your input contigs have many misassemblies. Lingering artifacts found in output genomes can be corrected by editing the info_frags.txt file, either by hand or with a script. Look at options by running the following:
instagraal-polish -h
The most common use case is to run all curation procedures at once:
instagraal-polish -m polishing -i info_frags.txt -f contigs.fasta -o curated_assembly.fa
You can add gaps with the parameter -j (necessary for subsequent gap filling), for instance gaps with 10 Ns in this example:
instagraal-polish -m polishing -i info_frags.txt -f contigs.fasta -o curated_assembly.fa -j NNNNNNNNNN
If the output is not as you would expect:
- check the Hi-C mapping rate; if the mapping rate is low, this may be due to:
- a poor contig completeness (check BUSCO and k-mer completeness)
- not using the parameter iterative or cutsite when running hicstuff
- make sure that there are few artefactual duplications
- if the assembly was obtained with low-accuracy Nanopore reads, polish the assembly with highly accurate reads prior to scaffolding
- check that there are trans contacts between contigs in the contact map prior to scaffolding, as insufficient contacts will lead to poor Hi-C scaffolding
- try switching aligner from bwa to bowtie2 or vice versa, as we have noticed sometimes different scaffolding outputs depending on the aligner used by hicstuff
By default, the parameter --level is set to 4. For genomes larger than 500 Mb, increasing it to 5 is often more adapted to improve runtime, and 6 for genomes larger than 3 Gb.
This is due to spaces and special characters in contig names. Check that the contig names match the ones in the outputs from hicstuff, and rename your contigs if necessary.
If you encounter the following error, despite having installed the NVIDIA CUDA Toolkit:
ImportError: libcurand.so.9.2: cannot open shared object file: No such file or directory
it probably means the CUDA-related libraries haven't been properly added to your $PATH for some reason. A quick solution is to simply add this at the end of your .bashrc or .bash_profile (replace the paths with wherever you installed the toolkit and change the version number accordingly):
export PATH=/usr/local/cuda-9.2/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-9.2/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
If you encounter the following error:
freeglut (instagraal.py): failed to open display ''
it most likely means you attempted to run an instaGRAAL instance remotely (e.g. over ssh) but didn't configure a proper $DISPLAY variable. In order to avoid this, simply run the following beforehand:
export DISPLAY=:0
Note that this will disable the movie (it will play on the remote machine instead).
However, instaGRAAL is based on OpenGL, which means there has to be an X server of some kind running on your target machine no matter what. While this allows for pretty movies and visualizations, it may prove problematic on an environment you don't have total control over, e.g. a server cluster. Currently, your best bet is asking the system administrator of the target machine to set up an X instance (possibly virtual, such as Xvfb or xserver-xorg-video-dummy) if they haven't already.
If you encounter the following:
NullFunctionError: Attempt to call an undefined function glutInit, check for bool(glutInit) before calling
check whether you have installed freeglut3-dev. It seems that the pyopengl library does not include a GLUT implementation when installed from PyPI. Alternatively, just installing pyopengl with your package manager (e.g. python3-pyopengl on Ubuntu) seems to work as well.
If you encounter an error like the following :
File "/usr/local/lib/python3.6/dist-packages/codepy/toolchain.py", line 382, in _guess_toolchain_kwargs_from_python_config
object_suffix = '.' + make_vars['MODOBJS'].split()[0].split('.')[1]
IndexError: list index out of range
You may need to upgrade to a more recent version of codepy.
sudo pip3 install --upgrade --no-cache-dir -e git+https://github.com/inducer/codepy.git@master#egg=codepyNo such error has been found as of commit 10a014f, so if you encounter regressions after this, you should stick to that version.
Depending on your system, you may also need to upgrade to gcc/g++ 8:
sudo apt install gcc-8 g++-8If for some reason your system does not automatically switch to gcc/g++-8, you should manually configure your system to do so, e.g. on Ubuntu:
sudo update-alternatives --remove-all gcc
sudo update-alternatives --remove-all g++
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-8 10
sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-8 10-
instaGRAAL will attempt to detect already-built pyramids in hdf5 format, but if building was interrupted for some reason, and you re-run the pyramid building step, the hdf5 files will still be there, but corrupted. You will need to manually delete the
pyramidsfolder and try again. -
If there is a mismatch between the version of CUDA you installed and the one instaGRAAL seems to rely on (e.g. you installed CUDA 10 but instaGRAAL complains that it can't find
libcurand.so.9.1), try reinstallingpycudaandinstagraalwith the--no-cache-diroption. -
You may experience issues if you handle dependencies with conda, such as
pycudafailing to build because some header files that would be present when you installedlibboost-all-devaren't automatically recognized. If you don't want to manually mess with your$PATH, it's probably best to just deactivate conda altogether and install everything with your OS's normal package manger (andpip).
As a Python package, instaGRAAL provides both a scaffolding and polishing library, as well as a convenient Hi-C matrix handling framework, and we've tried to expose much of the API behind these on readthedocs. If you wish to know more about how the scaffolder works, see the references, especially the supplementary method delving deeper into the details of the model.
- instaGRAAL: chromosome-level quality scaffolding of genomes using a proximity ligation-based scaffolder Lyam Baudry, Nadège Guiglielmoni, Hervé Marie-Nelly, Alexandre Cormier, Martial Marbouty, Komlan Avia, Yann Loe Mie, Olivier Godfroy, Lieven Sterck, J. Mark Cock, Christophe Zimmer, Susana M. Coelho & Romain Koszul
- High-quality genome assembly using chromosomal contact data, Hervé Marie-Nelly, Martial Marbouty, Axel Cournac, Jean-François Flot, Gianni Liti, Dante Poggi Parodi, Sylvie Syan, Nancy Guillén, Antoine Margeot, Christophe Zimmer and Romain Koszul, Nature Communications, 2014
- A probabilistic approach for genome assembly from high-throughput chromosome conformation capture data, Hervé Marie-Nelly, 2013, PhD thesis
- Proximity ligation scaffolding and comparison of two Trichoderma reesei strains genomes, Etienne Jourdier, Lyam Baudry, Dante Poggi-Parodi, Yoan Vicq, Romain Koszul, Antoine Margeot, Martial Marbouty, and Frédérique Bidard, Biotechnology for Biofuels, 2017
- Scaffolding bacterial genomes and probing host-virus interactions in gut microbiome by proximity ligation (chromosome capture) assay, Martial Marbouty, Lyam Baudry, Axel Cournac, and Romain Koszul, Science Advances, 2017