Update kraken2 and add support for custom databases

[pvanheus]

FOR CONTRIBUTOR:

• I have read the CONTRIBUTING.md document and this tool is appropriate for the tools-iuc repo.
• License permits unrestricted use (educational + commercial)
• This PR adds a new tool or tool collection
• This PR updates an existing tool or tool collection
• This PR does something else (explain below)

[pvanheus]

This both updates the kraken2 tool to version 2.1.6 and adds support for custom databases. The custom databases are provided by directory types and because zip files without a common prefix get unpacked to a directory prefixed by the name of the original dataset (see the galaxy code), it uses some shell code to find the complete path to the custom database.

[bernt-matthias]

• How big are those typically?
• Should we subclass directory for that?
• We should also add a tool that crates the K2 databases, or?

[pvanheus]

Gigabyte scale. E.g. the Mycobacterium genus databases from Hall is 7.6 GB (https://zenodo.org/records/8343322). And yes, ideally we need a datatype and tool for creating Kraken2 databases, but there are already several Kraken2 databases available online that I am using for e.g. my M. tuberculosis analyses. I'm happy to work on PRs for the Kraken2 database datatype and the tool that creates them, but I don't think that that work should block this PR.

[bernt-matthias]

Would this be of use for many users?

How about just adding this to the data manager?

We are just updating the dm anyway #6980

[pvanheus]

The DM is very useful... the use case here is in Galaxy's Workflow Landings API, which is oriented around inputs, not DMs. And also, once there is a kraken2-build tool, custom databases.

[bernt-matthias]

@mvdbeek when are data manager bundles expected. Would this help here?

[mvdbeek]

Directory datasets are a much better idea

[bernt-matthias]

the use case here is in Galaxy's Workflow Landings API

Would this then mean that the user uploads multiple GB "just" for a single workflow run? I have not yet used Galaxy's Workflow Landings API - how is this supposed to be used?

[pvanheus]

Some notes on the future kraken2 custom database builder tool:

The kraken2 db building process involves three steps:

1. Download the NCBI taxonomy (kraken2-build --download-taxonomy or k2 download-taxonomy) to the database directory. This downloads some 7GB of data from ftp.ncbi.nlm.nih.gov/pub/taxonomy, uncompressing to 54GB
2. Either or both of:
3. Download library sequences files (kraken2-build --standard / kraken2-build --special / kraken2-build --download-library or k2 download-library) to the DB directory
4. Add your own FASTA sequence data (kraken2-build --add-to-library or k2 add-to-library)
5. Build the database (kraken2-build --build or k2 build)
6. Clean up (remove temporary files)

This gives flexibility in how databases are built, and therefore it makes sense to have tools for (1), and some combination of (2), (3) and (4), balancing the need to not continually re-download the taxonomy files with the need to not pollute the history with intermediate products.

[bernt-matthias]

Download the NCBI taxonomy (kraken2-build --download-taxonomy or k2 download-taxonomy) to the database directory. This downloads some 7GB of data from ftp.ncbi.nlm.nih.gov/pub/taxonomy, uncompressing to 54GB

The data is available from the ncbi_taxonomy data table.

The procedure that you outlined is exactly what should happen for building custom entries with the DM https://github.com/galaxyproject/tools-iuc/blob/main/data_managers/data_manager_build_kraken2_database/data_manager/kraken2_build_database.xml

[pvanheus]

Just to clarify this most recent work: I have postponed working on the kraken2 database builder and, as per @bernt-matthias' suggestion, will draw on the similar code that is in the data manager. This PR simply updates to the latest kraken2 and adds support for custom databases (via a Directory input).