Post Processing Merged BIOMs extended #2698
Conversation
analysis.build_files will now run an arbitrary number of post-processes defined in qiita.software_command.post_processing_cmd column. It's assumed that each post-process is a python script, with a defined Conda environment. It's also currently assumed that each post-process overwrites the merged BIOM file. A step removed from calling plugins, but getting closer.
Codecov Report
@@ Coverage Diff @@
## dev #2698 +/- ##
==========================================
+ Coverage 94.33% 94.34% +<.01%
==========================================
Files 166 166
Lines 19840 19888 +48
==========================================
+ Hits 18717 18764 +47
- Misses 1123 1124 +1
Continue to review full report at Codecov.
|
Paths changed to reflect new location of script.
qiita_db/analysis.py
Outdated
| cmd['script_params'].items()]) | ||
|
|
||
| # append fragments file and output dir parameters | ||
| params = "%s --fp_fragments=%s --output_dir=%s" %\ |
There was a problem hiding this comment.
Can you change the name of the parameter from --fp_fragments= to --fp_archive so it's more generic; at the end of the day this can be fragments or anything else the plugin stores in the database/archive
There was a problem hiding this comment.
No problem - changes already made to qiita and qp_deblur. One thing though, the command line util that strips fragments from the biom that aren't in the tree takes --fp_phylogeny and --fp_biom as options; do you think they might be too specific as well?
Originally, I thought a chain of commands would each in turn iterate over the same biom file, but since the second command will need a tree and a biom file as input, the database entry will need to be updated with the fp_s to the data, or the second command will need to assume the name and location of the inputs.
There was a problem hiding this comment.
I think those names internally in deblur are fine but you are right about those here. Also all the comments that mention phylogeny vs. extra file for analysis ...
fragments-named variables renamed to archives, to be more general. --fp_fragments option also renamed to fp_archives, where appropriate.
qiita_db/analysis.py
Outdated
| # suitable for other files as well. | ||
| output_dir = base_fp | ||
|
|
||
| fp_archive = '%s/archive.json' % output_dir |
There was a problem hiding this comment.
sorry for not catching this before: you should use join to create this path and also use the analysis id to be safe something like: fp_archive = join(output_dir, 'archive_%d.json' % self.id)
There was a problem hiding this comment.
Ah gotcha, platform independence path creation.
…iita into post_processing
Modified test to address name change from --fp_fragments to --fp_archive.
3 participants
analysis.build_files will now run an arbitrary number of post-processes defined
in qiita.software_command.post_processing_cmd column. It's assumed that each
post-process is a python script, with a defined Conda environment. It's also
currently assumed that each post-process overwrites the merged BIOM file. A
step removed from calling plugins, but getting closer.