Skip to content

Post Processing Merged BIOMs extended #2698

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 19 commits into from
Nov 1, 2018
Merged

Post Processing Merged BIOMs extended #2698

Changes from 1 commit
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
519834b
Post Processing Merged BIOMs extended
Oct 10, 2018
848f30d
Removed two print statements
Oct 29, 2018
9a0e8a9
Move worker.py to new test/support_files subdirectory.
Oct 29, 2018
47c34b7
Child-Process Environment Instantiation generalized
Oct 29, 2018
62922ef
Moved worker.py to new location
Oct 29, 2018
7477c46
Fixes so tests will execute correctly.
Oct 29, 2018
6e604f5
Mods to support needs of updated Deblur plugin
Oct 30, 2018
f2abd23
Working directory for post-processing changed
Oct 30, 2018
1a253b4
flake8
Oct 30, 2018
5bf7ac7
worker.py path adjusted to reflect cwd in Travis
Oct 31, 2018
9d849e9
Resolve spelling error
Oct 31, 2018
c0ba812
fragments renamed to archives
Oct 31, 2018
912ceef
flake8
Oct 31, 2018
4d2880a
Replaced *NIX-specific patch creation w/join()
Oct 31, 2018
6252c98
Merge branch 'dev' into post_processing
charles-cowart Oct 31, 2018
4b1ee2c
Added analysis id to fp_archive.json
Oct 31, 2018
0dd79f9
Merge branch 'post_processing' of https://github.com/charles-cowart/q…
Oct 31, 2018
f61d37a
Modified test to address changes to parameters
Oct 31, 2018
d65103e
Removed unnecessary import
Nov 1, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fragments renamed to archives 
fragments-named variables renamed to archives, to be more general.
--fp_fragments option also renamed to fp_archives, where appropriate.
  • Loading branch information
Charles Cowart authored and Charles Cowart committed Oct 31, 2018
commit c0ba812074d8b97835d783f7a95ab1ede5b94054
28 changes: 14 additions & 14 deletions qiita_db/analysis.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -989,42 +989,42 @@ def _build_biom_tables(self,
# element of the list is a dictionary containing the Conda env
# to use, the script to run, and a dictionary of parameters.
if post_processing_cmds:
# assuming all commands require fragments, obtain
# fragments once, instead of for every cmd.
# assuming all commands require archives, obtain
# archives once, instead of for every cmd.
features = load_table(biom_fp).ids(axis='observation')
features = list(features)
fragments = Archive.retrieve_feature_values(
archives = Archive.retrieve_feature_values(
archive_merging_scheme=archive_merging_scheme,
features=features)

# remove fragments that SEPP could not match
fragments = {f: loads(fragments[f])
# remove archives that SEPP could not match
archives = {f: loads(archives[f])
for f, plc
in fragments.items()
in archives.items()
if plc != ''}

# since biom_fp uses base_fp as its location, assume it's
# suitable for other files as well.
output_dir = base_fp

fp_fragments = '%s/fragments.json' % output_dir
fp_archive = '%s/archive.json' % output_dir
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry for not catching this before: you should use join to create this path and also use the analysis id to be safe something like: fp_archive = join(output_dir, 'archive_%d.json' % self.id)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah gotcha, platform independence path creation.


with open(fp_fragments, 'w') as out_file:
dump(fragments, out_file)
with open(fp_archive, 'w') as out_file:
dump(archives, out_file)

for cmd in post_processing_cmds:
# assume fragments file is passed as:
# --fp_fragments=<path_to_fragments_file>
# assume archives file is passed as:
# --fp_archive=<path_to_archives_file>
# assume output dir is passed as:
# --output_dir=<path_to_output_dir>

# concatenate any other parameters into a string
params = ' '.join(["%s=%s" % (k, v) for k, v in
cmd['script_params'].items()])

# append fragments file and output dir parameters
params = "%s --fp_fragments=%s --output_dir=%s" %\
(params, fp_fragments, output_dir)
# append archives file and output dir parameters
params = "%s --fp_archive=%s --output_dir=%s" %\
(params, fp_archive, output_dir)

# if environment is successfully activated,
# run script with parameters
Expand Down