Additional Processing Command added to Software_Command #2694

charles-cowart · 2018-10-05T01:31:22Z

Adds an 'addtl_processing_cmd' property to Command, to obtain additional processing steps needed to merge BIOMs.

* fix #1999 * fix template error * rm maintenance conditional * some improvements * fix #1053

@ElDeveloper

* fix Keemei (#2510) * rename preprocessed_data_id artifact-id * prints to debug * Keemei fix (#2517) * fix Keemei * delete keemei * keemei delete * rewrite_fastq=True * rmtree * add : * testing EBI * add secure * travis tests * @ElDeveloper key! * Fix typo

@ElDeveloper

* fix Keemei (#2510) * fix #2321 * fixing get studies * fix qiita_db * addressing @ElDeveloper comments * fixing error with skipIf

* fix Keemei (#2510) * updating illumina models

* fix Keemei (#2510) * Keemei fix (#2517) * fix Keemei * delete keemei * keemei delete * Update index.rst

* fix Keemei (#2510) * plugin-api * improve format

* fix Keemei (#2510) * fix #2321 * fixing get studies * fix qiita_db * fix #1810 * adding self.ascp_pass * fix flake8 * flake8

* DOC: Cleanup of JavaScript libraries and licenses There were a few JavaScript files that were not used anywhere and we were missing a handful of license files. I've cleaned up all of that (and for consistency made all the "licence" into "license"). Fixes #2535 * DOC: Document all the JS files * BUG: Add natural sorting for data tables

* fix Keemei (#2510) * fix int/str errors with workflows * adding a patch to fix all ints parameters * all -> any patch 64

Strips UTF-8 characters from study titles UTF-8 characters that are not also printable ASCII characters are stripped from new study titles, as well as existing studies flagged 'public'. This is because study titles are passed as metadata to new analyses, and some analysis packages may not work well with them.

Merging charles-cowart->66.py with another topic branch 66.py

fix #2651 and partial #2652

@ElDeveloper

* fixes #2629 changed button text from Select None to Unselect All * adding ssh backend support * added paramiko * install scp package for python * testing scp directly in travis * testing scp directly in travis * testing scp directly in travis * testing scp directly in travis * testing scp directly in travis * testing scp directly in travis * addressing @ElDeveloper comments * fix _get_valid_files * glob.glob -> glob * fix error * fix error * bringing back download_remote * improving tests * rm docstrings as requested by @ElDeveloper * fixing tests - remote_files->local_files

Merged 66.py with that from #2660

Code wrapped in a for loop and iterates through a list of known Study status-types; seems a little cleaner than union-ing multiple result sets.

Software.Command extended to support additional metadata regarding extra processing that may be required to merge BIOMs.

antgonza

There are errors in the build and I think I highlighted the changes in here, please fix, thanks.

antgonza · 2018-10-05T12:27:22Z

qiita_db/test/test_analysis.py

        dflt_analysis = user.default_analysis
+        aid = 4
+        if addtl_processing_cmd:
+            cmd_id = qdb.artifact.Artifact(aid).processing_parameters.command


note that this is cmd, and the line below is cmd_id

I just broke it into two lines to pass flake8. In the next line, cmd_id is reassigned to cmd_id.id. I just didn't want to create an additional variable. It's probably bad form, now that I look at it.

antgonza · 2018-10-05T12:28:09Z

qiita_db/test/test_analysis.py


        self._wait_for_jobs(new)
+
+        with qdb.sql_connection.TRN:


you need to add the same if than above to only enter this block when needed ...

Added conditional check, where it was missing

Last-minute comment did not meet flake8's approval.

_methods renamed to Qmethods

codecov-io · 2018-10-05T23:46:29Z

Codecov Report

Merging #2694 into dev will increase coverage by 0.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##              dev    #2694      +/-   ##
==========================================
+ Coverage   94.32%   94.34%   +0.01%     
==========================================
  Files         166      166              
  Lines       19747    19788      +41     
==========================================
+ Hits        18626    18668      +42     
+ Misses       1121     1120       -1

Impacted Files	Coverage Δ
qiita_db/test/test_analysis.py	`98.54% <100%> (+0.37%)`	⬆️
qiita_db/test/test_software.py	`99.82% <100%> (ø)`	⬆️
qiita_db/analysis.py	`96.76% <100%> (+0.07%)`	⬆️
qiita_db/software.py	`98.05% <100%> (+0.01%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fd47ca8...6b49822. Read the comment docs.

antgonza

Good progress but a few comments.

qiita_db/software.py

qiita_db/analysis.py

qiita_db/support_files/patches/67.sql

antgonza · 2018-10-06T14:20:06Z

qiita_db/support_files/qiita-db.dbs

 ALTER TABLE oauth_software ADD CONSTRAINT fk_oauth_software FOREIGN KEY ( client_id ) REFERENCES oauth_identifiers( client_id )    ;

 ]]></string>
 		</script>


This is no right, correct? There should have been changes in this file, cause we added a new column. Could you double check?

Just added a line. Just to clarify, this is an internal format right? I don't see a schema reference at the top.

I think is fine now but here is more information: https://github.com/biocore/qiita/blob/master/CONTRIBUTING.md#making-database-changes, will wait for confirmation.

qiita_db/support_files/qiita-db.html

qiita_db/test/test_analysis.py

scripts/qiita-recover-jobs

qiita_db/software.py

Comments modified as requested. 'Q-functions' in qiita-recover-jobs have been reverted to begin with '_'.

Modified qiita-db.dbs to include additional column defined in software_command table.

antgonza

Getting closer but some minor things ...

antgonza · 2018-10-07T13:20:47Z

qiita_db/support_files/qiita-db.dbs

 ALTER TABLE oauth_software ADD CONSTRAINT fk_oauth_software FOREIGN KEY ( client_id ) REFERENCES oauth_identifiers( client_id )    ;

 ]]></string>
 		</script>


I think is fine now but here is more information: https://github.com/biocore/qiita/blob/master/CONTRIBUTING.md#making-database-changes, will wait for confirmation.

scripts/qiita

scripts/qiita-env

antgonza and others added 30 commits February 20, 2018 09:37

fix #1067

0c9598b

addressing @ElDeveloper comments

58858ae

Merge pull request #2507 from antgonza/fix-1067

82c546a

fix #1067

fix #2512

cbebd76

fix #1999 (#2514)

7659c88

* fix #1999 * fix template error * rm maintenance conditional * some improvements * fix #1053

addressing @ElDeveloper comments

617831b

Merge branch 'master' of https://github.com/biocore/qiita into dev

ff87431

Merge branch 'dev' of https://github.com/biocore/qiita into dev

b1924c3

adding secret

7462867

Merge pull request #2515 from antgonza/fix-2512

0994537

fix #2512

WIP: EBI-ENA test (#2516)

7b63710

* fix Keemei (#2510) * rename preprocessed_data_id artifact-id * prints to debug * Keemei fix (#2517) * fix Keemei * delete keemei * keemei delete * rewrite_fastq=True * rmtree * add : * testing EBI * add secure * travis tests * @ElDeveloper key! * Fix typo

fix secure

bb3cd2f

rm single quote

baf7739

upstrem dev

5e0f643

Fix 2321 (#2522)

ae5d1a3

* fix Keemei (#2510) * fix #2321 * fixing get studies * fix qiita_db * addressing @ElDeveloper comments * fixing error with skipIf

Merge branch 'dev' of https://github.com/biocore/qiita into dev

1d54381

Illumina models 0318 (#2532)

4ba96ab

* fix Keemei (#2510) * updating illumina models

adding secure

6a06964

Merge branch 'dev' of https://github.com/biocore/qiita into dev

b9a2cba

Update index.rst (#2533)

232c21a

* fix Keemei (#2510) * Keemei fix (#2517) * fix Keemei * delete keemei * keemei delete * Update index.rst

Plugin API Doc (#2534)

56aaeac

* fix Keemei (#2510) * plugin-api * improve format

Fix 1810 (#2523)

2dcdd93

* fix Keemei (#2510) * fix #2321 * fixing get studies * fix qiita_db * fix #1810 * adding self.ascp_pass * fix flake8 * flake8

Merge branch 'dev' of https://github.com/biocore/qiita into dev

39cc48d

adding initial download/redbiom info

b268a42

adding docs

77af7fd

addressing @ElDeveloper comments

c1c8e9b

Fix aid str workflow (#2537)

e7c5832

* fix Keemei (#2510) * fix int/str errors with workflows * adding a patch to fix all ints parameters * all -> any patch 64

Merge branch 'dev' of https://github.com/biocore/qiita into dev

d1d5e44

fix #2519

e356545

Charles Cowart and others added 14 commits August 31, 2018 17:19

Resolving conflict with another new patch file.

ad34d04

Merge branch 'dev' of http://github.com/biocore/qiita into dev

c73bd55

Merging charles-cowart->66.py with another topic branch 66.py

Rename file

8aed862

67.py renamed to 66.py. minor edit to edit_study.html

f6cfb15

Update study title only if UTF-8 characters were removed.

043f5eb

Merge pull request #2657 from antgonza/fix-2651

1bd63f8

fix #2651 and partial #2652

Merge branch 'dev' of http://github.com/biocore/qiita into dev

7b5f42d

Merged 66.py with that from #2660

UTF-8 stripper now processes all study-types.

6b099ca

Code wrapped in a for loop and iterates through a list of known Study status-types; seems a little cleaner than union-ing multiple result sets.

66.py now passes flake8 tests.

ebd99b0

Merge branch 'dev' of https://github.com/biocore/qiita into dev

2682c91

Additional Processing Command functionality

34aca8b

Software.Command extended to support additional metadata regarding extra processing that may be required to merge BIOMs.

Additional Processing Command functionality

85ae011

Software.Command extended to support additional metadata regarding extra processing that may be required to merge BIOMs.

antgonza reviewed Oct 5, 2018

View reviewed changes

Charles Cowart added 4 commits October 5, 2018 11:20

Fixed bug

15aad87

Added conditional check, where it was missing

Pass flake8

b8337eb

Last-minute comment did not meet flake8's approval.

Fix _method in Click

4f97e8b

_methods renamed to Qmethods

Specified Click 6.7 as a dependency

455b88f

antgonza requested changes Oct 6, 2018

View reviewed changes

Charles Cowart added 4 commits October 6, 2018 17:47

Misc cleanup

bf45f0e

Comments modified as requested. 'Q-functions' in qiita-recover-jobs have been reverted to begin with '_'.

Modified qiita-db.dbs

93efdb6

Modified qiita-db.dbs to include additional column defined in software_command table.

Added comment metadata to new column.

1ed506d

Refined tests

50cf38a

antgonza requested changes Oct 7, 2018

View reviewed changes

Charles Cowart added 2 commits October 7, 2018 11:43

Replaced Qtest w/_test

b1e0f8b

Reverted functions beginning w/Q to '_'.

6b49822

antgonza approved these changes Oct 8, 2018

View reviewed changes

antgonza merged commit 06e6d9e into qiita-spots:dev Oct 8, 2018

Uh oh!

Additional Processing Command added to Software_Command #2694

Additional Processing Command added to Software_Command #2694

Uh oh!

Conversation

charles-cowart commented Oct 5, 2018

Uh oh!

antgonza left a comment

Choose a reason for hiding this comment

Uh oh!

antgonza Oct 5, 2018

Choose a reason for hiding this comment

Uh oh!

charles-cowart Oct 5, 2018

Choose a reason for hiding this comment

Uh oh!

antgonza Oct 5, 2018

Choose a reason for hiding this comment

Uh oh!

codecov-io commented Oct 5, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

antgonza left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

antgonza Oct 6, 2018

Choose a reason for hiding this comment

Uh oh!

charles-cowart Oct 7, 2018

Choose a reason for hiding this comment

Uh oh!

antgonza Oct 7, 2018

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

antgonza left a comment

Choose a reason for hiding this comment

Uh oh!

antgonza Oct 7, 2018

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

codecov-io commented Oct 5, 2018 •

edited

Loading