[WIP] Fix the most obvious python3 errors in data managers

[mvdbeek]

These should be fairly uncontroversial (but I haven't tested them all ...).

• don't open file in binary mode when writing dumps of dictionaries, which are unicode on py3
• replace print statement with print function
• replace reduce with functools.reduce (which just points at the reduce builtin on py2)

FOR CONTRIBUTOR:

• - I have read the CONTRIBUTING.md document and this tool is appropriate for the tools-iuc repo.
• - License permits unrestricted use (educational + commercial)
• - This PR adds a new tool or tool collection
• - This PR updates an existing tool or tool collection
• - This PR does something else (explain below)

FOR REVIEWER:

• .shed.yml file ok
  • Toolshed user iuc has access to associated toolshed repo(s)
• Indentation is correct (4 spaces)
• Tool version/build ok
• <command/>
  • Text parameters, input and output files 'single quoted'
  • Use of <![CDATA[ ... ]]> tags
  • Parameters of type text or having optional="true" attribute are checked with if str($param) before being used
• Data parameters have a format attribute containing datatypes recognised by Galaxy
• Tests
  • Parameters are reasonably covered
  • Test files are appropriate
• Help
  • Valid restructuredText and uses <![CDATA[ ... ]]> tags
• Complies with other best practice in Best Practices Doc

[mvdbeek]

On the brightside I worked a bit on fixing data manager testing:

======================================================================
FAIL: ( testtoolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_dbkeys_all_fasta/data_manager_fetch_genome_all_fasta_dbkey/0.0.2 ) > Test-1
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/mvandenb/src/galaxy/test/functional/test_toolbox.py", line 97, in test_tool
    self.do_it(tool_version=tool_version, test_index=test_index)
  File "/Users/mvandenb/src/galaxy/test/functional/test_data_managers.py", line 28, in do_it
    verify_tool(tool_id, self.galaxy_interactor, resource_parameters=resource_parameters, test_index=test_index, tool_version=tool_version)
  File "/Users/mvandenb/src/galaxy/lib/galaxy/tools/verify/interactor.py", line 705, in verify_tool
    raise e
  File "/Users/mvandenb/src/galaxy/lib/galaxy/tools/verify/interactor.py", line 701, in verify_tool
    job_stdio = _verify_outputs(testdef, test_history, jobs, tool_id, data_list, data_collection_list, galaxy_interactor, quiet=quiet)
  File "/Users/mvandenb/src/galaxy/lib/galaxy/tools/verify/interactor.py", line 896, in _verify_outputs
    raise JobOutputsError(found_exceptions, job_stdio)
galaxy.tools.verify.interactor.JobOutputsError: History item  different than expected, difference (using diff):
( /Users/mvandenb/src/shed_tools/testtoolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_dbkeys_all_fasta/872d7cf608f9/data_manager_fetch_genome_dbkeys_all_fasta/test-data/phiX174.data_manager_json v. /var/folders/df/6xqpqpcd7h73b6jpx9t6cwhw0000gn/T/tmpi21sgxapphiX174.data_manager_json )
--- local_file
+++ history_data
@@ -1 +1 @@
-{"data_tables": {"all_fasta": [{"path": "anoGam1.fa", "dbkey": "anoGam1", "name": "A. gambiae Feb. 2003 (IAGEC MOZ2/anoGam1) (anoGam1)", "value": "anoGam1"}]}}+{"data_tables": {"all_fasta": [{"value": "anoGam1", "dbkey": "anoGam1", "name": "anoGam1", "path": "anoGam1.fa"}]}}

----------------------------------------------------------------------
Ran 3 tests in 44.852s

FAILED (errors=1, failures=1)

[mvdbeek]

Putting this into WIP, will set up actual testing soon

[jennaj]

@mvdbeek Would you be able to help me understand how this will roll out a bit more? I had problems with the Gemini DM on 5/1/2019, the latest version from the tool shed.

Do the changes get applied per-DM? If so, has Gemini been updated or are there distinct tickets for different DMs? Should I make one for Gemini or is there some master ticket with a "to-do" list in it for the DMs that need an update (couldn't find one -- maybe missed it?)

I don't see any recent changes in the repo about this and @natefoo thinks this is why our index runs for Gemini are failing: https://github.com/galaxyproject/tools-iuc/commits/master/data_managers/data_manager_gemini_database_downloader

ping @bgruening getting Gemini data into CVMFS is likely dependent on this. am still doing another test run today, just to double check

[mvdbeek]

I had problems with the Gemini DM on 5/1/2019, the latest version from the tool shed.

On python 3 ? This is are exclusively python 3 fixes here. We'll need a new planemo release with a new xsd to mark managers as python 3 compatible.

[jennaj]

@mvdbeek Yes, under Py3. We run the DMs on a special mirror of our Test server, which was updated early for, well, to test what is working and what isn't under version 3 :)

So the fix is not DM specific if am understanding correctly.

am cc-ing @jmchilton to keep him updated about the dependency.

Thanks for the extra info!

[mvdbeek]

It's done, we've got a new planemo release, I'll try to find some time to go through the data managers. Some of them don't have tests, so it's a little more work.

[jennaj]

@mvdbeek Super, thank you!!

Also, if there are part of the DM updates that others can help with, could distribute those tasks to more people. @jmchilton and I have a project going on in our lab where we have a few bugs or update fixes assigned per week so that everyone is involved in maintenance activities (not just a few). Is generally liked by the developers -- get to work on something short and clear endpoint with a bonus of learning something new in many cases (bio or technical). Feedback is often that it feels good to actually get something smaller completely done while also working on the longer-term infrastructure projects that don't have as many done moments. Win-win :)

[nsoranzo]

@mvdbeek Anything not already merged as part of the #2680-#2683 PRs? Or can we close this?

[mvdbeek]

There are still some that need fixing, I just focused on the high profile ones for now.