HydroShare Bulk Upload

[rajadain]

Overview

Previously we uploaded files one by one to HydroShare. When overwriting, we would have to check if the file existed and delete it before uploading the same file again. This took a long time, slowed by the need to make three calls per file.

With the new "unzip overwrite" functionality introduced in hydroshare/hydroshare#2943, we can do "bulk uploads" by zipping all the files into one archive, sending that once, and unzipping on HydroShare, overwriting any existing files.

This reduces the time taken to do a HydroShare Export.

Connects #3016

Notes

For some reason, mmw_project_snapshot.json is deleted, not overwritten, on subsequent uploads. This is being discussed here: hydroshare/hydroshare#2860 (comment).

Testing Instructions

• Ensure you are setup for working with HydroShare (talk to me if you're not)
• Check out this branch and restart Celery

  vagrant ssh worker -c 'sudo service celeryd restart'
  

• Create a new project and export it to HydroShare
  • Ensure that the project is created in HydroShare
  • Ensure the modal links to the new project in HydroShare
  • Ensure it has a CSV for every table, a GeoJSON and Shapefile for the area of interest, a GMS file for every MapShed scenario, the MMW Spreadsheet, and a mmw_project_snapshot.json file
• Export the project again
  • Ensure it succeeds
  • Ensure all files are still in HydroShare
• Add a new scenario and export again
  • Ensure the new scenario files are exported to HydroShare
• Change a scenario and export again
  • Ensure the scenario files in HydroShare are updated to reflect changes

[rajadain]

This is now ready for testing, as the bug encountered (and described in the Notes section above) has been remedied: hydroshare/hydroshare#2860 (comment)

[kellyi]

Had some trouble getting HydroShare set up. Reassigning to @caseycesari so that I don't block this one.

[caseycesari]

Everything worked without issue! Nice job.

[rajadain]

Thanks! Merging this now.