Cannot Download Directory

[adamrp]

I just generated a processed data for a study I'm working on (study ID 1902, pd_id 152). There are three downloadable files: the biom, the log, and the directory. I can download the biom and the log, but get a 404 when I try to download the directory.

[antgonza]

@adamrp not sure if this is a qiita.colorado.edu specific bug or a code/resources bug. The folder that is pointing to exists but nginx is not expecting to list a folder. Now, these folders have 3+ files but I think we should only allow their gzip version. So perhaps the solution is that we gzip all existing files and update the links in a patch.

[adamrp]

The download handler was never set up to work with directories. Currently, the download handler works by looking at the filepaths table. Some of the paths in that table are directories. If a user has access to a directory in the filepaths table and they try to download any of the files in that directory (or subdirectories) it will work. However, if they try to download the directory itself, it won't work. The simplest way to allow the download of directories would be to gzip the directory and add the gzipped file to the filepaths table.

[adamrp]

I don't think we want to gzip everything because there is code that assumes that files in that table are not gzipped, but if the filepath is of a "directory" type, then we might need to do something smarter if a download of the directory is requested

[antgonza]

What about automatically generating an index.html page with links to the files?

[wasade]

👍, like FileLinks

On Thu, Jun 4, 2015 at 3:21 PM, Antonio Gonzalez notifications@github.com
wrote:

What about automatically generating an index.html page with links to the
files?

—
Reply to this email directly or view it on GitHub
#1132 (comment).

[josenavas]

I think the smart think to do when a user request to download a file of type filepath will be to do a "tar" (not gziped) and then serve that. If it is of type directory it will be easy to detect.

[antgonza]

The problem with that option is that we will need to create a job so tornado doesn't block while taring ...

[wasade]

Creating/submitting the job is straight forward :)

On Thu, Jun 4, 2015 at 3:56 PM, Antonio Gonzalez notifications@github.com
wrote:

The problem with that option is that we will need to create a job so
tornado doesn't block while taring ...

—
Reply to this email directly or view it on GitHub
#1132 (comment).

[antgonza]

That's not the issue ... the question is how are we gonna deliver (page with possible download [possible duplication] or email [also duplication])?

[wasade]

A little spinny thing until the job completes? pubsub is already there, and
the moi list is embeddable in a div now too

On Thu, Jun 4, 2015 at 4:03 PM, Antonio Gonzalez notifications@github.com
wrote:

That's not the issue ... the question is how are we gonna deliver (page
with possible download [possible duplication] or email [also duplication])?

—
Reply to this email directly or view it on GitHub
#1132 (comment).

[antgonza]

How to avoid duplication of tar or tgz files?

[wasade]

Where is the duplication happening, and do we care about long term storage?
Could just put in a scratch space and let the system clean up the tgz 1-2
weeks out since that wouldn't delete the actual files packed up

On Thu, Jun 4, 2015 at 4:06 PM, Antonio Gonzalez notifications@github.com
wrote:

How to avoid duplication of tar or tgz files?

—
Reply to this email directly or view it on GitHub
#1132 (comment).