Upload displays FTP files while the the transfer is in progress - no warnings, can be imported as partial datasets

[jennaj]

Not sure how we can trap an aborted/resumed transfer and keep it out of the Upload tool, but it seems like we could catch those transfers in progress (even if resumed) and not display them until done. This might be a bug.

These are semi-large data (~4 GB each BAM, 14 GB for the SAM). I have had to accept cert/resume a few times. Many users load much larger datasets. Resuming is likely common.

A few more details are included here: #102 (comment) and here galaxyproject/tools-iuc#1774 (comment)

Thoughts? @guerler @natefoo

Example

UI/Upload tool versus Filezilla, screenshots taken a few mins apart:

Then the transfer. Note that all four have started up, but only three remain in the Upload tool at this point. Results are in the different tickets above that including testing results for the final datatype assignment and related comments.

[jennaj]

After some datasets fully moved into the history, the FTP area updates to reflect those datasets still in an active/resumed transfer mode.

This is good because datasets loaded into the history, sticking around in the FTP area, was a prior bug fix by @guerler (thank you!!).

But also not-so-good, because no end users should not access/history load those transferring files yet - are incomplete.

That's it! - we can follow up from here and decide what to do.

[natefoo]

Because the upload process is external to Galaxy there is really no way for it to know that the upload is incomplete. Verifying that the upload succeeded is up to the user. In addition to the file size on the info page, users can use the hash tool to verify that the file uploaded matches their local file.

We could add size and modification time columns to the FTP file selection window, which would allow for a quick visual indicator, but to be sure, users should be paying attention to their FTP client.

[martenson]

We could add size and modification time columns to the FTP file selection window

it already has the size column

I agree that the users are in charge here, they need to make sure to understand handling of their own files.

[natefoo]

Oh... right... it's right there in the screenshot, ha.

[hexylena]

Maybe error prone but could decrease complaints?: when listing FTP files could we time.sleep(1) and compare filesizes? Ignoring those that have changed?

[natefoo]

I considered this but worried it'd result in many false positives for bursty connections.

[jennaj]

Ok, it sounds like we can't keep actively transferring files out of the Upload/FTP listing. That was my primary question -- if there was some way to do that.

Closing, but please re-open if a working solution comes up. Partial FTP uploads is the root issue with many downstream tool errors and are difficult to diagnose. I suspect some users don't even know it is going on (incomplete results from downstream tools) and so never report a problem, just think the results are odd when using Galaxy. What happens depends on the datatype and tool(s) used.

Agree, not resuming/tracking the load is one thing users should do -- but I don't think the common (or new!) sci user expects the datasets to show up in the GUI until the transfer is completed. Now, could still be partial, if they didn't resume and the connection was interrupted, but nothing we can do about that case on our side.

I'll add some usage-warning blurb to the FTP upload FAQ in the hub and link to the Troubleshooting FAQ. Might help, or at least gives a handle/link to explain proper usage when sending bug/biostars replies.