Collections - deleting and purging problematic

[jennaj]

Test history: https://usegalaxy.org/u/jen/h/test-coll-delete-purge

Odd behaviors:

• Permanently deleting a collection purges the included datasets, but the collection itself in the history list is not marked as being in a deleted state (all collection types).
• Deleting/Undeleting and Hiding/Unhiding functions correctly (all collection types).
• The deleted or purged states are not always propagated to other collections that contain the same datasets.

The problems started to layer up the more I tested in the history above. If this can't be reproduced by anyone else I'll add more step-by-step with distinct test cases/pics.

@jmchilton Is it a known? Important to fix for 18.01 or not?

I'm testing now in a 17.09 local, maybe the pre-release 18.01 branch, and at another public server to double check is not just at Main. Feedback about that soon.

[jmchilton]

There are a trio of fixes open against 18.01 right now that address a lot of issues with purging and deletion including a few of the things listed here.

• [18.01] More intuitive options for collection deletion. #5442
• [18.01] Ensure jobStateSummariesCollection exists;  #5444
• [18.01] Don't silently fail rest of multi-action queue when attempting to purge a collection. #5443

To be clear though even after these fixes if you delete a dataset that is in a collection - it will not delete the collection. Between the collection uploaders, #5365, and enhancements that hides contents by default in 18.01, and enhancements that allow using datasets in the tool form without unhiding them #5384 - I think it will become more and more common to not see datasets in the history first and then in collections - so the inverse of this - deleting the collection having the option to delete datasets (#5442) will make more sense. I had a conversation with @martenson about this on Gitter - I think people will need to "experience" collections in 18.01 and 18.05 for a while before this becomes clear.

Put another way - I think by 18.05 if you have a dataset that appears in the unhidden at the top-level of a history and it also appears in the collection - you are doing something kind of advanced and mostly discouraged. So it is kind of on you to make sure if you delete that dataset the collection also gets deleted.

I could be wrong and maybe we need to go further toward ensuring datasets only appear in one collection and only appear in the collection (for instance we could copy the HDAs when we create collections via the collection builder - they would work more like operation tools then). I could also prove to be wrong and it may make sense to add a check every time you delete an HDA to see if it is a collection and I guess prompt the user to see if they want to also delete the collection.

[jennaj]

Thanks for the info and links. I'll run through my use-cases after we update main to 18.01 to see how the new functions work.

This is the part that's probably going to be confusing for users (a guess!):

So it is kind of on you to make sure if you delete that dataset the collection also gets deleted.

But we could explain it the same way we do as for dataset "copies" >> Deleting one copy does not delete them all. And that is a good thing, imo. Once the functions to select/unpack datasets from collections goes out (when this "multiple dataset copies" situation might be more common -- but I agree, still advanced), we can make an FAQ to explain how it all works. Specifically, outline the usage details for working with "many copies of the same dataset", where any single copy could be in a collection (or collections) and also one or more distinct datasets. Cover how to perm delete to recover space, what counts toward quota, what doesn't, etc.

Thx!

[jennaj]

Closing -- will test in 19.05 once on public servers