Next Steps to Bring Your Own Storage and Data

[jmchilton]

#18127 is a monster PR - a classic John framework (times two) and in rare form contains both a wide variety of well tested plugins and deep documentation - but it still could use a lot of polish and there are a lot of things we could potential do to make it more palatable for the usegalaxy * servers.

Verify before Create

#18222

• Implement “Verification” green/red dots before creating a plugin (think like wifi/connection testing vibes). Need to check a configuration before persisting it.
  • Green/red dot for plugin verification.
  • Green/red dot for variables expand to create valid configuration.
  • Green/red dot for connection established.
  • Test then create, would prevent a lot of misconfigured instances from being created I think.
• Allow verification re-checks after the fact from the manage menu. Verify the persisted thing instead of the payload.

OAuth Use Cases

• Work through a full oauth configuration.
  • Revise Dropbox file source plugin to use modern, functional APIs.
  • Work through functional template examples for Dropbox and Google Drive that work using auth.
  • Decide what the framework needs to support these use cases and implement it.

UI QOL Enhancements

• Disable form submission during testing and create/update/upgrade.
• User object store index should allow selecting an object store as the user default - feels odd to still have to navigate to another user option to make that the default.
• Revise removing instance - removing a plugin should maybe give a few options. The API makes a distinction between “hidden” (still works but not shown in the file tree for instance) - which is useful if you are done adding to something but you still want old dataset or old URIs to work, “inactive” which disable the use of the plugin (maybe you want to go a step further and disable the possibility for charges or whatever), and “purged” which also deletes all the secrets associated with the plugin. I’m not sure what @martenson will let me get away with.
• There is a log of good text in the description of the YAML files but it doesn’t appear in the form. It should maybe appear at the top of the form inside the the UI card.
• Implement a help section the way tools have and place below the form. Markdown instead through obviously.
• Implement “Agree to Terms” of some kind - make the downsides of sharing credentials clear.

Underlying Plugin Improvements

• Revise timeout handling for S3FS file source plugin. There should be a global variable that is defined at the Galaxy level and applies to each instance if a value isn’t set. This feels better than having each plugin instance define a magic constant - which seems to be what main is doing unfortunately (https://github.com/galaxyproject/usegalaxy-playbook/blob/main/env/common/templates/galaxy/config/file_sources_conf.yml.j2)

Testing:

• Unit or integration testing for disable_templating. Make sure Cheetah templating doesn’t apply after the fact to the Jinja templated file sources.
• Extend user object store selenium test cases with selection and use and verify
• Extend user file source selenium test cases with verify output
• Selenium tests for upgrading user object stores and user file sources.

Collect and work through existing issues around object store selection & private object stores

• Batch Move Between Object Stores:
• Fix for anonymous users - Object store selection is available to anon users (but only works partially?) #18153
• Warn before submitting to private object stores if it isn't going to work.
• TODO: we've seen more of these.

[itisAliRH]

I configured my local using the Hands-On Production Examples from the main PR description and have the following comments on it in addition to your comment:

Configuration and Documation

• I created config/file_source_templates.yml and config/object_store_templates.yml but the object store buttons (history and tool) are not showing up (because object_store_allows_id_selection from the config object is false).
• Also created object_store_conf.yml file using examples from object_store_conf.sample.yml but only the type: distributed ones are working (neither disk nor hierarchical)

UI

• I have created a new object store using AWS credentials in user settings and ran a tool by choosing this location to store the results; but it raised an error and the dataset detailed fetcher is returning 500 only and it blocks the Ui (the error is not handled here and it brakes the UI but not related to these changes)
• Preferred storage location is not visible to the user until hovering on the icon (I'd put the selected name next to its icon)
• It's not clear what will happen to the existing datasets in history when the user changes the preferred location store. Either if they are moving to the new place or only the new datasets will be stored in the new location, we should inform them (showing a storage icon in the dataset card with the name of the object store that it is stored in would be an idea in this case)
• I found some things in the user preferences that could be improved and will work on enhancing this if you agree in follow up PRs.

[jmchilton]

It's not clear what will happen to the existing datasets in history when the user changes the preferred location store.

This is not related to this PR right? You could say the same thing about storage options on main. I've updated the hover message to be bold in 24.0 (both of these message are from 24.0 and usegalaxy.org):

The message during creation also makes it clear the selection applies only to new datasets.

I wonder if making this a history option was a mistake - maybe it should just be a user option and an option on the tools and workflows.

[jmchilton]

[ ] I created config/file_source_templates.yml and config/object_store_templates.yml but the object store buttons (history and tool) are not showing up (because object_store_allows_id_selection from the config object is false).

[ ] Also created object_store_conf.yml file using examples from object_store_conf.sample.yml but only the type: distributed ones are working (neither disk nor hierarchical)

Lets move this conversation to #18157 ? I think both of those points are related to needing a distributed object store for this to work (which again main and EU both have). I think I've discussed this there but I will definitely updated the documentation inside the PR right now to make the current state of affairs clear.

[itisAliRH]

This is not related to this PR right? You could say the same thing about storage options on main. I've updated the hover message to be bold in 24.0 (both of these message are from 24.0 and usegalaxy.org):

You're right, my bad. I apologize! Having an icon next to each dataset would make it easier to understand their storage location.
Bolding the important parts is a good way.

[martenson]

I wonder if making this a history option was a mistake - maybe it should just be a user option and an option on the tools and workflows.

It is a useful option, but maybe for the most advanced users out of the four locations. However the most visible.

[jmchilton]

Updated Docs about the distributed object store:

[jmchilton]

@martenson , @itisAliRH I appreciate the comments regarding the history default indicator - including "Preferred storage location is not visible to the user until hovering on the icon (I'd put the selected name next to its icon)". It is clearly a pressing issue. In response to @itisAliRH's "You're right, my bad. I apologize!" - there is no need to apologize - everyone says this - there is an issue the help text does not resolve and I'd love to work with the UI/UX team on figuring out a solution.

Also though this is a pre-existing issue - I've created a discussion for it here #18163. Lets discuss this the context of object store selection instead of user object stores - scratch storage on main has the same issue for instance. None of this is an attempt by me to side step this though - we've got to get this resolved more urgently for it being an issue under more conditions than just user defined object stores.

[jmchilton]

I have created a new object store using AWS credentials in user settings and ran a tool by choosing this location to store the results; but it raised an error and the dataset detailed fetcher is returning 500 only and it blocks the Ui (the error is not handled here and it brakes the UI but not related to these changes)

This seems like a combination backend/frontend issue:

For the backend, any chance you can track down some kind of backend stack traces to help track down the problem?

In terms of the frontend...

I've seen a similar problematic lock up behavior on bad object stores - but for me it kills the backend - but I think the source is probably still the problamatic loop related to exception handling in the client (is there a pinia store that is looping infinitely or something do you think?).

My gut is also this is a pre-existing issue in some ways but this code makes it trivial to hit the issue.

I think every time I have observed this behavior - I had a misconfigured object store that got serialized and setup. My top priority in the above list is testing user defined objects before they are created for a reason (I've made some good initial progress this morning - 0bb53fd - it provides some nice feedback on a variety of issues). It would catch so many problems. Even once we validate object stores on creation though... the core frontend issues need to be resolved... invariably keys will be invalidated, remote APIs will disappear, connections will be bad - and we need to not crash the UI or the backend in those cases.

[jmchilton]

@itisAliRH In response to: "I found some things in the user preferences that could be improved and will work on enhancing this if you agree in follow up PRs.". I'm still working hard on the UI for this ahead of the GCC - any advice would be great. I'm happy to accept help too but if you're serious I would scope out the issues so we can divide them up and so I can warn you if you're working on something I'm touching.