Virtual datasets #8384
Description
Detailed Description
#8350 adds the storing of mag paths in the Postgres database. However, the source of truth for datasets is still the filesystem via the datastore. The next step will be to flip this source of truth. Datastores should still scan their folders and report datasets, but they should query Webknossos for accessing the datasets. This will also allow that datasets exist only in the database without a folder on disk, e.g. for fully remote datasets. Maybe this would also be a good opportunity to implement dataset IDs in the Webknossos <> datastore communication.
Motivation
- Remote datasets: users are using more and more remote datasets; it is a bit pointlesss to store a folder with a single json file on disk for that
- Object storage: in the future, we want to support object storage as primary storage for Webknossos datasets, which are technically also remote datasets
- Symlinks: symlinks currently create a lot of problems for the operation of Webknossos. With the capability to compose datasets in the database, we don't have to rely on symlinks anymore
Design Doc
https://www.notion.so/scalableminds/Design-Doc-Virtual-Datasets-177b51644c6380c5904fe91e605fd16d