Skip to content

New storage backend API based on ResourceContexts #27

Open

Description

Currently the storage backend API isn't very formalized and it has some problems which would be nice to fix.

  • Firstly, the backend entrypoint it doesn't support ResourceContexts natively, instead it's just a single method which provides the user's callback with an open "dataset root". In Use ResourceContexts.jl for resource handling #12 I needed to invent ResourceContexts.enter_do to expose the data handle from within the do block but enter_do is kind of hacky/complex/inefficient as it needs to use a separate task stack.
  • Secondly, the "dataset root" type which is opened by the entrypoint has an informally defined API for use with our BlobTree abstraction; it's kind of complex and poorly distinguished from the internals of BlobTree itself, which makes it hard to implement and leads to suboptimal code reuse between backends.

Likely we should have an AbstractDataStorage type and perhaps define an open_storage function to go with this to solve the first problem. The second problem needs some joint refactoring of the existing tree storage backends to extract the common code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions