Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support caching layer for metadata traversals on the responder to answer future requests quickly #375

Open
Tracked by #371
hannahhoward opened this issue Mar 29, 2022 · 0 comments

Comments

@hannahhoward
Copy link
Collaborator

hannahhoward commented Mar 29, 2022

Once we build Graphsync metadata only requests, the next step is to build a way to cache them.

Rather than try to build this in go-graphsync, I'd like for it to be an optional dependency (passed through config options at initialization) that satisfies an interface something like this:

type MetadataWriter interface {
   Write(path ipld.Path, c cid.Cid, linkAction graphsync.LinkAction) error
   Finish() error
}

type MetadataCache interface {
   Has(root cid.Cid, selector ipld.Node) (bool, error)
   Get(root cid.Cid, selector ipld.Node) (graphsync.LinkMetadata, error)
   Begin(root cid.Cid, selector ipld.Node) (MetadataWriter, error)
}

When present, at the beginning of serving a metadata request, a response manager would check for the presence of a cached set of responses, and short circuit the process if they were present.

If not present, it call Begin and then as the response executed, call Write on each CID traversed, and call Finish at the end.

It's worth considering if there is a generalized version of this cache that might be useful for selector traversals in general, and perhaps this should be tackled in collaboration with IPLD stewards.

The path parameter exists on the Write call so that an implementer wanting to build a more semantic cache can do so.

Perhaps it even makes sense to add a parameter to write that captures the current selector at the given path, but this probably doesn't make sense to do until we have a way to convert back to a non-compiled selector at a given node.

@hannahhoward hannahhoward changed the title Optimizing with a caching layer that saves metadata traversals on the responder to answer future requests quickly Caching layer for metadata traversals on the responder to answer future requests quickly Mar 30, 2022
@hannahhoward hannahhoward changed the title Caching layer for metadata traversals on the responder to answer future requests quickly Support caching layer for metadata traversals on the responder to answer future requests quickly Mar 30, 2022
@hannahhoward hannahhoward moved this from Icebox to In Progress in Project Thunder (Interop) May 10, 2022
@hannahhoward hannahhoward moved this from In Progress to In Review in Project Thunder (Interop) May 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant