Use Calamus in renku

[jsam]

Currently, we have a pretty magical JSON-LD serialization layer with several layers of indirection which is quite a challenge to maintain. To make things more maintainable and more stable I would like to have this as its own library which is properly tested and addresses all the current outstanding tickets (#548, #547).

Some, things to consider might be:

• Top-level context squashing
• Flexible way of handling/validating container types
• Schema migrations
• Attribute migrations
• Attribute validations
• Optionality of attributes
• (De) Serialization (to and from the given format)

Any draft proposals to address the current situation is more than welcome.

[Panaetius]

I think we should restrict ourselves to json-ld 1.1, as it has some nice new feature, such as scoped and nested contexts. If we rely on those features, not using 1.1 will break things.

This means we have to set @version in our contexts.

[Panaetius]

We should have support for xsd data types in jsonld, so

schema:Something: {
    "@type": "xsd:float",
    "@value": "3.14159"
}

gets deserialized correctly as a float value (instead of as a dictionary as in the current implementation)

[rokroskar]

This would be awesome!

[Panaetius]

General outline of problems we would like to solve (not necessarily specific to a separate library, but also to how it would/should be used in renku-python):

• JSON-LD is slow right now. On import we have to pre-calculate the whole context for all classes, each time, even if those classes aren't really used. Ideally we should only calculate json-ld contexts etc. when asjsonld is called (maybe with some caching). This probably means we can't rely on attrs, at least not in the way we currently do.
• Contexts of children should be propagated to parents at runtime so the json-ld context only contains relevant contexts (instead of always having contexts for all possible children).
• Don't go from commits/activities to entities as we do now. Entities (Dataset etc.) should be the main nodes, annotated with wprov stuff instead of the other way around
• Save json-ld after every renku command, keeping whole graph on disk inside .renku somewhere, we don't need to (re-)calculate this stuff every time if it never changes for a commit.
• Save the data as a flat list of nodes that refer to each other by id instead of a nested structure. New entries can just be appended to the end of the file. Possibly split this across multiple files by type, if possible
• Maybe save in expanded form instead of compacted, as it's easier to use and cuts down on processing when running renku log. 99% of the time, the json-ld will be read by machines, not humans, so compacted form is not that helpful. We should only show compacted form on demand, e.g. as a flag to renku log
• We should support python type annotations and only use manual annotations as a fallback
• Support migrations, possibly automated for simple cases.
• Still support nested contexts and type-scoped contexts
• Use framing (https://json-ld.org/spec/latest/json-ld-framing/ ) in conjunction with the flat list mentioned above.
• Support SHACL
• Generate SHACL? Based on annotations

[Panaetius]

For design of the library, we could go with the same approach attrs does, wrapping properties in functions and having decorators for classes. Though this doesn't lend itself to composition so much.

We could go with meta classes similar to marshmallow, which are quite clean, but require the definition of schema. I'm not sure if this is something that could have a workaround, I think it should be possible to have a decorator that wraps a class, returning a new class that uses the original class as meta class (plus json ld functionality/metadata). But I'm not sure how feasible that is.

Another option would be mixins.

I'm a bit unsure about how best to use decorators for this purpose. We can't decorate regular class properties, and setting metadata on a function from a decorator can break if there's another (3rd party) decorator in the chain.

[Panaetius]

If we use contexts, we need to explicitly unset parent contexts in case of ambiguities, to prevent problems with compaction.

Example:
In the parent context, we have '_label': 'rdfs:label', defined. in the child context, it is mapped to 'label': 'rdfs:label', (notice the missing _ at the beginning).

Since parent contexts are merged with child contexts, the child context now actually is {'_label': 'rdfs:label','label': 'rdfs:label'} and upon compaction, the parent context can be applied (depends on implementation of jsonld processor).

To fix this, we have to set '_label': None in the child context so it overrides the parent context.

TL;DR: We should not rely on contexts....

[rokroskar]

Can we close this? Or at least adapt the epic to include issues from calamus? /cc @Panaetius