Support for documenting not (yet) existing resources

[jesper-friis]

Resources that does not exists (yet), should be documented in the Tbox.

Suggested implementation strategy:

Create a new module tripper.datadoc.tboxdoc.

Add a general function that takes a dict-like description of a non-existing resource. For example a physics-based simulation that has not yet been executed:

{
    "@context": ["https://w3id.org/emmo/context", "https://w3id.org/emmo/domain/mydomain/context"],
    "@id": "ex:MyModelSimulation",
    "@type": "owl:Class",
    "subClassOf": "PhysicsBasedSimulation",
    "prefLabel": "MyModelSimulation",
    "elucidation": "A simulation using my physics-based materials models.",
    "isParticipatedBy": {
        "@id": "ex:MyModel",
        "@type": "owl:Class"  # restrictionType defaults to some
    },
    "command": "mymodel infile.txt -o outfile.txt",  # Command to execute the model
    "workdir": ".",  # Relative path to working directory
    "environment": {"VAR1": "val1", "VAR2": "val2"},  # Environment variables (JSON-encoded)
    "install_command": "pip install git+ssh://git@github.com:MyProj/myrepo.git@master",
    "hasInput": [
        {
            "@id": "ex:MyInputDataset",
            "@type": "owl:Class",
            "subClassOf": ["dcat:Dataset", "emmo:Dataset"],
            "restrictionType": "exactly 1",  # defaults to "some"
            "prefLabel": "MyInputDataset.",
            "elucidation": "My input dataset.",
            "distribution": {
                "accessURL": "file:./infile.txt",
                "accessService": {
                    "conformsTo": "https://github.com/SINTEF/dlite/",
                    "generator": "ex:inputplugin"  # for generating input
                }
            }
        }
    ],
    "hasOutput": [
        {
            "@id": "ex:MyOutputDataset",
            "@type": "owl:Class",
            "subClassOf": ["dcat:Dataset", "emmo:Dataset"],
            "restrictionType": "exactly 1",  # defaults to "some"
            "prefLabel": "MyOutputDataset.",
            "elucidation": "My output dataset.",
            "distribution": {
                "accessURL": "file:./outfile.txt",
                "accessService": {
                    "conformsTo": "https://github.com/SINTEF/dlite/",
                    "parser": "ex:outputplugin",  # for parsing output
                    "generator": "ex:outputplugin"  # for storing output to external storage
                }
            }
        }
    ]
}

When a type is "Class" (owl:Class), a class is generated in the KB instead of an individual. From the above input, our new function should generate the following RDF:

ex:MyModelSimulation rdf:type owl:Class ;
    rdfs:subClassOf emmo:PhysicsBasedSimulation ,
    [
        rdf:type owl:Restriction ;
        owl:onProperty emmo:hasInput ;
        owl:qualifiedCardinality "1"^^xsd:nonNegativeInteger ;
        owl:onClass ex:MyInputDataset ;
    ] ,
    [
        rdf:type owl:Restriction ;
        owl:onProperty emmo:hasOutput ;
        owl:qualifiedCardinality "1"^^xsd:nonNegativeInteger ;
        owl:onClass ex:MyOutputDataset ;
    ] ,
    [
        rdf:type owl:Restriction ;
        owl:onProperty emmo:isParticipatedBy ;
        owl:someValuesFrom ex:MyModel ;
    ] ;
    skos:prefLabel "PremodSimulation"@en ;
    emmo:elucidation "A simulation using my physics-based materials models."@en ;
    oteio:command "mymodel infile.txt -o outfile.txt" ;
    oteio:workdir "." ;
    oteio:environment "{\"VAR1\": \"val1\", \"VAR2\": \"val2\"}"^^rdf:JSON ;
    oteio:installCommand "pip install git+ssh://git@github.com:MyProj/myrepo.git@master" .

ex:MyInputDataset rdf:type owl:Class ;
    rdfs:subClassOf dcat:Dataset, emmo:Dataset ;
    skos:prefLabel "MyInputDataset"@en ;
    emmo:elucidation "My input dataset."@en ;
    dcat:distribution [
        rdf:type dcat:Distribution ;
        dcat:accessURL <file:./infile.txt> ;
        dcat:accessService [
            rdf:type dcat:DataService ;
            dcterms:conformsTo <https://github.com/SINTEF/dlite/> ;
            oteio:generator: ex:inputplugin ;  
        ] ;
    ] .

ex:MyOutputDataset rdf:type owl:Class ;
    rdfs:subClassOf dcat:Dataset emmo:Dataset ;
    skos:prefLabel "MyOutputDataset"@en ;
    emmo:elucidation "My output dataset."@en ;
    dcat:distribution [
        rdf:type dcat:Distribution ;
        dcat:accessURL <file:./infile.txt> ;
        dcat:accessService [
            rdf:type dcat:DataService ;
            dcterms:conformsTo <https://github.com/SINTEF/dlite/> ;
            oteio:parser: ex:myoutputplugin ;
            oteio:generator: ex:myoutputplugin ;
        ] ;
    ] .

[jesper-friis]

Tasks:

• Implement a function that serialise the dict to RDF as shown above. Object properties on classes should be represented as restrictions. This mean that the JSON-LD representation of the hasInput and hasOutput should be pre-processed from

  "hasInput": [{"@id": "ex:MyInputDataset", ...}]

  to

  "subClassOf": [
    {
      "@id": "owl:Restriction",
      "owl:onProperty": "emmo:hasInput",
      "owl:qualifiedCardinality": 1,
      "owl:onClass": {
        "@id": "ex:MyInputDataset",
        ...
    }
  ]

  before we give it to dataset.save_dict() for serialising to RDF.

  An alternative approach is to use a CONSTRUCT SPARQL query, as Simon Stier is doing in https://github.com/OO-LD/awl-python/blob/72bffb0339eb46ecb8942c2fb4d7f2279eb18345/tests/test_usecases.py#L6.

  Should we call this function tboxdoc.save_dict() or can we come up with a better name?

• Implement the opposite function, i.e. a function that loads the dict from the KB. Should we call it tboxdoc.load_dict()?

• Implement a function that, takes a list of IRIs to input datasets and a workdir as arguments and uses model documentation to generate the model input. The function should create an individual for the simulation in the KB and link the input datasets to this individual. The IRI of the new IRI should be returned.

  The @type of the datasets in the list of IRIs must match one and only one @id under hasInput in the model dict.

• Implement a function that can be called when the simulation has successfully succeeded. As input, it needs the IRI of the simulation individual and a workdir. From that it can find the documentation of ex:MyModelSimulation and where to read the output files. It can then read the output files and store them in a storage using the documented generator and document the output in the KB. Finally it links the output to the simulation process with emmo:hasOutput and the input to the output using emmo:isProcessedTo.

• Add a utility function that detects whether a dict documents a class. This function can then be used in dataset.save_dict() to determine whether dataset.save_dict() should save the dict the normal way (using json-ld) or invoking this module.

[jesper-friis]

Points for discussion:

1. Can we come up with a better name for the new module than tboxdoc?
2. The example above documents dlite plugins using dcat:accessService instead of parser/generator properties on the distribution. Is this how we want do to it?
3. We are also using dcat:accessURL (mandatory in DCAT-AP) for referring to the local distributions of the model input/output. Should we standardise on that?
4. Regarding the dcat:accessURL, the RFC 8089 standard doesn't allow relative paths using the file scheme. However, as discussed here, it is possible to add a custom extension supporting relative paths using the syntax: file:./infile.txt. Should we do that? The alternative would be to replace dcat:accessURL with a custom property expecting a file path.
5. The example above implicitly assumed special handling of emmo:hasInput and emmo:hasOutput. Can we generalise this?