Add links section to directory information model

[mmccool]

See also #32

It has been suggested that in some cases we want to store links to TDs or to other directories in a directory rather than just TDs. We have two possible ways to do this:

1. Store TDs with just links (note the interactions, etc. in a TD are optional, so...)
2. Have a separate links section in the information model of the directory.

Both are actually useful. Option 1 can be used if searchable metadata, eg. semantic annotation, is needed (eg for a remote directory where we want to list the types of Things indexed by that directory). But option 2 is also simple and useful, and can do things like have concrete links to all TDs stored on the directory, as well as links to self-identifying devices that for some reason (eg private information in the TD) don't want to store a copy of their TD in the directory. Note that links may be to both other information stored in the directory itself and to remote information (eg other directories).

The links section should probably have some metadata, i.e. at least a relation type (probably the same as the relation types we need to define for DID documents, and able at least to distinguish things from directories), and maybe also an optional spatial location or region (to facilitate federated spatial queries over a distributed directory service). But any more than that, and a TD should be used.

[benfrancis]

If a directory re-publishes a Thing Description at a different URL to the one it was originally retrieved from, I would suggest that it is acting as a proxy/gateway for that thing, not just a directory [1]. I would therefore suggest that the most appropriate design for a general purpose thing directory could be to enumerate all things by their URL, regardless of where the Thing Description itself is hosted.

The Thing Descriptions may well be hosted by the directory server (e.g. in the case of a gateway service), or they may be hosted elsewhere. Either way they can be enumerated as a single list of links. If the directory service does serve the content of the Thing Descriptions themselves, I would argue it should also provide a link to a canonical URL where the Thing Description originates from.

1. The front page of the W3C Web of Things website says "Each thing is identified by a URI that can be dereferenced to obtain a machine interpretable description." I interpret this to mean that if you re-publish a TD at a different URI, a client should consider it to be a separate web thing to do the original web thing. If a client does not apply this basic web principle, then anyone can post a Thing Description with the same id member as your Thing Description and claim to be an authoritative source of metadata and the client has no way of verifying the original source.

[relu91]

In general, I agree that a TDD could be just a TD with a set of typed links (i.e. rel=ThingDescription). However, I'm concerned about the possible scalability issues that an implementation might gonna hit if we specify that ALL TDDs must be conformed to this design. In particular, we still do not have pagination support for TDs and a TDD might contain a lot of them. One solution as mention in the first comment might be to have links to other TDDs and so split the "load" into different Thing Description Directory's TDs documents.

Also, we have to take in mind that exposing the registered TDs as links means that now a Thing Description can change dynamically (i.e. when a device de-register itself from the TDD). AFAICT, this is in contradiction to what we are designing for dynamically created resources and hypermedia control (i.e. TD should be as static as possible). Relevant issues about the topic: w3c/wot-thing-description#899 w3c/wot-thing-description#907

[benfrancis]

@relu91 wrote:

Also, we have to take in mind that exposing the registered TDs as links means that now a Thing Description can change dynamically (i.e. when a device de-register itself from the TDD). AFAICT, this is in contradiction to what we are designing for dynamically created resources and hypermedia control (i.e. TD should be as static as possible).

That's a good point I hadn't considered. I would argue that this is less dynamic than modifying the Thing Description every time an action is requested or changes status, but it is still something you would expect to change at runtime rather than only changing infrequently like when the software of a device is upgraded for example.

Thinking out loud... I guess the alternative is that the Thing Description for the directory has a single link to a separate resource which provides the list of links (or cached copies of the TDs themselves), as in the Fraunhofer LinkSmart proposal. This is actually similar to what the current Mozilla implementation does, except that it currently doesn't have a top level Thing Description for the directory linking to the Things resource. The downside of using a separate resource for the list of things is that it would no longer conform to the application/td+json MIME type and therefore may require defining an entirely different MIME type like application/tdir+json.

So alternatively, should the list of things be a property of the directory, rather than top level links?

{
  "title": "WebThings Gateway",
  "@context": "https://iot.mozilla.org/schemas/",
  "@type": "Directory",
  "properties": {
    "things": {
      "type": "array",
      "@type": "ThingList",
      "items": {
        "type": "string"
      }
      ...
    }
  },
  "actions": {
    "addThing": {
      "title": "Add Device",
      "@type": "AddThingAction",
      "input": {
        "type": "string"
      }
    },
    "removeThing": {
      "title": "Remove Device",
      "@type": "RemoveThingAction",
      "input": {
        "type": "string"
      }
    }
  },
  "events": {
    "thingAdded": {
      "title": "Device Added",
      "@type": "ThingAddedEvent",
      "data": {
        "type": "string"    
      }
    },
    "thingRemoved": {
      "title": "Device Removed",
      "@type": "ThingRemovedEvent",
      "data": {
        "type": "string"    
      }
    },
    "thingUpdated": {
      "title": "Device Updated",
      "@type": "ThingUpdatedEvent",
      "data": {
        "type": "string"    
      }
    }
  },
  "security": { ... },
  "links": [
    {
      "rel": "item",
      "href": "/things/thing1",
      "type": "application/td+json"
    },
    {
      "rel": "item",
      "href": "/things/thing2",
      "type": "application/td+json"
    },
    {
      "rel": "item",
      "href": "/things/thing3",
      "type": "application/td+json"
    }
  ]
}

[benfrancis]

I had a look at the Thing Description proposed for the LinkSmart directory and it makes for a bit of an awkward Thing Description, with the API of the directory being shoehorned into being described as properties and actions.

I made my own attempt at describing a directory with a Thing Description, with the list of things as a property rather than links, and that turned out quite awkward too.

I'm beginning to wonder whether trying to describe this API in terms of properties, actions and events is actually a good idea. Thing Descriptions are not great at describing collections of resources in general.

I'm wondering whether a better approach might actually be to add a single link to a separate directory resource, with an externally defined API/protocol for managing the directory. Something like this, with a separate specification of a concrete directory API for adding, removing, updating, deleting and searching things in the directory.

[farshidtz]

Would a TD structure like this be sufficient to describe links among other TDs inside a directory?

{
    "@context": "https://www.w3.org/2019/wot/td/v1",
    "@type": "Link",
    "title": "Title of linked TD",
    "security": ...,
    "securityDefinitions": ..., 
    "links": [{
        "rel": "furtherExploration",
        "href": "https://direct-introduction-endpoint.example.com",
        "type": "application/td+json"
    }]
}

@AndreaCimminoArriaga can SPARQL queries follow the link?

Edit: registered relation names such as "via" or "enclosure" may be suitable.

[egekorkan]

Also related to #440

[benfrancis]

Would a TD structure like this be sufficient to describe links among other TDs inside a directory?

I don't understand what you're trying to describe here, can you explain?

[farshidtz]

Would a TD structure like this be sufficient to describe links among other TDs inside a directory?

I don't understand what you're trying to describe here, can you explain?

It is to realize Option 1, described by OP above.

[benfrancis]

It is to realize Option 1, described by OP above.

That is a very complicated way of coding a hyperlink...

[relu91]

@AndreaCimminoArriaga can SPARQL queries follow the link?

I'll answer this, but @AndreaCimminoArriaga if you have anything to add, please 😃 . Long story short they can. So let's make an example:
First, let's assume that you have already sterilized your TD in RDF and is stored in a remote SPARQL endpoint linked in the TDD Thing Description. A federated query may look like the following:

# find the title of all remotes TDs
SELECT ?title
WHERE {
  ?td wot:link ?link . 
  ?link wot:rel "furtherExploration";
          wot:href ?service.

  SERVICE ?service {
     ?td  wot:title ?title . 
   } 
}

However, the section which describes the behavior of SERVICE used with a variable is only informative (see https://www.w3.org/TR/2013/REC-sparql11-federated-query-20130321/ section 4). This means that in practice some endpoints may differ in the implementation or do not even implement this feature. We probably need to see if this is actually working for a couple of most know SPARQL endpoints.

[AndreaCimminoArriaga]

There are two statements for following links, they are SERVICE and FROM. The former expects a URL that has behind a SPARQL endpoint 1.1, when the query is submitted to our local API it will forward the query to the URI of the SERVICE and provide an combined answer with the local and remote query answers. Delivery queries using the SERVICE statement is known as query federation. A similar example of the one provided by @relu91

PREFIX foaf:   <http://xmlns.com/foaf/0.1/>
SELECT ?person ?interest ?known
WHERE
{
  SERVICE <http://people.example.org/sparql> { 
    ?person foaf:name ?name .  
    OPTIONAL { 
      ?person foaf:interest ?interest .
      SERVICE <http://people2.example.org/sparql> { 
        ?person foaf:knows ?known . } }
  }    
}

In this case, the query answer will contain information coming from the SPARQL endpoint http://people.example.org/sparql and optionally from http://people2.example.org/sparql. Here it is assumed that both endpoints implement the SPARQL 1.1. protocol.

On the other hand, FROM is used to refer to online RDF documents. When it is specified in a query, the SPARQL endpoint provides a combined answer with the information stored in the local endpoint, and also, the query answer computed over the RDF documents present in the FROM clause. For instance:

PREFIX foaf: <http://xmlns.com/foaf/0.1/>
PREFIX dc: <http://purl.org/dc/elements/1.1/>

SELECT ?who ?g ?mbox
FROM <http://example.org/link1.ttl>
FROM <http://example.org/link2.ttl>
FROM <http://example.org/link3.ttl>
WHERE {
   ?g dc:publisher ?who .
}

Both FROM and SERVICE can be combined in order to retrieve a unified query answer:

PREFIX foaf:   <http://xmlns.com/foaf/0.1/>
SELECT ?name
FROM <http://example.org/myfoaf.rdf>
WHERE
{
  <http://example.org/myfoaf/I> foaf:knows ?person .
  SERVICE <http://people.example.org/sparql> { 
    ?person foaf:name ?name . } 
}

There is also a FROM Named, which allows to query only sub-graphs with specific names within an RDF document.

In the case of links, if the link points to a TDD then the SERVICE clause must be used (since TDD may implement SPARQL 1.1 protocol; on the other hand, if the link points to an online available TD (which can be publish by a TDD or any other service) then the clause to be used is FROM. I hope this will shed some light on this matter.

[relu91]

In the case of links, if the link points to a TDD then the SERVICE clause must be used (since TDD may implement SPARQL 1.1 protocol; on the other hand, if the link points to an online available TD (which can be publish by a TDD or any other service) then the clause to be used is FROM. I hope this will shed some light on this matter

Totally agree! +1

[mmccool]

Regarding the TD-link proposal by Farshid above, where links to other TDS (including links to TDs of other directories for federation) are included.

Use cases:

1. Private TDs. Endpoint does not want to share full metadata with the directory.
2. Dynamic TDs. TD includes information, such as as geolocation, that is constantly changing, and to getting it from the endpoint makes more sense than keeping a stale copy in the directory.
3. Large TDs. If the TD is very large, it might not make sense to store it in the directory.

Requirements:

• Should be easy to distinguish "Link" TDs (pointers) from "Thing" TDs (actual copies of TDs). Could be by @type as in the example above.

Alternatives:

• Use relation type to distinguish "Link" TDs (note: may require scanning a lot of links to check, @type is probably simpler)
• TDs with only link sections are treated as links (note: possible confusion with link-only TDs used for other purposes, for example to describe physical objects like rooms without a network API)
• Use a separate metadata section with its own datamodel that is unrelated to TDs (this would allow us to omit the security/securityDefinitions, but would discard some of the Pros below)
• As previous alternative, but allow "simplified" TD datamodel based on TDs for links. Note: might also allow for TMs in TDDs.
• Use a fully expanded URL for the @type (example will follow; see also below)

Pros:

• Reuses existing mechanism for TD search, storage, framing, validation, etc. (everything is a TD). This simplifies implementations.
• Endpoints that want just to register a link (for all the use cases above) can use the existing registration API.
• A "Link" TD pointing to another TD, possibly even in the same directory, can be a good place to put metadata "about" another TD without modifying the target. Note also that there can be more than one "Link" TD pointing to the same TD. Perhaps we could even extend the concept to allow adding semantic annotations to interactions in the target TD by giving such information in the Link TD (although this idea may run afoul of mandatory items in interactions... will have to think about).

Cons:

• A complicated way to write a hyperlink (@benfrancis)
• Links expressed as TDs will have to include security and securityDefinitions since they are mandatory in all TDs
• The @type "Link" would have to be reserved and could not be used (for example) for an endpoint. So it would be better to make it a fully expanded URL. The good news though is that if we use a URL, we won't need to use a prefix. Note: there is already a Link defined in the TD context, but it's for a class... so using it here, this way, might cause an error.

[mmccool]

Resolved in meeting 07.12.2020: Use a special "Link" TD as discussed in #34 to refer to TD held elsewhere from a directory

However, we should require an a new term for this to avoid conflicts with the existing link and Link terms, so something like...

{
    "@context": "https://www.w3.org/2019/wot/td/v11",
    "@type": "TDLink",
    "title": "Title of linked TD",
    
    "securityDefinitions": { "ns_sc:": "nosec" },
    "security": "ns_sc",
    "links": [{
        "rel": "furtherExploration",
        "href": "https://direct-introduction-endpoint.example.com",
        "type": "application/td+json"
    }]
}

To be discussed:

1. Relation types for various use cases above
2. Note that different use cases may have different security requirements
3. The exact name of the @type

TODO: PR @farshidtz

[ashimura]

during the call on 11 Jan 2021, we thought using "decribedby" based on the IANA registrry would make sense.

@farshidtz will generate a PR using "describedby", and we'll look into possible alternatives like "collection" and "item" based on some concrete use case and workflow.

[farshidtz]

The draft specification for describing links has been added. See https://w3c.github.io/wot-discovery/#exploration-mech

wot-discovery/index.html

Lines 491 to 532 in a097dbf

	<dt>Link Description</dt><dd>
	A TD which describes a reference to another TD.
	<span class="rfc2119-assertion" id="exploration-link-description-type">
	The Link Description MUST use type `LinkDescription` from the
	discovery context or URI `https://www.w3.org/2021/wot/discovery#LinkDescription`.
	</span>
	<span class="rfc2119-assertion" id="exploration-link-description-link">
	The Link Description MUST define the reference TD as a Link with
	`describedby` link relation type, `application/td+json` media type
	and `href` set to the target URL.
	</span>
	<p>
	[[[#example-td-link-type]]] is an example Link Description.
	</p>
	<!-- Using https://tools.ietf.org/html/rfc6963 for ID of examples -->
	<aside class="example" id="example-td-link-type" title="Example Link Description">
	<pre>
	{
	"@context": [
	"https://www.w3.org/2019/wot/td/v1",
	"https://w3c.github.io/wot-discovery/context/discovery-context.jsonld"
	],
	"@type": "LinkDescription",
	"id": "urn:example:link",
	"links": [{
	"rel": "describedby",
	"href": "https://example.com/td.jsonld",
	"type": "application/td+json"
	}],
	"security": "basic_sc",
	"securityDefinitions": {
	"basic_sc": {
	"scheme": "basic"
	}
	},
	"title": "Example TD referencing another"
	}
	</pre>
	</aside>
	</dd>