Which aspects of the Directory Service API's TD are normative?

[benfrancis]

A question which has come out of the discussion in #133 is which aspects of the Thing Description describing the Directory Service API are normative?

i.e. which of the following should be the same for all Thing Descriptions describing a directory vs. left up to the developer of the directory?

1. Interaction affordance names, e.g. createTD, updateTD, updatePartialTD, deleteTD, retrieveTD, searchJSONPath, searchXPath, searchSPARQL, registration
2. URIs in the href of forms, e.g. /td/{id}
3. uriVariable names, e.g. id, query, type

Would it be better to instead standardise something else like operations or semantic annotations (or some combination of the two), thereby leaving the names of interaction affordances, URIs and URI variables up to the developer?

Option 1: Interaction affordance names
Example: https://w3c.github.io/wot-discovery/#directory-thing-description

Pros

• Easiest for consumers to parse, because interaction affordances can be parsed directly from their names as object keys without any level of indirection via operation names or semantic annotations

Cons

• The names of affordances alone do not provide enough information to fully describe the API. It would also be necessary to standardise the names of URI variables (or add additional semantic annotations) so that consumers know what each URI variable is for. This doesn't allow developers any flexibility in the naming of interaction affordances or URI variables.

Option2: Operations
Define a new list of operations (op) for the forms of TDs describing directories, e.g. readallthings, readthing, readmultiplethings, creatething, updatething, deletething

Example: #133 (comment)

Pros

• Consistent with the way the Thing Description is standardised. Operation names are fixed but the names of interaction affordances, URIs and URI variables can be set to whatever the developer wants.

Cons

• No obvious precedent for how to define operations for events like "thing created", "thing updated" and "thing deleted"
• No way to give semantic meaning to URI variables, would need additional semantic annotations

Option 3: Semantic Annotations

Semantic annotations defined in a Directory capability schema (in the style of those at https://webthings.io/schemas/) which defines:

• The type of Thing, e.g. Directory
• Interaction affordance types, e.g. ThingListProperty, CreateThingAction, UpdateThingAction, DeleteThingAction, ThingCreatedEvent, ThingUpdatedEvent, ThingDeletedEvent
• Data schema types, e.g. Thing
• URL variable types, e.g. ThingID, JSONPathQuery, XPathQuery, SPARQLQuery

Example: #133 (comment)

Note: it seems the intention is to define this schema regardless in order to add "@type": "Directory" to the top of the Thing Description to denote that it describes a directory, the question is just whether that schema could be used for other purposes too.

Pros

• Consistent with the way Thing Description is standardised in that the names of interaction affordances, URLs and URL variables can be set to anything the developer wants and semantic meaning can be added using semantic annotations.
• A capability schema can be expressive enough to annotate all aspects of the API on its own including the type of thing being described (a directory), the list of required interaction affordances, data schemas for payloads and and URI variables.

Cons

• Adds a level of indirection where consumers have to look up which interaction affordances and URI variables correspond to the standardised semantic types

[benfrancis]

See also: w3c/wot-thing-description#1078

[farshidtz]

I think the whole protocol binding should be standardized (uris, methods, status codes, headers, request/response bodies, content types, ...).

The reason for that is not all clients of a directory would dynamically go through the Directory Description to find the endpoints. I'm pretty sure that most developers look at the spec (either in TD or equivalent OpenAPI spec) and implement the logic or use tools to generate stubs. The implementations will then be expected to work with every directory, not just the one instance. IMO, enforcing the clients to understand the TD to be able to search and find href of some device which they probably know how to work with will be a major step back in the adoption of WoT directories.

[benfrancis]

@farshidtz wrote:

I think the whole protocol binding should be standardized (uris, methods, status codes, headers, request/response bodies, content types, ...).

I agree with all of that, except for URIs (and URI variables), which I think should be left up to the developer. Requiring a fixed URI structure may be overly restrictive because there may be strange cases (particularly in large deployments like smart cities) where endpoints need to span multiple hosts or use implementation-specific URI structures. This isn't always under the developer's control.

You didn't mention whether you think the interaction affordance names (createTD, updateTD, updatePartialTD, deleteTD, retrieveTD, searchJSONPath, searchXPath, searchSPARQL, registration) should also be standardised?

The reason for that is not all clients of a directory would dynamically go through the Directory Description to find the endpoints. I'm pretty sure that most developers look at the spec (either in TD or equivalent OpenAPI spec) and implement the logic or use tools to generate stubs. The implementations will then be expected to work with every directory, not just the one instance. IMO, enforcing the clients to understand the TD to be able to search and find href of some device which they probably know how to work with will be a major step back in the adoption of WoT directories.

This is a bit of a strange argument to make given your rationale for Thing Description Directories is the exact opposite of the rationale for Thing Descriptions, which leave everything open ended - yet both claim to enable interoperability. It's going from one extreme to the other.

What I suggest would be a good compromise is to standardise the REST API (methods, status codes, headers, request/response bodies, content types etc.), but leave the interaction names and URIs up to developers, using semantic annotations to identify which affordance is which.

This is a pattern which has worked really well in WebThings for years, where we have over a hundred add-ons and 17 WoT server implementations which all follow a common API (which specifies methods, status codes, headers, request/response bodies, content types etc.) and capability schemas (which define a fixed set of interaction affordances and data schemas), but allow WoT producers to pick their own interaction names and URIs. The semantic annotations tell the consumer which interaction is which. It's not a lot of work for a consumer to use @types to tell which interaction affordances correspond to which pre-defined operations and it has proved to be a good tradeoff between ad-hoc interoperability and flexibility for developers.

[farshidtz]

I agree with all of that, except for URIs (and URI variables), which I think should be left up to the developer. Requiring a fixed URI structure may be overly restrictive because there may be strange cases (particularly in large deployments like smart cities) where endpoints need to span multiple hosts or use implementation-specific URI structures. This isn't always under the developer's control.

You didn't mention whether you think the interaction affordance names (createTD, updateTD, updatePartialTD, deleteTD, retrieveTD, searchJSONPath, searchXPath, searchSPARQL, registration) should also be standardised?

The deployments can still decide the base path. But I still think the relative href and uri variables should be standardized; at the very least, they should be recommended.

The interaction affordance names are currently used to reference the interactions. Using semantic annotations and operation types as you suggested is a good alternative, if those interaction names are reserved for something else. Using semantic annotations and op values to find interactions is slightly more complicated, programatically.

The reason for that is not all clients of a directory would dynamically go through the Directory Description to find the endpoints. I'm pretty sure that most developers look at the spec (either in TD or equivalent OpenAPI spec) and implement the logic or use tools to generate stubs. The implementations will then be expected to work with every directory, not just the one instance. IMO, enforcing the clients to understand the TD to be able to search and find href of some device which they probably know how to work with will be a major step back in the adoption of WoT directories.

This is a bit of a strange argument to make given your rationale for Thing Description Directories is the exact opposite of the rationale for Thing Descriptions, which leave everything open ended - yet both claim to enable interoperability. It's going from one extreme to the other.

What I suggest would be a good compromise is to standardise the REST API (methods, status codes, headers, request/response bodies, content types etc.), but leave the interaction names and URIs up to developers, using semantic annotations to identify which affordance is which.

This is a pattern which has worked really well in WebThings for years, where we have over a hundred add-ons and 17 WoT server implementations which all follow a common API (which specifies methods, status codes, headers, request/response bodies, content types etc.) and capability schemas (which define a fixed set of interaction affordances and data schemas), but allow WoT producers to pick their own interaction names and URIs. The semantic annotations tell the consumer which interaction is which. It's not a lot of work for a consumer to use @types to tell which interaction affordances correspond to which pre-defined operations and it has proved to be a good tradeoff between ad-hoc interoperability and flexibility for developers.

I am against the all-in WoT approach, when it comes to exploration. There are consumers who fully comply with WoT and can find the right directory endpoint, query a TD, authenticate and interact with a thing. This is perfectly reasonable. However, there are brownfield consumers who just care about searching for a TD and extracting the href without having to worry about other TD features. At the time of writing, the size of the latter community consumers is significantly larger and we can't simply ignore that and expect adoption.

We do use the TDDs in smart city projects and most clients currently use hard-coded paths and query arguments developed based on OpenAPI specs. They filter TDs using embedded domain-specific extensions and extract the links and hrefs from the TD object, or filter and select just the links and hrefs.

[mmccool]

TDs are open-ended because they are descriptive, and need to be able to describe existing systems. But since discovery is a "new" thing, we felt it was better to make it prescriptive. However, I agree that URIs don't have to be fixed, since the purpose of the directory TD is to provide variable information like that. Likewise since we can't seem to agree on a security scheme, that will also not be fixed but provided by the TD (although the "bootstrap" mechanism, which we still have to sort out, will have to be prescriptive). There are also a number of optional features that we have to clearly specify.

Right now we have an "example TD" which makes this confusing. What we should have is a normative TM, then we can define all of this clearly. Or at least, most of it. There are a few things that TMs lack (for example, there is no mechanism to allow picking among a restricted set of options...) so we'll have to add a few assertions (for instance, to limit what protocols should/may be supported), but I would prefer to depend on a TM as the definition of the interface.

[mmccool]

I think the motivation to have prescriptive URLs is to simplify implementations, though, since then a system can use a directory without having to consume a TD. Since we are talking about not requiring very small devices to consume TDs, either we (a) don't let them use a directory to discover other devices or services, which might include things like hubs (b) we make the URIs prescriptive so small devices only have to figure out they are talking to a directory and its base URI.

[mmccool]

• I am OK with having normative URIs, and agree it makes implementations simpler
• We should really have a TM instead of an "example TD". Then everything in the TM would be normative... and we would give relative URLs for the interactions, and then an implementation would just fill in the "base" to get the TD.
• We could leave the base out, but it might be clearer to use something like the following so it's clear that filling in a base is necessary:

   "base": "{{{BASE_URL}}}"

[mmccool]

Since there are a lot of pending PRs against the example TD, we should wait to update this until those are merged.

[mmccool]

Resolution: This is essentially decided, that we will use a TM to define most aspects of the API normatively. What is left is just to convert our example TD to a TM, which is covered by a new issue: #86