URI naming scheme for user/one-off/non-standard keywords

[awwright]

In some of the discussion around annotation-only or SVA (simple value annotation) keywords, some commenters seem to suggest the "@" naming scheme would be useful for writing custom keywords. This is backwards from my understanding, see my opinion here.

However user keywords are important, and I think establishing how to do it may clarify some of the discussion around SVA-only keywords. I would like to propose a convention where user keywords — keywords that are experimental, or are unlikely to see a use beyond a single party — can be defined with a URI as a name, without requiring any outside coordination, or even declared a vocabulary.

{
"type": "string",
"http://example.com/schema/distance": 40
}

Is this similar to the x- convention? How is it different?

URI names are similar to the "x-" convention in that users can mint names however they see fit. However, the problem with "x-" names is that it was not intended to be a global namespace, but by agreement or internal/private use only; and sometimes these uses "leak" out of private use, and into the "global" namespace (which is how you get headers like X-Frame-Options). Using URIs solves this because URIs exist in a global namespace (no consideration for keeping them internal is needed), and each user can select a URI that doesn't interfere with anyone else.

The only downside is that names for user keyword names may be very long. But if a keyword becomes popular, the keyword can be standardized with a more typical "standard" name; since keywords do not overlap, you can define a keyword with two different names, and schemas can use one, or the other, or both, with identical behavior.

[jdesrosiers]

I like this idea. I've suggested before that I'd like to see that all keywords have a URI and that vocabularies are just mappings of plain keyword names to the keyword identifier. Your proposal is compatible with that idea, but adds the concept of being able to use the keyword URI directly in the schema without being required to use the vocabulary system to map to a plain keyword name.

[gregsdennis]

However, the problem with "x-" names is that it was not intended to be a global namespace, but by agreement or internal/private use only

I thought we had already addressed that we recognize the history of x- and were using it because

• it's commonly recognized and popular
• we're explicitly dissociating our usage from the HTTP origins
• it's simple for users
• it's easily recognizable for implementors

While I agree with what you're saying, I don't think that our users will agree that it's a problem or that they'd want to use lengthy URI key names.

(Not to mention that we've already made a big announcement that x- is the convention we're following. We need to start nailing this stuff down and moving on or else the spec will never have a next release.)

[awwright]

I thought we had already addressed that we recognize the history of x- and were using it

The "x-" convention I'm referring to here is merely the one seen in mail/HTTP, not related to marking SVAs in JSON Schema.

[gregsdennis]

The "x-" convention I'm referring to here is merely the one seen in mail/HTTP, not related to marking SVAs in JSON Schema.

I understand that's the one that you're referring to, but you're also suggesting we use URIs for custom SVA keywords, which is counter to the decision to use x-.

If that's not what you're proposing, then what you're proposing is not clear to me.

[jdesrosiers]

you're also suggesting we use URIs for custom SVA keywords, which is counter to the decision to use x-.

I don't think that's what's being proposed. At least, that's not how I understood it. I see this as a way to introduce custom validation keywords, not custom SVA keywords. Implementations need to understand URI keyword, but they don't need to understand x- keywords. URI keywords aren't stepping on the toes of -x, but it does sort of step on the toes of the vocabulary system by providing an alternative way to declaring custom keywords.

We've long talked about how the vocabulary system is better designed for organizations creating a custom dialect than for users wanting to use custom keywords. Something like this offers them a simpler alternative. If we think about it the way I mentioned in my previous comment, it's actually a stepping stone toward using the vocabulary system rather than an alternative.

[awwright]

@gregsdennis Do you have any specific objections to this? Are there any better alternatives, on their merits?

If x- was supposed to be for one-off/user keywords only, I think a URI name overcomes many of the shortcomings of x-.

If x- was supposed to indicate validation/annotation behavior, I think we should figure out how to notate one-off user keywords first, so that we have a concrete understanding of the problem that SVA keywords are solving (what to do when an unknown, custom keyword is encountered... an SVA keyword can be ignored in more cases than other kinds of keywords, rather than causing an error).

[gregsdennis]

@awwright I don't fully understand what you're proposing.

If you're proposing that we use a URI instead of x- for custom SVA keywords, then, yes, I object to that because we've already had an extensive discussion, received user opinion, and made a very official decision to use x-. That matter is closed.

If you're proposing that we use a URI to indicate custom non-SVA (i.e. non-vocab validation) keywords, I don't understand how using a URI enables this. The implementation still needs to know what to do with it. And if you're sharing the schema, then you have to ensure that all implementations that may consume it know what to do with it. This feels like your trying to subvert vocabularies.

[awwright]

In my understanding, custom/user keywords and SVA keywords are orthogonal concepts: you can have a custom assertion keyword (you would tell it's custom because it's a URI), or you can have a standardized SVA keyword (you would know it's an annotation because it has a symbol prefix). Or you can have both (e.g. @http://example.com/kw/status would be a custom annotation keyword).

I opened this issue partly because I'm not so sure that we actually agree on what x- is doing, I can't discern what the consensus opinion actually is.

Also, I think committing ourselves to a solution because "that issue is decided" asks the editing process to make commitments that aren't realistic... practically speaking, a specification isn't final until someone has started relying on its interoperability guarantees. If the editors publish a change that doesn't capture the consensus, that's no problem, you can always go back and revise it until it does. And even an adopted specification can still be changed, so long as the change remains interoperable. So if it turns out to be a mistake, admitting "x- is ambiguous, use these alternate naming schemes instead" remains possible.

[gregsdennis]

I'm not so sure that we actually agree on what x- is doing, I can't discern what the consensus opinion actually is.

The consensus opinion was that x- keywords indicate custom SVAs, that is ad-hoc annotation-only keywords. They are functionless and their values are reported as annotations in the output.

This is simple for an implementation to support based on that convention, and it's simple for schema authors to understand.

you can have a custom assertion keyword

No you can't. They need to be declared in a vocabulary.

If you're proposing that such custom assertion keywords be supported outside of vocabularies, then you'll have to define an entirely new system for implementations to be able to say whether they support those custom assertion keywords.

But we already have such a system: vocabularies.

you can have a standardized SVA keyword (you would know it's an annotation because it has a symbol prefix)

No, standardized SVAs don't need to have, and in fact are forbidden from having, the prefix (which is x-, not a symbol).

See this addition which was added after @jdesrosiers' comment that this convention needs to be reserved for ad-hoc SVAs.

SVAs in vocabularies will be normal keywords that do not follow the convention.

@http://example.com/kw/status would be a custom annotation keyword

No. That's what x- is for.

---

I still don't understand the reasoning behind this proposal. You're saying that you want:

• custom non-vocabulary assertion keywords, which reduces interoperability and is the exact reason the vocabulary system was introduced
• to change x- to URIs... why? There's no real reason to do this.

[awwright]

@gregsdennis The reasoning is there are several orthogonal problems that x- is treating as the same problem, but this isn't always true.

① What if an update to the specification introduces a new annotation keyword? Older implementations ought to be able to ignore it for assertion purposes. The most straightforward way to identify this is with a naming scheme.

② What if the new standard keyword shadows a custom assertion keyword with different behavior? Or suppose you want to use multiple vocabularies with overlapping keyword names. You can't do this. With a URI, you could distinguish these keywords, and you could automatically know they come from a vocabulary, even if not otherwise declared.

If you like, you could even map keywords from vocabularies to URIs by some mapping like <vocabulary URI + keyword name>, e.g. https://json-schema.org/draft/2020-12/vocab/validation#type, that makes sense to me.

The two issues #1399 and (this issue) #1401 together solve these problems, by treating user keyword naming, and identifying annotation keywords, as separate problems. So you can have user keywords that are assertions, and you can have standard keywords that are annotation-only.

So describe for me, do these problems exist with x-, or are they not important enough to address, or is it something else?

[gregsdennis]

@awwright I think you are ascribing to x- purposes that it doesn't have.

The sole purpose of the x- prefix is to indicate ad-hoc, non-vocabulary, annotation-only keywords. This restores to users some of the functionality that was removed when we disallowed unknown keywords, namely the ability to add custom non-validation data to a schema.

① What if an update to the specification introduces a new annotation keyword? Older implementations ought to be able to ignore it for assertion purposes. The most straightforward way to identify this is with a naming scheme.

The proposal (#1387) disallows vocabularies (either defined in the spec or by a third party) from creating any keywords that begin with x-. This eliminates the possibility of collisions between these custom keywords and those in vocabularies.

Vocbaularies may still define annotation keywords, but they must not start with x-.

② What if the new standard keyword shadows a custom assertion keyword with different behavior? Or suppose you want to use multiple vocabularies with overlapping keyword names. You can't do this. With a URI, you could distinguish these keywords, and you could automatically know they come from a vocabulary, even if not otherwise declared.

The possibility of vocabulary keywords having collisions does still exist, but it has nothing to do with the x- prefix.

The idea to use URIs for keywords was proposed by @jdesrosiers in #1065 about augmenting the output formats with keyword->vocab mapping information. In that discussion I was opposed to it, but upon reflection, I think this is a decent idea, and I'm happy to revisit that in the proper context.

Extending that idea to be able to specify which vocab a keyword refers to in the event of a conflict is a good idea as well. This would enhance the existing vocabulary system (instead of subverting it). I recommend a new core keyword could provide such a mapping. Again, I'm happy to discuss that in its own context.

Regardless, the introduction of support for x--prefixed keywords as SVAs intersects neither of these cases.

[awwright]

The proposal (#1387) disallows vocabularies (either defined in the spec or by a third party) from creating any keywords that begin with x-.

Vocbaularies may still define annotation keywords, but they must not start with x-.

That's the problem though, implementations should be able to look at a keyword, even from a vocabulary, and understand that it's an annotation-only keyword. In an update to the specification, how would you indicate this, if not with x-?

The possibility of vocabulary keywords having collisions does still exist

I don't mean to imply x- causes this problem; rather, doesn't this proposal solve that problem?

[gregsdennis]

implementations should be able to look at a keyword, even from a vocabulary, and understand that it's an annotation-only keyword

Why? What benefit does this provide?

• If it's x-*, then it's an annotation and not from a vocab.
• If it's from a vocab, then the implementation needs to know the keywords defined in that vocab in order to process the schema, so it would already know if a given keyword is an annotation.

doesn't this proposal solve that problem?

I think it moves toward solving the problem of vocabulary collisions, but I don't know that it completely solves it.

For one, this problem doesn't exist for just annotations, but for all keywords.

Secondly, we should consider the case where a vocab keyword doesn't collide. Are you still going to require users to write out the full {vocabURI}#{keyword}, or is just the keyword sufficient? If just the keyword is sufficient, are you requiring implementations to recognize {vocabURI}#{keyword} and {keyword} to be the same?

If keyword collisions is the problem you're describing, then let's talk about a solution that fits within existing systems. We don't have a precedent for keywords having two representations (the plain keyword and a "vocab-qualified" keyword). However we could introduce a new core keyword (I'll use $using here because I'm used to C#) that can resolve these conflicts simply.

Suppose a meta-schema/dialect/whatever uses two vocabs that both define foo. The meta-schema can be written as:

{
  "$schema": "https://example.com/my-meta-schema.json",
  "$id": "https://example.com/my-meta-schema.json",
  "$vocabulary": {
    // all the core vocabs
    "https://example.com/vocabs/custom-foo-1": true,
    "https://example.com/vocabs/custom-foo-2": true
  },
  "$using": {
    "foo1": "https://example.com/vocabs/custom-foo-1#foo",
    "foo2": "https://example.com/vocabs/custom-foo-2#foo"
  }
}

Then, when the implementation can properly link the two foo keywords to foo1 and foo2, and the schema author uses foo1 and foo2 as needed in their schema.

We can then also add a requirement of implementations that if a meta-schema doesn't resolve this ambiguity, the meta-schema is invalid or indeterminant (implementation-defined).

[awwright]

Why? What benefit does this provide?

Suppose we want to publish an update to the specification that introduces a new annotation keyword with a standard meaning. Take for example, "JSON-LD context" to associate a JSON-LD/RDF semantics with a value. The keyword won't affect assertion at all, and schema authors should be able to introduce it into their schemas without disrupting validation. So, you name the keyword with a naming scheme that indicates this—perhaps @LDContext or @context.

It seems like "optional" vocabularies were supposed to solve this problem—but since the vocabulary won't even be loaded, the implementation won't understand where the @LDContext is defined or that it's safe to ignore... unless there's a naming scheme that says @ keywords are annotation-only.

In contrast, suppose you want to define a cheap assertion keyword, used by a single party. By "cheap" I mean, nobody else is going to use it, so you want to avoid the overhead of standardization, or even writing a vocabulary and associated meta-schema. This proposal provides that—just pick a URI that's under your control.

For one, this problem doesn't exist for just annotations, but for all keywords.

Well yes, URI keywords can do anything, unless specified otherwise. Again, you could have an annotation-only URI keyword e.g. @http://example.com/kw

Secondly, we should consider the case where a vocab keyword doesn't collide. Are you still going to require users to write out the full {vocabURI}#{keyword}, or is just the keyword sufficient? If just the keyword is sufficient, are you requiring implementations to recognize {vocabURI}#{keyword} and {keyword} to be the same?

I haven't fully thought out the idea, but I suppose this URI form would be required if the keyword would otherwise be ambiguous (if it's defined in multiple vocabularies).

[gregsdennis]

You've missed the important part of your supposition:

Suppose we want to publish an update to the specification that introduces a new annotation keyword.

The specification is being updated, and it defines the keyword. It uses vocabularies to define keywords. Thus, an implementation that understands the specification should understand this keyword and know that it's an annotation. If it doesn't understand the keyword, then the implementation is out of date and needs to be updated.

(BTW - This leads back to all of the other discussions that Jason and I have been having around whether to include unevaluatedProperties: false in the meta-schema`.)

suppose you want to define a cheap assertion keyword, used by a single party

I think anything that is not intended to be interoperable (i.e. schema author and consumer are the same party) needs to remain outside of the scope of the specification and something that individual implementations can support, if they want. Our job in writing the specification should remain focused on ensuring interoperability across implementations.

The point of the specification is to standardize behavior and processes for common tasks. Something that is to be used by a single party needs no such standardization, and is therefore outside of our scope. Every implementation that supports ad-hoc assertion keywords is going to have their own way of doing it anyway. The spec cannot be expected to add requirements to implementation-specific behavior.

(I've raised the question about scope in our charter PR.)