An alternate approach to meta-schemas

[jdesrosiers]

Since there's been some discussion about $recusriveAnchor/$recursiveRef lately, it might be a good time to share one of the experimental things I've tried out with Hyperjump Validation. It requires a big change to the concept of meta-schemas, but it eliminates the need for dynamic scope keywords and you basically get vocabulary support for free. It's not perfect and the syntax could certainly use some cleanup, but it has some pretty interesting properties.

In Hyperjump Validation, instead of having a meta-schema that describes a schema, it uses a separate meta-schema for each keyword.

Here's how it works... Keywords are identified by URLs. Each keyword the schema can use is declared using the $meta keyword. $meta works in a very similar way to @context in JSON-LD mapping a plain name to a URL.

{
  "$meta": {
    "type": "https://validation.hyperjump.io/common/type",
    "properties": "https://validation.hyperjump.io/common/properties"
  },
  "type": "object",
  "properties": {
    "foo": { "type": "string" }
  }
}

The URL given to a keyword should be dereferenceable to a schema that validates that keyword. Therefore, a validator can determine if a keyword is valid even if it doesn't know how to evaluate the constraint that keyword expresses.

Declaring keywords this way is like in any programming language where you would import a
dependency before you use it. Any keyword that appears in the schema that is not declared in $meta is an error. Many people have wanted to include "additionalProperties": false in the JSON Schema meta-schema. This constraint provides the protection people want from typos in keyword names without constraining extensibility.

Most people wouldn't want to have to declare every keyword they use this way, so most schemas will likely choose to link to a common $meta that declares all the keywords they use. This list of keywords is effectively a vocabulary. This example uses the standard Hyperjump Validation vocabulary.

{
  "$meta": { "$href": "https://validation.hyperjump.io/common" },
  "type": "object",
  "properties": {
    "foo": { "type": "string" }
  }
}

A vocabulary can be constructed by creating a new $meta document that declares all the keywords for the vocabulary. This could include new keywords mixed with standard keywords or keywords from other vocabularies. When constructing a vocabulary, you can change the name of keywords (like definitions to $defs) just by changing the name of the keyword in the $meta document. Another reason to change the name of something might be when combining two vocabularies that each have a keyword with the same name.

That's all great, but we still need something like the old meta-schema so keyword meta-schemas that include schemas (like properties) can declare that something is a schema. To solve this problem, I created a new keyword called validation (or schema in JSON Schema terms) that is a flag that indicates that the value should be a schema. It doesn't matter what vocabulary of the schema is. The schema will declare it's own vocabulary. This keyword only indicates that it's a schema of some kind.

{
  $meta: { $href: "https://validation.hyperjump.io/common" },
  validation: true
}

I don't imagine that anyone wants to make a change of this magnitude to JSON Schema, but hopefully it inspires people to look at the problem from a different angle and maybe come up with some new ideas that could be applied in a way that wouldn't completely change the way meta-schemas work.

[handrews]

@jdesrosiers I think there are good ideas in here that would fit with the plan to have a machine-readable vocabulary file, which would handle things like "is this a validation assertion", "is this an in-place applicator", etc. #602 talks about this, but it's kind of a rambling mess and I should re-file it.

This might not be as dramatic of a change, but might be a way to accomplish some of what you are talking about in way that fits a little more smoothly. Or by the time of the next draft (after this one), we might have more feedback on vocabularies etc. to justify a larger change. Thanks for filing this!

[gregsdennis]

I really like this. Nice job @jdesrosiers.

we still need something like the old meta-schema so keyword meta-schemas that include schemas (like properties) can declare that something is a schema. To solve this problem, I created a new keyword called validation

I don't think we need this (expanded upon below). When a meta-schema validates itself, it's not aware that this is what's going on. It's just validating an instance. It just so happens that the instance is the JSON representation of itself.

---

@handrews in line with your ideas of machine-readable vocabs, the dereferenced schema would need to contain additional data about the keyword. The properties that define this metadata would only have meaning when dereferenced within the context of the $meta keyword.

{
  "$keywordClass": "applicator",
  ... // more metadata
  ...  // schema definition of the keyword
}

Outside of the $meta context, these additional keywords could be ignored (hence not needing a specific validation keyword).

---

Lastly, just as with $vocab, it's important to note that just because the keyword can be semantically validated by an implementation does not mean that the implementation supports it. The implementation still needs to contain the logic for the intended operation of the keyword.

Also, @handrews, I appreciate your openness to this idea. I remember you had asked for other approaches before we published draft 2019-09, and the response was underwhelming. This, I think, counts toward that.

[handrews]

@gregsdennis my intention would be to use the URI from $vocabulary as the URI for the semantic description file, so I don't think a new $meta keyword would be needed in the meta-schema (or schema).

Caveat: I have not delved into this issue deeply enough due to other distractions, just skimmed it, so this might not be right, I just wanted to get a general "yes there's stuff worth considering" comment on the record ASAP.

[gregsdennis]

I see where you're going with that, but I hesitate to reuse a keyword, even if it's relatively new and lightly adopted. This is quite a shift in design, even though the purpose is similar. I'd prefer to make the hard break and use $meta (or whatever).

[handrews]

@gregsdennis OK I'll have to put this on hold because I'm 800% overloaded right now, am farther behind on the current draft than I was a week ago, and seem to be missing something.

But the explicit purpose of setting up $vocabulary the way it was set up was for it to eventually identify a machine-readable semantic, and possibly partially syntactic, description of the vocabulary. That has always been the point. My initial interest is seeing what from this proposal can fit with that long-standing intention.

We didn't create that semantic description file specifically in oder to see what people came back with after implementing, which @jdesrosiers has done.

I might end up sold on the more dramatic reworking, but where I'd like to start is lining up this proposal with the (very vaguely articulated) original direction and seeing if that produces a good balance between disruption (keep in mind we're putting draft 2020-0? out with OAS 3.1/4.0 so unlike 2019-09 it will immediately have a huge user base) and functionality.

[gregsdennis]

where I'd like to start is lining up this proposal with the (very vaguely articulated) original direction

That's a fair goal, but I think that this achieves the same end goals in a cleaner way.

I imagine we'll continue to discuss. Join back in when you can.

[handrews]

That's a fair goal, but I think that this achieves the same end goals in a cleaner way.

I agree that this could do that in the sense of not needing $recursive* specifically for meta-schemas, but while meta-schemas are the main rationale for those keywords, they are not the only recursive schemas. So I'm not sold that we would drop the keywords even if we did this (and I still don't see why we'd need to use $meta instead of $vocabulary).

If y'all think we can/should dump $recursive* entirely (and also decline $extends) and could do that in the next two weeks, then I'm interested in that discussion.

But this feels to me more like a solution that overlaps rather than entirely replaces.

[jdesrosiers]

@handrews

while meta-schemas are the main rationale for those keywords, they are not the only recursive schemas.

That's true. Moving to keyword-level meta-schemas doesn't make $recursive* completely redundant. But, other than meta-schemas, extending recursive schemas is extremely rare. It could be argued that it's rare enough that it isn't worth including. If it wasn't used for meta-schemas, it certainly wouldn't exist and we wouldn't be losing sleep over it.

If y'all think we can/should dump $recursive* entirely (and also decline $extends) and could do that in the next two weeks

I definitely don't think that's a reasonable timeline.

[jdesrosiers]

@gregsdennis
The validation keyword is my least favorite part of how this ended up working, but I couldn't come up with any way around it (or something like it). Let's look at the meta-schema for the properties keyword.

{
  "$meta": { "$href": "https://validation.hyperjump.io/common" },
  "type": "object",
  "patternProperties": {
    "": { "validation": true }
  }
}

If I don't use validation here, what can I do? Normally, we would use { "$ref": "#" }, but then # would be the keyword meta-schema, not a full schema.

[gregsdennis]

2019-09 uses $recursive* for that. The difference is that you're splitting on keyword, and the current spec is splitting on "vocabulary."

I guess that's the heart of the issue here. Maybe that can lead us to a more unified solution.

[thinking out loud] If we break down the vocabularies further to their constituent keywords, we're pretty close to your idea with $meta.

Each vocabulary meta-schema would merely be a collection of keyword meta-schemas, and they're all specified as required.

Current implementation of applicator meta-schema

{
  "$schema": "https://json-schema.org/draft/2019-09/schema",
  "$id": "https://json-schema.org/draft/2019-09/meta/applicator",
  "$vocabulary": {
    "https://json-schema.org/draft/2019-09/vocab/applicator": true
  },
  ...
}

The idea here is that https://json-schema.org/draft/2019-09/meta/applicator and https://json-schema.org/draft/2019-09/vocab/applicator are separate documents. The original concept was that the vocab one would eventually point to a differently-formatted document that a validator could use to assert some things about the keywords that need to be supported.

However, the meta-schema goes on to declare each keyword under properties.

I think these can be combined using the $meta idea.

Thinking out loud model

// applicator meta-schema
{
  "$schema": "https://json-schema.org/draft/2020-XX/schema",
  "$id": "https://json-schema.org/draft/2020-XX/meta/applicator",
  "$vocabulary": {
    "https://json-schema.org/draft/2020-XX/vocab/applicator/multipleOf": true,
    "https://json-schema.org/draft/2020-XX/vocab/applicator/maximum": true,
    "https://json-schema.org/draft/2020-XX/vocab/applicator/properties": true,
    ....
  },
  "$recursiveAnchor": true // [edit: added]
}

// properties meta-schema
{
  "$schema": "https://json-schema.org/draft/2020-XX/schema"
  "$id": "https://json-schema.org/draft/2020-XX/vocab/applicator/properties",
  "$vocabulary": {
    "https://json-schema.org/draft/2020-XX/vocab/keywordMeta": true,
    ... // do we need to list others?  core is included implicitly...
  },
  "type": "object",
  "additionalProperties": { "$recursiveRef": "#" }, // this
  "default": {},
  ...
}

This new document would also contain additional metadata about the keyword, whatever we need.

• keyword type
• annotation dependencies (e.g. what other keywords are required to be processed first)
• etc.

The key here is that $vocabulary now assumes a new "import" type role where it not only aggregates other vocabularies, but it also aggregates specific keywords, and each keyword is defined independently.

tl;dr This uses the ideas from $meta while expanding on the existing $vocabulary.

[gregsdennis]

It's worth mentioning that one thing that $vocabulary brings that $meta doesn't address is the ability to reference a vocab without it being required (by using a false value with the declaration).

[jdesrosiers]

@gregsdennis I've been really busy today and haven't had time to look at your last comment in detail, but I wanted to point this out ...

If I don't use validation here, what can I do? Normally, we would use { "$ref": "#" }, but then # would be the keyword meta-schema, not a full schema.

2019-09 uses $recursive* for that.

$recursiveRef doesn't work either. Where does the $recursiveAnchor go? Without a dialect level schema, there's nothing for the $recursiveRef to point to.

[gregsdennis]

$recursiveAnchor is still at the root schema and each meta-schema. (I'll edit the comment.) I don't see how the $recursive* keywords wouldn't work the same with this.

[handrews]

Regarding validation, my original plan (which may or may not be written down anywhere) involved specifying the classification of each keyword.

• assertion: this keyword can produce a false boolean result
• annotation: this keyword produces a result beyond the boolean result
• applicator: this keyword applies one or more subschemas, so a plugin would need to call back into the main validator implementation
  • in-place: this is important b/c keywords like unevaluated* require that they are run after all in-place applicators
  • child: not sure if it's useful to say how the subschemas are mapped to child instances?
  • by-reference: the value of this keyword needs to be resolved to get the actual subschema
  • inline: opposite of by-reference
  • automatic: $schema and contentSchema are weird applicators that are not automatically applied
• identifier: these keywords assign or modify (e.g. set base URIs) identifiers, and probably get processed in some way on schema load to build the URI-to-schema-object map
• reserved locations: $comment and $defs
  • schema location: the other thing (than applicator) that is relevant for finding schemas
  • non-schema location: everything else

Some of those very clearly map to either the behavior of the generic implementation or the behavior of the plugin. Others are not as clear and maybe are not necessary. One version of the idea allowed keywords to specify on which annotations they depended, and whether they depended on them from adjacent keywords only, or also from subschemas of adjacent in-place applicators. I decided that was probably complicating things.

This is just a brain dump so y'all can pick over the ideas and see if any are helpful.