Validate schema with relative paths and within-document references

[tillahoffmann]

I would like to use a schema base.json that defines basic properties together with a schema derived.json which adds further validation to the base schema. The derived schema needs to reference the base schema using a relative path as discussed in #98. However, I cannot get both the relative reference and within-document references to work at the same time. Here's a minimum example to illustrate the problem.

base.json

{
    "definitions": {
        "string_alias": {
            "type": "string"
        }
    },
    "properties": {
        "my_string": {
            "$ref": "#/definitions/string_alias"
        }
    }
}

derived.json

{
    "allOf": [
        {
            "$ref": "base.json"
        },
        {
            "properties": {
                "my_number": {
                    "type": "number"
                }
            }
        }
    ]
}

data.json

{
    "my_string": "Hello world!",
    "my_number": 42
}

validate.py

import json
import os
import sys
import jsonschema


if len(sys.argv) > 1:
    resolver = jsonschema.RefResolver('file://%s/' % os.path.abspath(os.path.dirname(__file__)), None)
else:
    resolver = None


with open('base.json') as fp:
    base = json.load(fp)

with open('derived.json') as fp:
    derived = json.load(fp)

with open('data.json') as fp:
    data = json.load(fp)


try:
    jsonschema.validate(data, base, resolver=resolver)
    print("Passed base schema.")
except Exception as ex:
    print("Failed base schema: %s" % ex)

try:
    jsonschema.validate(data, derived, resolver=resolver)
    print("Passed derived schema.")
except Exception as ex:
    print("Failed derived schema: %s" % ex)

If I use the resolver, the relative path gets resolved correctly but the within-document reference fails. If I don't use the resolver, the within-document reference succeeds but the relative path does not get resolved.

# With resolver
python validate.py resolver
Failed base schema: Unresolvable JSON pointer: 'definitions/string_alias'
Passed derived schema.

# Without resolver
python validate.py         
Passed base schema.
Failed derived schema: unknown url type: 'base.json'

Any suggestions on how to get both to work?

Edit: Looks like this may be related to #306.
Edit: A temporary workaround is to have base.json reference itself explicitly, i.e. use base.json#/definitions/string_alias instead of #/definitiosn/string_alias.

[smittysmee]

I agree, this seems to be an issue when the base uri is not an HTTP or HTTPS scheme.
When the item is other than these, reference is lost.

The workaround is to have the reference to self explicitly defined. Instead of #/definitions/variable_name one must use <current_id>#/definitions/variable_name

[Lordnibbler]

+1 on this

[chimeno]

+1

[joecabezas]

+1

[gaoxinyang]

+1

[SamuelePilleri]

Any news?

[handrews]

@Julian looking through this and the related #98 it seems that the problem is uncertainty over how to set the initial base URI in the absence of a root $id (or id for older drafts).

The way I've handled this using other implementations is to give all schemas a root $id, which is the RECOMMENDED approach in the spec (more on that at the end). But in the Python implementation I worked on for a bit (before realizing you were still active and dropping it :-), I interpreted RFC 3986 §5.1.3's rules around establishing a base URI from the "retrieval" URI as meaning that any document loaded from a local files system should have a base URI that is the file:// URI of the local file.

1. Does this seem like a reasonable interpretation to you?
2. Should I clarify this in the spec?
3. As an implementor, do you see any problems with implementing that interpretation?

This only affects resolving URIs, and imposes no requirements on whether the implementation can automatically load the files or not.

My assumption would be that your command-line script would construct the file:// URI and pass it to the library, although I don't care how it's actually implemented. However, I do not think that an implementation is responsible for determining a retrieval base URI when it is instantiated as a library and just handed a data structure. The library has no way of knowing what file:// URI to construct, or if there even is a filesystem involved. In that case, I believe that RFC 3986 §5.1.4 applies, which basically says "eh, whatever".

More specifically, it reads in part:

A sender of a representation containing relative references is responsible for ensuring that a base URI for those references can be established.

The proper way to do this is to set $id (or id for draft-04). The most recent work I did was with a JavaScript implementation, and I just set $id with https:// URIs for every schema document, pre-loaded all of them as documented by the library, and references worked even though no HTTP requests were ever made.

@tillahoffmann, @SamuelePilleri, etc: JSON Schema documents are intended to have $schema and $id (or id) set in the root schema objects specifically to avoid this problem.

I think we should ensure that the spec makes clear whether or how to set a base URI when a file path is known, but the proper solution is to set your own base URI.

[Julian]

@handrews have been meaning to respond in some more detail but haven't had the moment to sit down and do it, so might as well throw a short response up in the meanwhile :)

Does this seem like a reasonable interpretation to you?

Yes!

Should I clarify this in the spec?

Maybe, though I'd not expect this to be different for file / the filesystem than any other URI, yeah? So if there's what to clarify [I don't remember the current language] maybe it's just saying "if users forget to specify an $id and you loaded the schema from somewhere, assume one using the URI you used to fetch the document".

As an implementor, do you see any problems with implementing that interpretation?

Nope should be reasonable!

[handrews]

@Julian

"if users forget to specify an $id and you loaded the schema from somewhere, assume one using the URI you used to fetch the document".

Yeah that's basically what I'm going for. Somewhere there's a section on establishing a base URI, so I'll look at how that's worded. Perhaps including a note that a local filesystem is one such "somewhere" rather than calling out the file:// scheme.

[topher515]

I had a hard time figuring this out myself, and I kept googling into this github issue. (The solution is pretty much what @clenk says above.)

Here's a SO Q&A which covers it: https://stackoverflow.com/questions/53968770/how-to-set-up-local-file-references-in-python-jsonschema-document

Hopefully this helps someone in the future.

[joecabezas]

I might add, external references in json schemas is the worst documented feature in the specification and by consequence the worst implemented on every implementation across different languages, because of that is very sensitive between languages and even between implementation in the same language, some require some extra work to make it work, and other a complete redeclaration if the IDs, I strongly suggest to use the same library across projects or leave the validation to an external common service that only does that (maybe asynchronously) Happy New year everyone!

…

On Sat, Dec 29, 2018, 7:42 AM Chris Wilcox ***@***.***> wrote: I had a hard time figuring this out myself, and I kept googling into this github issue. (The solution is pretty much what @clenk <https://github.com/clenk> says above.) Here's a SO Q&A which covers it: https://stackoverflow.com/questions/53968770/how-to-set-up-local-file-references-in-python-jsonschema-document Hopefully this helps someone in the future. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#343 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGuZiONQ7GJNmndYd3nPX0hWvET0RgRks5u90cMgaJpZM4ONNy-> .

[handrews]

@joecabezas I did add json-schema-org/json-schema-spec#686 for the next draft which should nudge people a bit more in the right direction. The problem is that there are endless ways to store and find "external" documents. The spec cannot cover every single filesystem, IoT device, network protocol, etc. etc. We have to rely on people understanding URIs (and how they don't necessarily directly reflect storage) or implementing a loading mechanism that works for their environment.

In the last draft we did add a section on Loading Reference Schemas and Dereferencing in an effort to make this more clear, but that was after the initial "draft-07" version so I'm not sure how many implementations have made use of it.