Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inlining schemas with json-schema-ref-parser is unnecessary, and may not work in case inlined schemas have $id #1426

Open
teq0 opened this issue Feb 3, 2021 · 5 comments

Comments

@teq0
Copy link
Contributor

teq0 commented Feb 3, 2021

I think this might be a V7 bug as my schemas were all passing with V6.

If an external schema is referenced using $ref by more than one property, or within a property defined in the definitions section, and the $refs are resolved by putting the schemas inline, as with json-schema-ref-parser, then compilation fails with the error

reference "https://schemas.example.com/samples/child" resolves to more than one schema

What version of Ajv are you using? Does the issue happen if you use the latest version?
7.0.4

Ajv options object
{}

JSON Schema

samples/parent.schema.json

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "https://schemas.example.com/samples/parent",
  "title": "Schema that references another schema multiple times",
  "type": "object",
  "properties": {
    "name": {
      "type": "string"
    },
    "eldestChild": {
      "$ref": "./child.schema.json"
    },
    "youngestChild": {
      "$ref": "./child.schema.json"
    }
  }
}

samples/child.schema.json

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "https://schemas.example.com/samples/child",
  "title": "Schema referenced from a parent",
  "description": "Example description",
  "type": "object",
  "properties": {
    "name": {
      "type": "string"
    }
  }
}

It also happens if the $ref only appears once in the file but it's inside an object in the definitions section:

samples/parent2.schema.json

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "https://schemas.example.com/samples/parent2",
  "title": "Schema that references another schema",
  "description": "Example description",
  "type": "object",
  "properties": {
    "name": {
      "type": "string"
    },
    "children": {
      "type": "array",
      "items": {
        "$ref": "#/definitions/child"
      }
    }
  },
  "definitions": {
    "child": {
      "type": "object",
      "properties": {
        "order": {
          "type": "number"
        },
        "details": {
          "$ref": "./child.schema.json"
        }
      }
    }
  }
}

This seems to be because traverse() finds it multiple times.

Your code

const Ajv    = require('ajv').default
const $RefParser = require('json-schema-ref-parser')

const ajv = new Ajv();

$RefParser.dereference("samples/parent.schema.json", function(err, schema) {
  if (err) throw err
  try {
    ajv.compile(schema);
    console.log('[ PASS ] Schema validation passed.')
  }
  catch (e) {
    console.log(`[ FAIL ] Schema validation failed: "${e.message}"]`)
    process.exit(1)
  }
})

Validation result, data AFTER validation, error messages

[ FAIL ] Schema validation failed: "reference "https://schemas.example.com/samples/child" resolves to more than one schema"]

What results did you expect?
Schemas pass compilation. It's perfectly valid to reference an external schema multiple times or from inside local definitions, and it used to work fine.

Are you going to resolve the issue?
Maybe, but I'm not quite sure where to start. It seems like it's not necessary to throw an error if a schema is referenced more than once, since it didn't used to. But I assume that code was added for a reason, and I don't know what that reason is, or if anything bad would happen if it were removed.

@teq0 teq0 added the bug report label Feb 3, 2021
@teq0
Copy link
Contributor Author

teq0 commented Feb 4, 2021

I put the code and sample into a repo to make it easy to reproduce - https://github.com/teq0/ajv-duplicate-demo

@epoberezkin
Copy link
Member

epoberezkin commented Feb 7, 2021

The problem here is that after you use json-schema-ref-parser, you replace two identical refs with two different schema objects, that have the same $id (which is exactly what the error says).

The difference with Ajv v6 is that v7 changed from comparing serialized schemas (and they had to be stable-stringified in v6, not just JSON.serialize). Because of that what v6 saw as identical schemas, v7 sees as different schemas (because it compares them by reference).

While this change does cause some confusion - see #1413 for example - it also highlights the cases when Ajv is used incorrectly (e.g. using ref resolution outside of Ajv as in your example - see below the working approach - or not reusing compiled schemas, and instead at best serialising the schema every time validation is needed as in #1413 or even worse - compiling them every time).

The motivation for this change was:

  • avoiding expensive schema serialization when a custom serialised has to be used (to ensure the same order of object members).
  • reducing memory consumption - it is substantial for large schemas.
  • availability of Map in all browsers that can use schema object itself as a key.

For your particular case, you do not need to use json-schema-ref-parser, instead you need to change $refs in schemas to use schema $ids (absolute, or relative, but not file paths) and add both schemas to Ajv - it should correctly do reference resolution:

const Ajv = require("ajv").default
const ajv = new Ajv({
  schemas: [
    require("./samples/parent.schema.json"),
    require("./samples/child.schema.json"),
  ]
})
const validate = ajv.getSchema("https://schemas.example.com/samples/parent")

You could also use addSchema method to add them not via options:

const ajv = new Ajv()
ajv.addSchema(require("./samples/parent.schema.json"))
ajv.addSchema(require("./samples/child.schema.json"))

Parent schema has to be changed in this way (using relatively URI in $ref):

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "https://schemas.example.com/samples/parent",
  "title": "Schema that references another schema multiple times",
  "type": "object",
  "properties": {
    "name": {
      "type": "string"
    },
    "eldestChild": {
      "$ref": "child" // absolute URI can also be used here: https://schemas.example.com/samples/child
    },
    "youngestChild": {
      "$ref": "child"
    }
  }
}

This is working repl: https://runkit.com/esp/6020572c572324001a6a3e55

While Ajv does not inline the schema, it still, when possible, inlines the code - as in this case. This behaviour can be changed using the option inlineRefs.

If for some other usage you need a resolved schema you can still use json-schema-ref-parser, as Ajv does not produce it (in general case, it is not possible, because of recursion)

If for some other reason you do need a resolved schema AND it has to be the same schema that you pass to Ajv (although I cannot think why it may be required) then you need to preprocess the schema to make sure all $ids are unique (or they can be simply removed from the inlined fragments - they are not needed there.

Not sure what happened with the parent2 example, when I inline clild inside parent2 it compiles: https://runkit.com/esp/6020565602c3d0001aa8e771. It is possible that you have both parent and parent2 compiled by the same Ajv instance in which case child IDs would conflict for the same reason.

@epoberezkin epoberezkin changed the title Error when compiling schema if external schema referenced more than once, or from within definitions Inlining schemas with json-schema-ref-parser is unnecessary, and may not work in case inlined schemas have $id Feb 7, 2021
@benjamin-mogensen
Copy link

I use json-schema-ref-parser to create bundled and de-referenced schemas. The bundled schemas are primarily used for OpenAPI definitions as the tooling seem to deal very well with internal references while the fully de-referenced ones are used for software that have trouble resolving $ref (nothing to do with AJV or json-schema-ref-parser though).

For the bundling I had to program in to the script the removal of all the $id lines, with the exception of the first one encountered as I want to keep the schemas own id. This allowed the AJV (ajv-cli 3.3.0) to be able to validate the bundled JSON schemas.

However any later AJV version are not able to validate several of my bundled JSON schemas; it fails on resolving $refs that are referring to schemas defined within an item in an array definition. Unfortunately I cannot influence where json-schema-ref-parser decides to inline the re-used schemas so I am a bit stuck with ajv-cli 3.3.0 :) - but this is another issue of course...

@benjamin-mogensen
Copy link

Should mention that we found out that ajv-cli 3.3.0 has a big that causes it to report that schemas are valid even if they are not. This happens when I used glob pattern to validate many bundled schemas with ajv-cli. I have since moved to AJV newest version which correctly fails validation of the schemas - however some errors in the schemas result from bundling with json-schema-ref-parser as it creates these $ref to $ref paths which is not allowed.

@harveyconnor
Copy link

We are also getting this issue, and struggling to find any answers...

padamstx added a commit to padamstx/spectral that referenced this issue Feb 22, 2023
This commit modifies the oasExample function so
that example fields are removed from the schema
to be used for validation.  This is needed because
the presence of an "example" field in a schema
confuses ajv in certain scenarios.
References:
- stoplightio#2081
- stoplightio#2140
- ajv-validator/ajv#1426
padamstx added a commit to padamstx/spectral that referenced this issue Feb 23, 2023
This commit modifies the oasExample function so
that example fields are removed from the schema
to be used for validation.  This is needed because
the presence of an "example" field in a schema
confuses ajv in certain scenarios.
References:
- stoplightio#2081
- stoplightio#2140
- ajv-validator/ajv#1426
padamstx added a commit to padamstx/spectral that referenced this issue Mar 1, 2023
This commit modifies the oasExample function so
that example fields are removed from the schema
to be used for validation.  This is needed because
the presence of an "example" field in a schema
confuses ajv in certain scenarios.
References:
- stoplightio#2081
- stoplightio#2140
- ajv-validator/ajv#1426
padamstx added a commit to padamstx/spectral that referenced this issue Apr 5, 2023
This commit modifies the oasExample function so
that example fields are removed from the schema
to be used for validation.  This is needed because
the presence of an "example" field in a schema
confuses ajv in certain scenarios.
References:
- stoplightio#2081
- stoplightio#2140
- ajv-validator/ajv#1426
P0lip added a commit to stoplightio/spectral that referenced this issue Apr 25, 2023
* fix(rulesets): avoid false errors from ajv

This commit modifies the oasExample function so
that example fields are removed from the schema
to be used for validation.  This is needed because
the presence of an "example" field in a schema
confuses ajv in certain scenarios.
References:
- #2081
- #2140
- ajv-validator/ajv#1426

* docs(repo): fix lint warning in README

Signed-off-by: Phil Adams <phil_adams@us.ibm.com>

* chore(rulesets): use traverse

---------

Signed-off-by: Phil Adams <phil_adams@us.ibm.com>
Co-authored-by: Jakub Rożek <jakub@stoplight.io>
P0lip added a commit to stoplightio/spectral that referenced this issue Apr 25, 2023
* fix(rulesets): avoid false errors from ajv

This commit modifies the oasExample function so
that example fields are removed from the schema
to be used for validation.  This is needed because
the presence of an "example" field in a schema
confuses ajv in certain scenarios.
References:
- #2081
- #2140
- ajv-validator/ajv#1426

* docs(repo): fix lint warning in README

Signed-off-by: Phil Adams <phil_adams@us.ibm.com>

* chore(rulesets): use traverse

---------

Signed-off-by: Phil Adams <phil_adams@us.ibm.com>
Co-authored-by: Jakub Rożek <jakub@stoplight.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

No branches or pull requests

4 participants