Multiple array items type validation with oneOf, anyOf

[mgendre]

Hi everyone,

I've encountered a problem to validate multiple items type in an array.

We have an array with 2 items types. We try to validate each item to 2 different schemas types.

Here is the schema:

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "type": "object",
  "properties": {
    "transfer": {
      "type": "object",
      "properties": {
        "modes": {
          "type": "array",
          "minItems": 1,
          "additionalItems": false,
          "items": {
            "oneOf": [
              {
                "type": "object",
                "properties": {
                  "mode": {
                    "enum": [
                      "ftp"
                    ]
                  },
                  "account": {
                    "type": "string",
                    "pattern": "^[a-f|0-9]{8}$"
                  }
                },
                "required": [
                  "mode",
                  "account"
                ],
                "additionalProperties": false
              },
              {
                "type": "object",
                "properties": {
                  "mode": {
                    "enum": [
                      "email"
                    ]
                  },
                  "mailingList": {
                    "type": "string",
                    "pattern": "^[a-f|0-9]{8}$"
                  }
                },
                "required": [
                  "mode",
                  "mailingList"
                ],
                "additionalProperties": false
              }
            ]
          }
        }
      },
      "required": [
        "modes"
      ],
      "additionalProperties": false
    }
  },
  "additionalProperties": false,
  "required": [
    "transfer"
  ]
}

And the data:

{
  "transfer": {
    "modes": [
      {
        "mode": "ftp",
        "account": "e69b3f54"
      },
      {
        "mode": "email",
        "mailingList": "c3d12752"
      }
    ]
  }
}

I tried to validate the data against the schema to different online tools:
http://json-schema-validator.herokuapp.com/index.jsp
http://jsonschemalint.com/draft4/#

All these tools validate successfully my data

It's like the validator try to match the 1st item to the 1st schema and the second item to the second schema. Order should be not important.

I use version 3.8.1.

Could you help me ?

Thank you in advance

[mgendre]

When I remove the configuration "additionalProperties": false from the type definition on "oneOf" the problem disappear.

[epoberezkin]

What are your ajv options? I suspect you may be using removeAdditional option. Without it it validates ok: https://tonicdev.com/esp/56d57c034b05fd0c00e8978b

If that is the case this issue is essentially a duplicate of #129 and it is an expected behaviour according to the JSON-schema standard.

[epoberezkin]

I would suggest refactoring the schema to take the duplicated keywords and all property definitions apart from required out of oneOf:

"items": {
  "type": "object",
  "properties": {
    "mode": {
      "enum": [
        "ftp"
      ]
    },
    "account": {
      "type": "string",
      "pattern": "^[a-f|0-9]{8}$"
    },
    "mailingList": {
      "type": "string",
      "pattern": "^[a-f|0-9]{8}$"
    }
  },
  "additionalProperties": false,
  "oneOf": [
    { "required": [ "mode", "account" ] },
    { "required": [ "mode", "mailingList" ] }
  ]
}

This way it will work with removeAdditional option the way you expect it to.

The previous schema works correctly, but not how you expect probably

[mgendre]

You are right @epoberezkin !

AJV is configured with removeAdditional -> true. The schema has been refactored to avoid multiple definitions inside oneOf, like you suggested and it's working.

Thank you @epoberezkin for your support !

[epoberezkin]

You're welcome. I'll add a note in readme about it :)

[ribaptista]

I noticed the original schema would not validate an object containing both account and mailingList properties (that's expected), while the later refactored schema does.
Is there any workaround for this?

[epoberezkin]

You can use { "not": { "required": ["account"] } } to explicitly prohibit property account.

[ngryman]

@epoberezkin Sorry to revive this but what if you do need to reference external schemas in oneOf and can't really split them like your proposed solution?

To take the above example with ref:

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "type": "object",
  "properties": {
    "transfer": {
      "type": "object",
      "properties": {
        "modes": {
          "type": "array",
          "minItems": 1,
          "additionalItems": false,
          "items": {
            "oneOf": [
              { "$ref": "ftp" },
              { "$ref": "email" }
            ]
          }
        }
      },
      "required": [
        "modes"
      ],
      "additionalProperties": false
    }
  },
  "additionalProperties": false,
  "required": [
    "transfer"
  ]
}

Thanks!

[epoberezkin]

Btw, "additionalItems": false does nothing unless the schema in items keyword is an array.

The solutions to additionalProperties limitations are:

• don't worry about additionalProperties and simply allow them. I think the obsession with removing them is not always justified, in many cases you make your life MUCH easier by allowing them (especially when you have multiple applications using the same data/schema). The argument against it though is that by allowing additional properties you may miss some typos in property names.
• refactor schemas as suggested above
• use custom switch keyword (or if/then/else) instead of anyOf - it allows to control what is validated based on some conditions (so only correct properties will be treated as additional)
• use $merge/$patch keywords to extend schemas rather than joining them with compound keywords such as anyOf etc.

Please see FAQ that points to other related issues that show these approaches to using additionalProperties.

[dnvyrn]

schema:

"items": {
  "type": "object",
  "properties": {
    "mode": { "enum": [  "ftp" ]
    },
    "account": { "type": "string" },
    "mailingList": { "type": "string" },
    "otherProp": { "type": "string" }
  },
  "additionalProperties": false,
  "oneOf": [
    { "required": [ "mode", "account" ] },
    { "required": [ "mode", "mailingList", "otherProp" ] }
  ]
}

data:

{
  "transfer": {
    "modes": [
      {
        "mode": "ftp",
        "account": "e69b3f54",
        "mailingList": "c3d12752",
      },
      {
        "mode": "email",
        "mailingList": "c3d12752",
        "otherProp": "fasfasfaf"
      }
    ]
  }
}

It will validate pass. How to use { "not": { "required": ["account"] } } ?

[epoberezkin]

I don’t understand the question - what are you trying to achieve? “not” can be a sibling to required and any other keyword, if that’s what you are asking.

Also it as a general JSON schema question - it should be posted to Stack Overflow.

[wigy-opensource-developer]

#134 (comment)

I cannot express enough that this is an actual bug in the removeAdditional implementation that interferes with backtracking, it is not just a valid speed optimization. The removeAdditional must only remove properties after the anyOf decided on which branch to take in the matching. If any backtracking is still ongoing, no properties should be removed. As I read through your comments in related issues, I will not be able to convince you about this though 😄

In the workaround you provided above, if I fix the mode property (discriminator) to what it was in the schema OP posted:

...
"mode": {
  "enum": [
    "ftp",
    "email"
  ]
},
...

I also had to modify the oneOf

"oneOf": [
  { "mode": { "enum": [ "ftp" ] }, "required": [ "mode", "account" ] },
  { "mode": { "enum": [ "email" ] }, "required": [ "mode", "mailingList" ] }
]

otherwise it validates something with mode: email and having an account property. It works now, but cannot be modularized by using refs inside the oneOf as OP already mentioned.

See the working fix: https://runkit.com/wigy-opensource-developer/ajv-remove-additional-gotcha

[epoberezkin]

@wigy-opensource-developer I understand that the way removeAdditional option is defined may be not fit for your purposes, it’s a much simpler thing than you want it to be.

It is consistent though with how additionalProperties keyword is defined in JSON Schema spec draft-07 and it’s a conscious design choice for Ajv.

What is important the behaviour that is created by “removeAdditional” option is not part of JSON Schema spec.

I am not 100% sure what do you mean by “backtracking“ - Ajv does no such thing. Also anyOf does not at any point decide which branch to use “to match against the data” - it simply follows the algorithm prescribed by the specification, which is to validate the data against all branches and if the data is valid against at least one branch, then the data is valid against anyOf keyword - it’s simply Boolean “or” operation on the validation results against all subschemas. At no point any decision is made about one branch “matching” the data better than the other. Ajv obviously short-circuits it and stops as soon as it finds the first branch against which the data is valid, but it is not the source of the problem you want to solve - you want the decision about which properties to remove postpone until Ajv decides what is/are the right branch(es).

As a side but related comment, it’s a common practice of JSON Schema users to use “schema matching the data” and “data valid against the schema” as equivalent interchangeable statements, but they are not equivalent; the spec only defines what the second statement means (“data valid against the schema” - just the Boolean outcome of a formal process of validating the data against each set of dependent keywords following a certain algorithm for each set of keywords) and the users also have reasonable expectations about what the first statement means (“schema matching the data” - meaning schema looking “similar” to the data, but there is no formal definition of what it actually means); the users equate these two statements that leads to massive confusion, frustration and complicates JSON schema adoption) cc @Relequestual.

Also JSON Schema has no concept of “discriminator” - it only exists as part of OpenAPI spec in connection with oneOf keyword (not anyOf), and even there, reading their v3 spec, from the validation result point of view “discriminator” is no-op - it is only there to optimise, to support tools, to choose the branch that is possibly correct based on one property and have more helpful error reporting - but its presence should have no effect on validation result (as in valid/invalid). We can discuss it further, but it’s my reading of OpenAPI spec.

There is a ticket #1119 that is about “discriminator” and how it could be supported by Ajv to improve error reporting and to optimise validation process. As a side effect, it would also lead to changes in removeAdditional behaviour that would likely make its behaviour closer to what you want (unless it’s purposefully prevented). I am not 100% sure yet whether it is a good or bad thing, it would definitely make it only more confusing though. So removeAdditional option behaviours have to be considered in the context of that ticket by whoever implements it.

[wigy-opensource-developer]

Thanks for your time to explain all these. Many people who just try to build on their computer science backround of algebraic types or who just used to work with unions in C or enums in Rust will have to relearn this working with Ajv.

How I see now is that if a developer has a choice to define their own schema (which is of course not always the case), they should rather use a key as a discriminator rather than a value.

{
  "modes": [
    { "ftp": { "account": "blah" }},
    { "email": { "mailingList": "foobar" }}
  ]
}

[epoberezkin]

I completely agree that JSON Schema is very counter-intuitive to anybody who has CS background and expects it to be in any way similar to algebraic types (frankly, it is confusing to almost anybody who adopts it - many design decisions of JSON Schema are counter-intuitive). I’ve been trying to make this point to the community behind JSON Schema spec, on many occasions, but the attempts to reconcile it only made it worse resulting in draft-08... So, unfortunately, it’s not Ajv issue that I can address without abandoning JSON Schema spec - it is the fundamental issue of the specification itself. Still, I strongly believe that somewhat confusing, but widely adopted and well documented spec, that it supported, at least partially, across many languages and platforms, is much better than ad hoc decisions about schema language made on a library level - there are many such libraries and none of them is widely adopted or cross-platform. It is a shame that JSON Schema, in my opinion, is in some evolutionary dead end at the moment (I am referring to draft-08) - I do hope that it will be resolved in due course. I do think that draft-07 is quite usable though, and once you accept its restrictions and differences with type systems it can be used and give you a lot of value. There is no point trying to make it into what it’s not, it just requires a bit of understanding and acceptance... Some developers don’t author JSON Schemas directly - instead they compile typescript types into schemas to do run-time validation - this approach makes lots of sense to me. Hope it all makes sense, and if you want to contribute to discriminator implementation (the issue I referenced) it will likely address this issue as well. cc @Relequestual

…

On 12 Dec 2019, at 06:56, wigy ***@***.***> wrote:  Thanks for your time to explain all these. Many people who just try to build on their computer science backround of algebraic types or who just used to work with unions in C or enums in Rust will have to relearn this working with Ajv. How I see now is that if a developer has a choice to define their own schema (which is of course not always the case), they should rather use a key as a discriminator rather than a value. { "modes": { "ftp": { "account": "blah" }, "email": { "mailingList": "foobar" } } } — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[hansiemithun]

I have just handled like this for now with sass:

Maybe it will benefit some

.error-detail {
  .text-danger{
    display: none;
    &:first-child{
      display: block;
    }
  }
}