Add annotation support refactor keywords to use annotations implement output formats

CHANGELOG.md

@@ -1,8 +1,9 @@
 # Change Log
 All notable changes to this project will be documented in this file.
 
-The format is based on [Keep a Changelog](http://keepachangelog.com/)
-and this project adheres to [Semantic Versioning](http://semver.org/).
+This format is based on [Keep a Changelog](http://keepachangelog.com/).
+
+This project does not adhere to [Semantic Versioning](https://semver.org/) and minor version changes can have incompatible API changes. These incompatible API changes will largely affect those who have custom validator or walker implementations. Those who just use the library to validate using the standard JSON Schema Draft specifications may not need changes.
 
 ## [Unreleased]
 

doc/compatibility.md

@@ -1,13 +1,20 @@
 ## Compatibility with JSON Schema versions
 
-This implementation does not currently generate annotations.
-
 The `pattern` validator by default uses the JDK regular expression implementation which is not ECMA-262 compliant and is thus not compliant with the JSON Schema specification. The library can however be configured to use a ECMA-262 compliant regular expression implementation.
 
+Annotation processing and reporting are implemented. Note that the collection of annotations will have an adverse performance impact.
+
+This implements the Flag, List and Hierarchical output formats defined in the [Specification for Machine-Readable Output for JSON Schema Validation and Annotation](https://github.com/json-schema-org/json-schema-spec/blob/8270653a9f59fadd2df0d789f22d486254505bbe/jsonschema-validation-output-machines.md).
+
 ### Known Issues
-* The `anyOf` applicator currently returns immediately on matching a schema. This results in the `unevaluatedItems` and `unevaluatedProperties` keywords potentially returning an incorrect result as the rest of the schemas in the `anyOf` aren't processed.
-* The `unevaluatedItems` keyword does not currently consider `contains`.
 
+There are currently no known issues with the required functionality from the specification.
+
+The following are the tests results after running the [JSON Schema Test Suite](https://github.com/json-schema-org/JSON-Schema-Test-Suite) as at 29 Jan 2024 using version 1.3.1. As the test suite is continously updated, this can result in changes in the results subsequently.
+
+| Implementations | Overall                                                                 | DRAFT_03                                                          | DRAFT_04                                                            | DRAFT_06                                                           | DRAFT_07                                                               | DRAFT_2019_09                                                        | DRAFT_2020_12                                                          |
+|-----------------|-------------------------------------------------------------------------|-------------------------------------------------------------------|---------------------------------------------------------------------|--------------------------------------------------------------------|------------------------------------------------------------------------|----------------------------------------------------------------------|------------------------------------------------------------------------|
+| NetworkNt       | pass: r:4703 (100.0%) o:2369 (100.0%)<br>fail: r:0 (0.0%) o:1 (0.0%)    |                                                                   | pass: r:600 (100.0%) o:251 (100.0%)<br>fail: r:0 (0.0%) o:0 (0.0%)  | pass: r:796 (100.0%) o:318 (100.0%)<br>fail: r:0 (0.0%) o:0 (0.0%) | pass: r:880 (100.0%) o:541 (100.0%)<br>fail: r:0 (0.0%) o:0 (0.0%)     | pass: r:1201 (100.0%) o:625 (100.0%)<br>fail: r:0 (0.0%) o:0 (0.0%)  | pass: r:1226 (100.0%) o:634 (99.8%)<br>fail: r:0 (0.0%) o:1 (0.2%)     |
 
 ### Legend
 
@@ -79,15 +86,15 @@ The `pattern` validator by default uses the JDK regular expression implementatio
 
 #### Content Encoding
 
-Since Draft 2019-09, the `contentEncoding` keyword does not generate assertions. As the implementation currently does not collect annotations this only generates assertions in Draft 7.
+Since Draft 2019-09, the `contentEncoding` keyword does not generate assertions.
 
 #### Content Media Type
 
-Since Draft 2019-09, the `contentMediaType` keyword does not generate assertions. As the implementation currently does not collect annotations this only generates assertions in Draft 7.
+Since Draft 2019-09, the `contentMediaType` keyword does not generate assertions.
 
 #### Content Schema
 
-The `contentSchema` keyword does not generate assertions. As the implementation currently does not collect annotations this doesn't do anything.
+The `contentSchema` keyword does not generate assertions.
 
 #### Pattern
 
@@ -108,7 +115,7 @@ This also requires adding the `joni` dependency.
 </dependency>
 ```
 
-### Format
+#### Format
 
 Since Draft 2019-09 the `format` keyword only generates annotations by default and does not generate assertions.
 
@@ -119,7 +126,7 @@ This can be configured on a schema basis by using a meta schema with the appropr
 | Draft 2019-09         | `https://json-schema.org/draft/2019-09/vocab/format`          | `true`            |
 | Draft 2020-12         | `https://json-schema.org/draft/2020-12/vocab/format-assertion`| `true`/`false`    | 
 
-This behavior can be overridden to generate assertions on a per-execution basis by setting the `setFormatAssertionsEnabled` to `true`.
+This behavior can be overridden to generate assertions by setting the `setFormatAssertionsEnabled` option to `true`.
 
 | Format                | Draft 4 | Draft 6 | Draft 7 | Draft 2019-09 | Draft 2020-12 |
 |:----------------------|:-------:|:-------:|:-------:|:-------------:|:-------------:|
@@ -143,7 +150,7 @@ This behavior can be overridden to generate assertions on a per-execution basis
 | uri-template          | 🚫 | 🟢 | 🟢 | 🟢 | 🟢 |
 | uuid                  | 🚫 | 🚫 | 🟢 | 🟢 | 🟢 |
 
-### Footnotes
+##### Footnotes
 1. Note that the validation are only optional for some of the keywords/formats.
 2. Refer to the corresponding JSON schema for more information on whether the keyword/format is optional or not.
 

doc/quickstart.md

@@ -1,83 +1,65 @@
 ## Quick Start
 
-To use the validator, we need to have both the `JsonSchema` object and `JsonNode` object constructed. 
-There are many ways to do that. 
-Here is base test class, that shows several ways to construct these from `String`, `Stream`, `Url`, and `JsonNode`. 
-Please pay attention to the `JsonSchemaFactory` class as it is the way to construct the `JsonSchema` object.
+To use the validator, we need to have the `JsonSchema` loaded and cached.
 
-```java
-public class BaseJsonSchemaValidatorTest {
+For simplicity the following test loads a schema from a `String` or `JsonNode`. Note that loading a schema in this manner is not recommended as a relative `$ref` will not be properly resolved as there is no base IRI.
 
-    private ObjectMapper mapper = new ObjectMapper();
+The preferred method of loading a schema is by using a `SchemaLocation` and by configuring the appropriate `SchemaMapper` and `SchemaLoader` on the `JsonSchemaFactory`.
 
-    protected JsonNode getJsonNodeFromClasspath(String name) throws IOException {
-        InputStream is1 = Thread.currentThread().getContextClassLoader()
-                .getResourceAsStream(name);
-        return mapper.readTree(is1);
-    }
+```java
+package com.example;
 
-    protected JsonNode getJsonNodeFromStringContent(String content) throws IOException {
-        return mapper.readTree(content);
-    }
+import static org.junit.jupiter.api.Assertions.assertEquals;
 
-    protected JsonNode getJsonNodeFromUrl(String url) throws IOException {
-        return mapper.readTree(new URL(url));
-    }
+import java.util.Set;
 
-    protected JsonSchema getJsonSchemaFromClasspath(String name) {
-        JsonSchemaFactory factory = JsonSchemaFactory.getInstance(SpecVersion.VersionFlag.V4);
-        InputStream is = Thread.currentThread().getContextClassLoader()
-                .getResourceAsStream(name);
-        return factory.getSchema(is);
-    }
+import org.junit.jupiter.api.Test;
 
-    protected JsonSchema getJsonSchemaFromStringContent(String schemaContent) {
-        JsonSchemaFactory factory = JsonSchemaFactory.getInstance(SpecVersion.VersionFlag.V4);
-        return factory.getSchema(schemaContent);
-    }
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.JsonMappingException;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.networknt.schema.*;
+import com.networknt.schema.serialization.JsonMapperFactory;
 
-    protected JsonSchema getJsonSchemaFromUrl(String uri) throws URISyntaxException {
+public class SampleTest {
+    @Test
+    void schemaFromString() throws JsonMappingException, JsonProcessingException {
         JsonSchemaFactory factory = JsonSchemaFactory.getInstance(SpecVersion.VersionFlag.V4);
-        return factory.getSchema(SchemaLocation.of(uri));
+        /*
+         * This should be cached for performance.
+         * 
+         * Loading from a String is not recommended as there is no base IRI to use for
+         * resolving relative $ref.
+         */
+        JsonSchema schemaFromString = factory
+                .getSchema("{\"enum\":[1, 2, 3, 4],\"enumErrorCode\":\"Not in the list\"}");
+        Set<ValidationMessage> errors = schemaFromString.validate("7", InputFormat.JSON);
+        assertEquals(1, errors.size());
     }
 
-    protected JsonSchema getJsonSchemaFromJsonNode(JsonNode jsonNode) {
+    @Test
+    void schemaFromJsonNode() throws JsonMappingException, JsonProcessingException {
         JsonSchemaFactory factory = JsonSchemaFactory.getInstance(SpecVersion.VersionFlag.V4);
-        return factory.getSchema(jsonNode);
-    }
-
-    // Automatically detect version for given JsonNode
-    protected JsonSchema getJsonSchemaFromJsonNodeAutomaticVersion(JsonNode jsonNode) {
-        JsonSchemaFactory factory = JsonSchemaFactory.getInstance(SpecVersionDetector.detect(jsonNode));
-        return factory.getSchema(jsonNode);
-    }
-
-}
-```
-And the following is one of the test cases in one of the test classes that extend from the above base class. As you can see, it constructs `JsonSchema` and `JsonNode` from `String`.
-
-```java
-class Sample extends BaseJsonSchemaValidatorTest {
-
-    void test() {    
-        JsonSchema schema = getJsonSchemaFromStringContent("{\"enum\":[1, 2, 3, 4],\"enumErrorCode\":\"Not in the list\"}");
-        JsonNode node = getJsonNodeFromStringContent("7");
-        Set<ValidationMessage> errors = schema.validate(node);
-        assertThat(errors.size(), is(1));
-
-        // With automatic version detection
-        JsonNode schemaNode = getJsonNodeFromStringContent(
-            "{\"$schema\": \"http://json-schema.org/draft-06/schema#\", \"properties\": { \"id\": {\"type\": \"number\"}}}");
-        JsonSchema schema = getJsonSchemaFromJsonNodeAutomaticVersion(schemaNode);
-
-        schema.initializeValidators(); // by default all schemas are loaded lazily. You can load them eagerly via
-                                       // initializeValidators() 
-
-        JsonNode node = getJsonNodeFromStringContent("{\"id\": \"2\"}");
-        Set<ValidationMessage> errors = schema.validate(node);
-        assertThat(errors.size(), is(1));
+        JsonNode schemaNode = JsonMapperFactory.getInstance().readTree(
+                "{\"$schema\": \"http://json-schema.org/draft-06/schema#\", \"properties\": { \"id\": {\"type\": \"number\"}}}");
+        /*
+         * This should be cached for performance.
+         * 
+         * Loading from a JsonNode is not recommended as there is no base IRI to use for
+         * resolving relative $ref.
+         *
+         * Note that the V4 from the factory is the default version if $schema is not
+         * specified. As $schema is specified in the data, V6 is used.
+         */
+        JsonSchema schemaFromNode = factory.getSchema(schemaNode);
+        /*
+         * By default all schemas are preloaded eagerly but ref resolve failures are not
+         * thrown. You check if there are issues with ref resolving using
+         * initializeValidators()
+         */
+        schemaFromNode.initializeValidators();
+        Set<ValidationMessage> errors = schemaFromNode.validate("{\"id\": \"2\"}", InputFormat.JSON);
+        assertEquals(1, errors.size());
     }
-
 }
-
 ```

doc/upgrading.md

@@ -1,7 +1,76 @@
 ## Upgrading to new versions
 
+This library can contain breaking changes in `minor` version releases.
+
 This contains information on the notable or breaking changes in each version.
 
+### 1.3.1
+
+This does not contain any breaking changes from 1.3.0
+
+* Annotation collection and reporting has been implemented
+* Keywords have been refactored to use annotations for evaluation to improve performance and meet functional requirements
+* The list and hierarchical output formats have been implemented as per the [Specification for Machine-Readable Output for JSON Schema Validation and Annotation](https://github.com/json-schema-org/json-schema-spec/blob/main/jsonschema-validation-output-machines.md).
+* The fail fast evaluation processing has been redesigned and fixed. This currently passes the [JSON Schema Test Suite](https://github.com/json-schema-org/JSON-Schema-Test-Suite) with fail fast enabled. Previously contains and union type may cause incorrect results.
+* This also contains fixes for regressions introduced in 1.3.0
+
+The following keywords were refactored to improve performance and meet the functional requirements.
+
+In particular this converts the `unevaluatedItems` and `unevaluatedProperties` validators to use annotations to perform the evaluation instead of the current mechanism which affects performance. This also refactors `$recursiveRef` to not rely on that same mechanism.
+
+* `unevaluatedProperties`
+* `unevaluatedItems`
+* `properties`
+* `patternProperties`
+* `items` / `additionalItems`
+* `prefixItems` / `items`
+* `contains`
+* `$recursiveRef`
+
+This also fixes the issue where the `unevaluatedItems` keyword does not take into account the `contains` keyword when performing the evaluation.
+
+This also fixes cases where `anyOf` short-circuits to not short-circuit the evaluation if a adjacent `unevaluatedProperties` or `unevaluatedItems` keyword exists.
+
+This should fix most of the remaining functional and performance issues.
+
+#### Functional
+
+| Implementations | Overall                                                                 | DRAFT_03                                                          | DRAFT_04                                                            | DRAFT_06                                                           | DRAFT_07                                                               | DRAFT_2019_09                                                        | DRAFT_2020_12                                                          |
+|-----------------|-------------------------------------------------------------------------|-------------------------------------------------------------------|---------------------------------------------------------------------|--------------------------------------------------------------------|------------------------------------------------------------------------|----------------------------------------------------------------------|------------------------------------------------------------------------|
+| NetworkNt       | pass: r:4703 (100.0%) o:2369 (100.0%)<br>fail: r:0 (0.0%) o:1 (0.0%)    |                                                                   | pass: r:600 (100.0%) o:251 (100.0%)<br>fail: r:0 (0.0%) o:0 (0.0%)  | pass: r:796 (100.0%) o:318 (100.0%)<br>fail: r:0 (0.0%) o:0 (0.0%) | pass: r:880 (100.0%) o:541 (100.0%)<br>fail: r:0 (0.0%) o:0 (0.0%)     | pass: r:1201 (100.0%) o:625 (100.0%)<br>fail: r:0 (0.0%) o:0 (0.0%)  | pass: r:1226 (100.0%) o:634 (99.8%)<br>fail: r:0 (0.0%) o:1 (0.2%)     |
+
+#### Performance
+
+##### NetworkNT 1.3.1
+
+```
+Benchmark                                                          Mode  Cnt       Score      Error   Units
+NetworkntBenchmark.testValidate                                   thrpt   10    6776.693 ±  115.309   ops/s
+NetworkntBenchmark.testValidate:·gc.alloc.rate                    thrpt   10     971.191 ±   16.420  MB/sec
+NetworkntBenchmark.testValidate:·gc.alloc.rate.norm               thrpt   10  165318.816 ±    0.459    B/op
+NetworkntBenchmark.testValidate:·gc.churn.G1_Eden_Space           thrpt   10     968.894 ±   51.234  MB/sec
+NetworkntBenchmark.testValidate:·gc.churn.G1_Eden_Space.norm      thrpt   10  164933.962 ± 8636.203    B/op
+NetworkntBenchmark.testValidate:·gc.churn.G1_Survivor_Space       thrpt   10       0.002 ±    0.001  MB/sec
+NetworkntBenchmark.testValidate:·gc.churn.G1_Survivor_Space.norm  thrpt   10       0.274 ±    0.218    B/op
+NetworkntBenchmark.testValidate:·gc.count                         thrpt   10      89.000             counts
+NetworkntBenchmark.testValidate:·gc.time                          thrpt   10      99.000                 ms
+```
+
+###### Everit 1.14.1
+
+```
+Benchmark                                                          Mode  Cnt       Score       Error   Units
+EveritBenchmark.testValidate                                      thrpt   10    3719.192 ±   125.592   ops/s
+EveritBenchmark.testValidate:·gc.alloc.rate                       thrpt   10    1448.208 ±    74.746  MB/sec
+EveritBenchmark.testValidate:·gc.alloc.rate.norm                  thrpt   10  449621.927 ±  7400.825    B/op
+EveritBenchmark.testValidate:·gc.churn.G1_Eden_Space              thrpt   10    1446.397 ±    79.919  MB/sec
+EveritBenchmark.testValidate:·gc.churn.G1_Eden_Space.norm         thrpt   10  449159.799 ± 18614.931    B/op
+EveritBenchmark.testValidate:·gc.churn.G1_Survivor_Space          thrpt   10       0.001 ±     0.001  MB/sec
+EveritBenchmark.testValidate:·gc.churn.G1_Survivor_Space.norm     thrpt   10       0.364 ±     0.391    B/op
+EveritBenchmark.testValidate:·gc.count                            thrpt   10     133.000              counts
+EveritBenchmark.testValidate:·gc.time                             thrpt   10     148.000                  ms
+```
+
 ### 1.3.0
 
 This adds support for Draft 2020-12

src/main/java/com/networknt/schema/AbstractCollector.java

@@ -15,6 +15,11 @@
  */
 package com.networknt.schema;
 
+/**
+ * Base collector.
+ * 
+ * @param <E> the type
+ */
 public abstract class AbstractCollector<E> implements Collector<E> {
 
     @Override