Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: implement JSON Schema to NodeType adapter #1401

Merged
merged 12 commits into from
Feb 16, 2024

Conversation

tatomyr
Copy link
Contributor

@tatomyr tatomyr commented Jan 20, 2024

What/Why/How?

Implemented JSON Schema to NodeType adapter.

This streamlines merging external config schemas while allowing deriving TypeScript types from JSON Schema schemas.

The adapter itself is here. It uses AJV to determine the correct oneOf schema if there's no discriminator defined.

The adapter is limited to JSON Schemas that are not booleans or arrays themselves, nor types are arrays.
Also, the schemas should not contain allOf or anyOf (we're not using them anyway).
The most crucial limitation is that an array's items could not contain oneOf. I even had to change the adapters schema schema of the developerOnboarding section. That is because NodeType cannot alter items (like it can do with properties or additionalProperties). -- Upd: fixed in 2fe3a1c


I also fixed a wrong snapshot test, added missing configurable rules subject types, and updated the config schema to the latest version.

Reference

Testing

Performance

Investigated performance losses. It doesn't do that bad on linting. For the current Rebilly API description it doesn't show any significant difference comparing to other versions:

Command Mean [s] Min [s] Max [s] Relative
node node_modules/cli-140/bin/cli.js lint api-definitions/openapi/openapi.yaml 1.471 ± 0.035 1.431 1.554 1.00 ± 0.03
node node_modules/cli-150/bin/cli.js lint api-definitions/openapi/openapi.yaml 1.470 ± 0.022 1.430 1.501 1.00 ± 0.02
node node_modules/cli-160/bin/cli.js lint api-definitions/openapi/openapi.yaml 1.468 ± 0.017 1.454 1.511 1.00
node node_modules/cli-170/bin/cli.js lint api-definitions/openapi/openapi.yaml 1.514 ± 0.023 1.481 1.547 1.03 ± 0.02
node node_modules/cli-180/bin/cli.js lint api-definitions/openapi/openapi.yaml 1.492 ± 0.020 1.452 1.514 1.02 ± 0.02
node node_modules/cli-next/bin/cli.js lint api-definitions/openapi/openapi.yaml 1.496 ± 0.025 1.464 1.537 1.02 ± 0.02

On the other hand, for smaller museum.yaml API description the losses are higher (which most likely means the config itself gets resolved longer):

Command Mean [ms] Min [ms] Max [ms] Relative
node node_modules/cli-140/bin/cli.js lint ../resources/museum.yaml 469.3 ± 11.5 457.2 491.4 1.00
node node_modules/cli-150/bin/cli.js lint ../resources/museum.yaml 472.4 ± 5.9 464.5 482.1 1.01 ± 0.03
node node_modules/cli-160/bin/cli.js lint ../resources/museum.yaml 479.0 ± 13.3 467.3 504.1 1.02 ± 0.04
node node_modules/cli-170/bin/cli.js lint ../resources/museum.yaml 473.1 ± 5.0 467.2 482.0 1.01 ± 0.03
node node_modules/cli-180/bin/cli.js lint ../resources/museum.yaml 485.0 ± 4.0 477.5 489.3 1.03 ± 0.03
node node_modules/cli-next/bin/cli.js lint ../resources/museum.yaml 490.0 ± 3.2 485.1 495.6 1.04 ± 0.03

Screenshots (optional)

Check yourself

  • Code is linted
  • Tested with redoc/reference-docs/workflows (internal)
  • All new/updated code is covered with tests

Security

  • Security impact of change has been considered
  • Code follows company security practices and guidelines

@tatomyr tatomyr self-assigned this Jan 20, 2024
Copy link

changeset-bot bot commented Jan 20, 2024

⚠️ No Changeset found

Latest commit: ab62c1a

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

Copy link
Contributor

github-actions bot commented Jan 20, 2024

Command Mean [s] Min [s] Max [s] Relative
redocly lint packages/core/src/benchmark/benches/rebilly.yaml 1.087 ± 0.031 1.054 1.162 1.00
redocly-next lint packages/core/src/benchmark/benches/rebilly.yaml 1.113 ± 0.017 1.091 1.136 1.02 ± 0.03

Copy link
Contributor

github-actions bot commented Jan 20, 2024

Coverage report

St.
Category Percentage Covered / Total
🟡 Statements 76.49% 4425/5785
🟡 Branches 66.4% 2336/3518
🟡 Functions 69.86% 721/1032
🟡 Lines 76.66% 4156/5421
Show new covered files 🐣
St.
File Statements Branches Functions Lines
🟢
... / json-schema-adapter.ts
83.87% 75% 100% 82.76%
Show files with reduced coverage 🔻
St.
File Statements Branches Functions Lines
🟢 core/src/resolve.ts 95% 91.49% 100% 94.77%

Test suite run success

712 tests passing in 101 suites.

Report generated by 🧪jest coverage report action from ab62c1a

@tatomyr tatomyr force-pushed the json-schema-to-node-type-adapter branch from c59c675 to 31d2a6e Compare February 3, 2024 17:30
@tatomyr tatomyr marked this pull request as ready for review February 7, 2024 08:07
@tatomyr tatomyr requested a review from a team as a code owner February 7, 2024 08:07
@tatomyr tatomyr marked this pull request as draft February 9, 2024 13:02
@tatomyr tatomyr force-pushed the json-schema-to-node-type-adapter branch from 2fe3a1c to b72833f Compare February 9, 2024 14:05
@tatomyr tatomyr requested a review from RomanHotsiy February 9, 2024 19:07
packages/core/src/lint.ts Outdated Show resolved Hide resolved
packages/core/src/resolve.ts Outdated Show resolved Hide resolved
? itemsType(node[i], joinPointer(nodeAbsoluteRef, i))
: itemsType;
// we continue resolving unknown types, but stop early on known scalars
if (itemType === undefined && type !== unknownType && type !== SpecExtension) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this if statement maybe be causing slower benchmark but looks like we need this code

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did not review this file.

@@ -126,7 +127,7 @@ export async function lintConfig(opts: {
rules: { spec: 'error' },
});

const types = normalizeTypes(ConfigTypes, config);
const types = normalizeTypes(opts.externalConfigTypes || ConfigTypes, config);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to move to universal plugins approach so won't need it.
Let's talk offline.

packages/core/src/types/redocly-yaml.ts Outdated Show resolved Hide resolved
@tatomyr tatomyr force-pushed the json-schema-to-node-type-adapter branch from 872292a to 672bc50 Compare February 14, 2024 10:44
@tatomyr tatomyr marked this pull request as ready for review February 14, 2024 13:30
@tatomyr tatomyr force-pushed the json-schema-to-node-type-adapter branch from a87cc5f to f748ab6 Compare February 14, 2024 19:54
@tatomyr tatomyr requested a review from RomanHotsiy February 14, 2024 20:13
Copy link
Contributor

@IgorKarpiuk IgorKarpiuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@tatomyr tatomyr force-pushed the json-schema-to-node-type-adapter branch from f748ab6 to ef029ff Compare February 16, 2024 10:58
…config schemas according to the latest monorepo changes
update perf benchmark

switch to museum for a moment

improve walk and resolve for performance

cleanup the code

change fn syntax

make the adapter errors nicer
@tatomyr tatomyr force-pushed the json-schema-to-node-type-adapter branch from ef029ff to ab62c1a Compare February 16, 2024 10:59
Copy link
Contributor

github-actions bot commented Feb 16, 2024

Command Mean [ms] Min [ms] Max [ms] Relative
redocly lint packages/core/src/benchmark/benches/rebilly.yaml 992.7 ± 63.7 943.3 1156.6 1.00
redocly-next lint packages/core/src/benchmark/benches/rebilly.yaml 1007.1 ± 17.3 985.1 1037.1 1.01 ± 0.07

@tatomyr tatomyr merged commit fc21cfd into main Feb 16, 2024
30 checks passed
@tatomyr tatomyr deleted the json-schema-to-node-type-adapter branch February 16, 2024 11:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants