toEmbeddedRefract

[char0n]

AFAICT the toEmbeddedRefract is not part of the library any more, yet it is still mentioned in the README. Is it correct observation?

[kylef]

That's right, we removed it in Minim 0.17 (https://github.com/refractproject/minim/releases/tag/v0.17.0). Missed that it was still in the README and I'll fix this shortly.

We settled on only supporting a single form of JSON serialisation for Refract (and API Elements) and it was removed from the Refract spec in 2017 via refractproject/refract-spec#73.

[char0n]

Cool, thanks for explanation ;]

[kylef]

With GZIP of JSON serialisation we don't currently face any performance problems due to the over-the-wire format, if we did in future it might make sense to explore binary serialisation formats like protobuf and the likes.

[char0n]

Yes, I remember having discussion about it with Wilda and we agreed GZIP + JSON is more than enough.

Maybe offtopic, but still related. What was the background for making the decision, that fury should resolve all the references $ref when parsing any ADD and encoding dereferenced ADD into Api Elements? Instead of parser handling this (which creates huge payloads), I'd expect it's a thing of runtime - when I say: "I want to resolve this ApiELements".

[kylef]

What was the background for making the decision, that fury should resolve all the references $ref when parsing any ADD and encoding dereferenced ADD into Api Elements? Instead of parser handling this (which creates huge payloads), I'd expect it's a thing of runtime - when I say: "I want to resolve this ApiELements".

I think the best approach for the parser is that it shouldn't blanket handle $refs but instead translate them to API Element references to push that down to API Element consumers to keep the parse result small, and to prevent duplication.

The parser(s) do need to ensure that the $ref makes sense within the context of where it is used. I think this is key to providing decent warning/error handling too so the parser is not duplicating work, and in turn validations. Generally we can parse each component (in OAS 3 terms) individually (a single time) and then validate it makes sense in the given context.

Theres some reasons where each individual parser may vary so I'll break them down below. For API Blueprint, this rule is followed with exception of body generation (more on that below). OpenAPI 2 (not 3) in particular is implemented different (more at the end).

General Exception: Message Body Examples

There are parser features (which IMO i'm not entirely sure they really belong in the parser, but they are here historically for performance reasons):

• HTTP Request/Response Body: Example Generation
• HTTP Request/Response Body Schema: JSON Schema Example Generation

Generating a JSON example body requires the dereferencing to take place which is the case where the API Blueprint (drafter) and OpenAPI 3 (fury-adapter-oas3-parser) parser will do dereferencing. JSON Schema generation is similiar, in most cases generated JSON Schemas are "self-contained" and themselves contain all their $ref which causes duplicate information across different self-contained JSON Schemas.

Long term, I don't think this work nessecerily should be done directly in the parser but its ended up that way for performance reasons in API Blueprint and the OAS 2/3 parsers follow the same design.

🥇 This feature of generating example message bodies and schemas is now optional (as of today) in apiaryio/api-elements.js#281 and available in subsequent releases of Drafter and the Fury adapters. Next step is to provide some kind of fury serialisers for these message body formats which I've outlined at apiaryio/api-elements.js#372.

The plan is that we can push some of this expansion and generation in this case down to the API Elements consumer and do it "on the fly" when they want JSON example bodies or JSON Schemas. This particular aspect counts for most of API Blueprint parsing time (apiaryio/api-elements.js#281 (comment)) and larger API Elements trees. It will remaing enabled by default to maintain current behaviour as to not break any tooling Apiary, and others have.

⚠️ Under some cases this approach will cause loss of information in particular for JSON Schema generation, that's because API Element's is not fully feature complete for Swagger Schema/OpenAPI Schema. Thus schemas generated directly from the Swager Schema directly to JSON Schema in fury-adapter-swagger contain more information that generating the JSON Schema from API Elements (which does not contain some schema constraints that cannot be expressed in API Elements). We need to support a constraint based system in API Elements to allow that (tracked in apiaryio/api-elements#63).

Thus this is something we're actively working towards changing but there's work to do:

• Provide adapters for consumers to generate (serialise) message bodies into other formats (JSON, multipart form, etc): Message body serialization adapters apiaryio/api-elements.js#372.
• Provide adapters to serialise a data structure into JSON Schema. This is part of the above, but to do right we need to bring JSON Schema <> API Element feature parity Full support for expressing JSON Schema constraints in API Element data structures apiaryio/api-elements#63. Generate JSON Schema messageBody asset from OpenAPI 3 Schema apiaryio/api-elements.js#96 is very related.

API Elements and/or tooling doesn't support referencing a particular component

While API elements has a generic referencing system, most tooling and Apiary products does not have generic support for referencing. For example adding referenced hrefVariables is theoretically possible for some aspects of the data, for example the following would be valid in API Elements:

{
  "element": "hrefVariables",
  "content": [
    {
      "element": "member",
      "content": {
        "key": { "element": "string", "content": "username" },
        "value": { "element": "username" }
      }
    }
  ]
}

Perhaps placing inside a data structure category:

{
  "element": "string",
  "meta": {
    "id": { "element": "string", "element": "username" }
  },
  "content": "doe"
}

Most tooling I am aware of (Dredd, ApiaryUI etc) don't allow for this and thus need adapting to move towards this direction. We should introduce further categories to be able to store particular elemenents which do not fit into the existing structure (such as reusable header structures -- possibly they can be put in data structures but they cannot contain header or parameter specific information). A problem here is that our referencing system does not have any concept of name spacing and is done via id's and thus translating OAS names into API Element's tree can have either collisions (if theres a Schema Object named "user" and a header object named "user"). The route that the OAS 2 parser went down to solve this is to namespace the reusable element id's such as definitions/user which has downsides because some documentation products show that as-is.

I don't think there are any GitHub issues tracking any of this. Although it will need to be solved to implement apiaryio/api-blueprint-rfcs#3 and full OpenAPI 3.0 compatibility in API Elements.

OpenAPI 2 Parser

❗ Our OpenAPI 2 parser (fury-adapter-swagger) is the exception to all of this, the way that parser was architected is that the parser takes the approach of dereferencing the entire tree upfront (because that's how swagger-parser, it's dependency does it). I think this is a poor approach which I've slowly been trying to solve to improve the performance of that particular adapter but it is something that has not been fully resolved (and I'm not sure it makes sense for us to invest effort in doing so).

[char0n]

Thank you for this analysis. I will have a lot of followup questions to this ;]

[char0n]

I think the best approach for the parser is that it shouldn't blanket handle $refs but instead translate them to API Element references to push that down to API Element consumers to keep the parse result small, and to prevent duplication.

The parser(s) do need to ensure that the $ref makes sense within the context of where it is used. I think this is key to providing decent warning/error handling too so the parser is not duplicating work, and in turn validations. Generally we can parse each component (in OAS 3 terms) individually (a single time) and then validate it makes sense in the given context.

So if I understand what you said correctly, what you're saying is: Parser do need to understand the JSON-References and should encode them to RefElements. And then every found reference ($ref) inside the tree should be decorated (for example in meta) by actual RefElement that have been created e.g. by calling toRef() on referenced Element. Would that be correct assumption ?