Description
DataFrame has the ability to import Data Schemas from the type definitions of OpenAPI files: https://kotlin.github.io/dataframe/schemasimportopenapigradle.html, https://kotlin.github.io/dataframe/schemasimportopenapijupyter.html.
This works, but I keep feeling we're missing something:
Just Types?
OpenAPI files contain both the API endpoints (aka urls, parameters etc.) as well as the type schema's of potentially returned types of these endpoints. Currently, DataFrame ignores endpoints and just extracts the types to convert them to DataSchema's. It's up to the user to call the API itself using DataFrame.readJson("url") and convert the result to the generated type (or they call GeneratedType.readJson("url")), however, all of this breaks down when the API has any authentication or requires a PUT instead of a GET request.
At the moment, even with our OpenAPI support, it's not trivial to use DataFrame with a Json API.
Also, OpenAPI with "just types" has a name, it's JSON Schema, but that we don't support right now: #683
Issues with implementation
Our implementation also has a couple issues:
- We don't support OpenAPI 3.1 OpenApi 3.1 and JSON Schema support #683
- Bugs: OpenAPI: Circular references break generation #288, OpenAPI: broken error reporting and code generation for two-dimensional array inside an object #255
- Annoying logs: isOpenApiStr logger leaks to Gradle #785
- No JSON Schema support: OpenApi 3.1 and JSON Schema support #683
- While we have plenty of tests for our DataSchema generation, we know it's not perfect and there are probably many breaking cases to be found for valid OpenAPI schemas.
- Feature: JSON parse options #459
How forward
I have a couple of ideas of how to remedy this situation, but feedback would be welcome.
First of all, we need to take a look at other tools to handle the code generation. An (apparently) industry leading project appears to be OpenAPITools OpenAPI Generator. It can for instance generate a ktor project automatically and with a lot of options. This project can then be used as a dependency in a notebook (even now!) or in another project. I found it works wonders together with our new compiler plugin in this example project. We can use any JSON parser the generator provides, as converting Kotlin objects to DataFrames is easy :) (except perhaps for maps).
However, for this to take off, I'd imagine a situation where a user would just have to import an openAPI file with a single statement and everything would be generated, imported, and ready to go. Especially in notebooks, this would be awesome.
Next, we should probably support "just" Json Schemas to generate DataSchemas. A user could still feed it an openapi file, since they're now a superset of json schema, but it would make it easier to market for sure.
What do you think?