Schema parser test using Apis.guru (about 2128 new test cases)

[sergey-tihon]

I am the author of SwaggerProvider the F# type provider for Swagger 2.0. This is a small tool that parses schemas and generates .NET types defined in schema and methods for paths to call API (compatible with F# & C#)

For couple years I do test my own F# schema parser using APIs.guru collection of schemas, to be sure that I cover all real-world schemas in my parser. (APIs.guru team also maintain json with the list of links to all schemas in the collection in both formats json and yaml - https://api.apis.guru/v2/list.json)

I would really love to stop supporting my own parser and migrate to Microsoft.OpenApi.Readers, that not simplify my work, but also will allow supporting OpenAPI v3 schemas much easier.

BUT, seems current state of readers is not perfect yet, I did run it on APIs.guru schemas and saw couple parse failures: runtime crashes with StackOverflow, NullReferenceException, duplicate key in dictionary and etc.

I propose you also to use these schemas to test OpenApiStreamReader. If you cannot merge this PR for some reason, please take a look at failed test cases to find and fix bugs in OpenApiStreamReader.

Thanks.

[msftclas]

All CLA requirements met.

[PerthCharern]

Thanks @sergey-tihon

These APIs are great for testing; we should make sure our reader works against them. I am not sure if we want to make an out-of-box call in our unit tests given that the tests are used in CI, so we will have to figure out the best way to integrate them.

For now, we won't be able to merge this PR until we fix the issues in the reader that are causing the failures you mentioned. I am taking a look at those issues.

[darrelmiller]

@sergey-tihon Thanks for the PR. We are working on the stackoverflow issue and also on including external files. If it would be possible to identify the OpenAPI descriptions that generate Null Reference Exceptions and duplicate key exceptions and create issues for those, that would be really helpful.

As @PerthCharern mentioned we probably wouldn't want to incorporate testing the complete set of Apis.guru documents into the standard tests, but we may be able to have standalone tools that will run those tests.

[PerthCharern]

Some issue I saw so far:

• Null Ref exception in the Reference happens to the parameters that are of body or formData type. This is tracked in Handling body and form parameter reference for V2 #151.

• Duplicate key exception in OpenOpiOperationDeserializer.CreateFormBody. The formParameters temp variable were not cleared properly when multiple operations are processed. "formparameters" temp variable must be cleared after each operation is processed #191

• Handling of empty URI in security scheme deserializers. Handle empty URI when reading #192

[sergey-tihon]

Let's keep it open until this PR will become green and then decide what to do with it. I am absolutely fine if you decide not to merge it, also I could convert it to the collection of integration tests (that you can skip on CI) or create a tool.

The most annoying bug is StackOverflow for now, it crashes my test execution process (at least on macos) which greatly complicates the testing.

[darrelmiller]

Hey @sergey-tihon I have pushed a branch dm/smoketests2 that has a slightly modified version of your test. I started fixing some of the issue that the tests expose but still a fair few to resolve. Once I have ensured that these tests won't break our build I will create a PR to merge them. Thanks for the taking the initiative to pull these tests in.

[sergey-tihon]

@darrelmiller thanks for fix StackOverflow issue. Now at least we can see clean test results 994 passed vs 77 errored test schemas from apis.guru

[darrelmiller]

Replaced by PR #220