Hawaii throws: Unhandled exception. Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: D. Path '', line 0, position 0.

[abelbraaksma]

Hi @Zaid-Ajaj, I was trying out this awesome project to create a client for Stripe (using this: https://github.com/stripe/openapi/blob/master/openapi/spec3.yaml), which has an OpenApi spec in JSON and YAML. However, either way I try it, hawaii crashes with the following exception:

Unhandled exception. Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: D. Path '', line 0, position 0.
   at Newtonsoft.Json.JsonTextReader.ParseValue()
   at Newtonsoft.Json.Linq.JObject.Load(JsonReader reader, JsonLoadSettings settings)
   at Newtonsoft.Json.Linq.JObject.Parse(String json, JsonLoadSettings settings)
   at Newtonsoft.Json.Linq.JObject.Parse(String json)
   at Program.getSchema(String schema, FSharpOption`1 overrideSchema) in /Users/zaid/projects/Hawaii/src/Program.fs:line 290
   at Program.runConfig(String filePath) in /Users/zaid/projects/Hawaii/src/Program.fs:line 2767
   at Program.main(String[] argv) in /Users/zaid/projects/Hawaii/src/Program.fs:line 2920

This exception appears to incorrectly render the path/file name that causes the error, and "line 0, pos 0" suggests something odd is going on.

Tbh, I'm fully aware that this is a massive API from Stripe, so I wouldn't entirely expect it to go as smoothly as any smaller API. Though I also tried it with some smaller APIs and still got the same error, so maybe something else is the matter. To that end, this is the config I was using:

{
  "schema": "testapi.json",        // tried with stripe's spec3.json and spec3.yaml as well
  "project": "StripeAutogen",
  "output": "./output",
  "target": "fsharp",
  "synchronous": true,
  "asyncReturnType": "async",
  "resolveReferences": false,
  "emptyDefinitions": "ignore"
  //[ "overrideSchema" ],<JSON schema subset>,
  //[ "filterTags" ]
}

EDIT: I get exactly the same error when I use "schema": "thisfiledoesnotexist.yml", in other words, this appears to happen prior to opening the file, as with a non-existing file I receive the same error...

Version of Hawaii:

hawaii --version
0.59.0

[Zaid-Ajaj]

HI @abelbraaksma thanks for filing the issue, not being able to parse the JSON suggests that the JSON is malformed which you can get if you are not referencing the RAW file contents from github when pointing the schema configuration at it

I will have a look when I can, right now only with access to my phone

[Zaid-Ajaj]

@abelbraaksma This is what I meant, the following works for me using latest Hawaii v0.60.0

{
    "schema": "https://raw.githubusercontent.com/stripe/openapi/master/openapi/spec3.json",
    "project": "StripeAutogen",
    "output": "./output",
    "target": "fsharp",
    "synchronous": true,
    "asyncReturnType": "async",
    "resolveReferences": false,
    "emptyDefinitions": "ignore"
}

The project is generated successfully but unfortunately it doesn't compile. Stripe has a really complex API and uses $all and $anyOf I believe which are yet supported in Hawaii

This is definitely another issue though I want to get Hawaii to the point where everything it generates, at least compiles and not generate things that are not supported yet

[abelbraaksma]

@Zaid-Ajaj, thanks for getting back to me so quickly! I tried to read the YAML file (looks like I never correctly tried the JSON file) and I was meanwhile debugging Hawaii to see what was going on. Going over this code, it became clear:

let getSchema(schema: string) (overrideSchema: JToken option) =
    let schemaContents =
        if File.Exists schema && schema.EndsWith ".json" then
            let content = File.ReadAllText schema
            JObject.Parse(content)
        elif File.Exists schema && schema.EndsWith ".xml" then
            Console.WriteLine "Detected local OData schema"
            let openApiJson = readLocalODataSchema schema
            JObject.Parse openApiJson
        elif schema.StartsWith "http" && schema.EndsWith "$metadata" then
            Console.WriteLine "Detected external OData schema"
            let openApiJson = readExternalODataSchema schema
            JObject.Parse openApiJson
        elif schema.StartsWith "http" then
            let content =
                client.GetStringAsync(schema)
                |> Async.AwaitTask
                |> Async.RunSynchronously
            JObject.Parse(content)
        else
            // assume the schema is coming in as a string
            // convert it into a memory stream
            // this is useful for unit tests
            JObject.Parse(schema)

From this, I found that:

• YAML is not supported (yet), I mistakenly assumed that from the readme.md here
• JSON is only supported if it has the proper filename. If it happens to be *.JSON or just *.txt or something, it won't be read
• If it's a file and the extension isn't *.json or *.xml, the file won't be opened. Instead, the path will be interpreted as a string, which is parsed in the final else. That explains the "Unexpected character encountered while parsing value: D" (the D being the drive letter, apparently the path is resolved prior to getting to the function getSchema)

So, I tried with the following to circumvent the file-extension restriction:

elif File.Exists schema then
    // another file that is not `.json` or `.xml`, assume yaml or json with a different extension
    let content = File.ReadAllText schema
    JObject.Parse(content)

which allowed it to parse the local file correctly. Now I have the same results as you. Your config worked because you referenced the online file. Mine didn't work because I started out playing with YAML and not with JSON (and made some wrong assumptions along the way).

I may issue a PR with a few changes in the function so that the user feedback is a little better in case it falls through to the else branch.

[abelbraaksma]

I also noted that it can't process everything of Stripe. It's not just the anyOf etc, but also that Stripe has specific extensions to OpenAPI (starting with x-, like x-exandable-fields below) that aren't recognized. Furthermore it looks like it doesn't do well with style: deepObject. For instance, one of the first errors in the generated file Client.fs is caused by this snippet in Stripe:

# apologies this is yaml, but it is slightly more readable than JSON, 
# which I ended up using for generating the code
    get:
      description: "<p>Returns a list of people associated with the account’s legal
        entity. The people are returned sorted by creation date, with the most recent
        people appearing first.</p>"
      operationId: GetAccountPeople
      parameters:
      - description: A cursor for use in pagination. `ending_before` is an object
          ID that defines your place in the list. For instance, if you make a list
          request and receive 100 objects, starting with `obj_bar`, your subsequent
          call can include `ending_before=obj_bar` in order to fetch the previous
          page of the list.
        in: query
        name: ending_before
        required: false
        schema:
          maxLength: 5000
          type: string
        style: form
      - description: Specifies which fields in the response should be expanded.
        explode: true
        in: query
        name: expand
        required: false
        schema:
          items:
            maxLength: 5000
            type: string
          type: array
        style: deepObject
      - description: A limit on the number of objects to be returned. Limit can range
          between 1 and 100, and the default is 10.
        in: query
        name: limit
        required: false
        schema:
          type: integer
        style: form
      - description: Filters on the list of people returned based on the person's
          relationship to the account's company.
        explode: true
        in: query
        name: relationship
        required: false
        schema:
          properties:
            director:
              type: boolean
            executive:
              type: boolean
            owner:
              type: boolean
            representative:
              type: boolean
          title: all_people_relationship_specs
          type: object
        style: deepObject
      - description: A cursor for use in pagination. `starting_after` is an object
          ID that defines your place in the list. For instance, if you make a list
          request and receive 100 objects, ending with `obj_foo`, your subsequent
          call can include `starting_after=obj_foo` in order to fetch the next page
          of the list.
        in: query
        name: starting_after
        required: false
        schema:
          maxLength: 5000
          type: string
        style: form
      requestBody:
        content:
          application/x-www-form-urlencoded:
            encoding: {}
            schema:
              additionalProperties: false
              properties: {}
              type: object
        required: false
      responses:
        '200':
          content:
            application/json:
              schema:
                description: ''
                properties:
                  data:
                    items:
                      "$ref": "#/components/schemas/person"
                    type: array
                  has_more:
                    description: True if this list has another page of items after
                      this one that can be fetched.
                    type: boolean
                  object:
                    description: String representing the object's type. Objects of
                      the same type share the same value. Always has the value `list`.
                    enum:
                    - list
                    type: string
                  url:
                    description: The URL where this list can be accessed.
                    maxLength: 5000
                    type: string
                required:
                - data
                - has_more
                - object
                - url
                type: object
                x-expandableFields:
                - data
          description: Successful response.
        default:
          content:
            application/json:
              schema:
                "$ref": "#/components/schemas/error"
          description: Error response.

The generated function is as follows, note the .Value.director etc on the JObject instances, which cannot possibly work (and the escaped HTML <p> in the comment, but that's so minor I couldn't be bothered ;) ):

    ///<summary>
    ///&amp;lt;p&amp;gt;Returns a list of people associated with the account’s legal entity. The people are returned sorted by creation date, with the most recent people appearing first.&amp;lt;/p&amp;gt;
    ///</summary>
    ///<param name="endingBefore">A cursor for use in pagination. `ending_before` is an object ID that defines your place in the list. For instance, if you make a list request and receive 100 objects, starting with `obj_bar`, your subsequent call can include `ending_before=obj_bar` in order to fetch the previous page of the list.</param>
    ///<param name="expand">Specifies which fields in the response should be expanded.</param>
    ///<param name="limit">A limit on the number of objects to be returned. Limit can range between 1 and 100, and the default is 10.</param>
    ///<param name="relationship">Filters on the list of people returned based on the person's relationship to the account's company.</param>
    ///<param name="startingAfter">A cursor for use in pagination. `starting_after` is an object ID that defines your place in the list. For instance, if you make a list request and receive 100 objects, ending with `obj_foo`, your subsequent call can include `starting_after=obj_foo` in order to fetch the next page of the list.</param>
    member this.GetAccountPeople
        (
            ?endingBefore: string,
            ?expand: list<string>,
            ?limit: int,
            ?relationship: Newtonsoft.Json.Linq.JObject,   // a default JObject is generated here, but wrecks the calls later on
            ?startingAfter: string
        ) =
        async {
            let requestParts =
                [ if endingBefore.IsSome then
                      RequestPart.query ("ending_before", endingBefore.Value)
                  if expand.IsSome then
                      RequestPart.query ("expand", expand.Value)
                  if limit.IsSome then
                      RequestPart.query ("limit", limit.Value)
                  if relationship.IsSome then
                      RequestPart.query ("relationship[director]", relationship.Value.director)  // this is problematic
                      RequestPart.query ("relationship[executive]", relationship.Value.executive)// and this
                      RequestPart.query ("relationship[owner]", relationship.Value.owner)        // and this
                      RequestPart.query ("relationship[representative]", relationship.Value.representative) // and this
                  if startingAfter.IsSome then
                      RequestPart.query ("starting_after", startingAfter.Value) ]

            let! (status, content) = OpenApiHttp.getAsync httpClient "/v1/account/people" requestParts

            if status = HttpStatusCode.OK then
                return GetAccountPeople.OK(Serializer.deserialize content)
            else
                return GetAccountPeople.DefaultResponse(Serializer.deserialize content)
        }

[Zaid-Ajaj]

Hi @abelbraaksma, thanks again for digging into this 🙏 I understand the problem here it seems that Hawaii is not adequate enough to account for Stripe API at the moment. Requiring an object to be spread inside the query string parameters is always tricky, especially because infinite nesting is possible and we don't generate a specific type for the input (I think we should generate an object here like GetAccountPeopleRelationshipInput and use it in the function definition

    ///<summary>
    ///&amp;lt;p&amp;gt;Returns a list of people associated with the account’s legal entity. The people are returned sorted by creation date, with the most recent people appearing first.&amp;lt;/p&amp;gt;
    ///</summary>
    ///<param name="endingBefore">A cursor for use in pagination. `ending_before` is an object ID that defines your place in the list. For instance, if you make a list request and receive 100 objects, starting with `obj_bar`, your subsequent call can include `ending_before=obj_bar` in order to fetch the previous page of the list.</param>
    ///<param name="expand">Specifies which fields in the response should be expanded.</param>
    ///<param name="limit">A limit on the number of objects to be returned. Limit can range between 1 and 100, and the default is 10.</param>
    ///<param name="relationship">Filters on the list of people returned based on the person's relationship to the account's company.</param>
    ///<param name="startingAfter">A cursor for use in pagination. `starting_after` is an object ID that defines your place in the list. For instance, if you make a list request and receive 100 objects, ending with `obj_foo`, your subsequent call can include `starting_after=obj_foo` in order to fetch the next page of the list.</param>
    member this.GetAccountPeople
        (
            ?endingBefore: string,
            ?expand: list<string>,
            ?limit: int,
            ?relationship: GetAccountPeopleRelationshipInput, 
            ?startingAfter: string
        ) =
        async {
            let requestParts =
                [ if endingBefore.IsSome then
                      RequestPart.query ("ending_before", endingBefore.Value)
                  if expand.IsSome then
                      RequestPart.query ("expand", expand.Value)
                  if limit.IsSome then
                      RequestPart.query ("limit", limit.Value)
                  if relationship.IsSome then
                      RequestPart.query ("relationship[director]", relationship.Value.director)  // this is problematic
                      RequestPart.query ("relationship[executive]", relationship.Value.executive)// and this
                      RequestPart.query ("relationship[owner]", relationship.Value.owner)        // and this
                      RequestPart.query ("relationship[representative]", relationship.Value.representative) // and this
                  if startingAfter.IsSome then
                      RequestPart.query ("starting_after", startingAfter.Value) ]

            let! (status, content) = OpenApiHttp.getAsync httpClient "/v1/account/people" requestParts

            if status = HttpStatusCode.OK then
                return GetAccountPeople.OK(Serializer.deserialize content)
            else
                return GetAccountPeople.DefaultResponse(Serializer.deserialize content)
        }

In any case, a simpler solution is to change the indexers from say

 RequestPart.query ("relationship[director]", relationship.Value.director)

to

RequestPart.query ("relationship[director]", relationship.Value.["director"].ToObject<string>())

At the moment, I am very low on time to pick up this issue but always happy to accept PRs and review/merge them 😄

[Eliemer]

wait, does it or does it not support .yaml files? Im getting this error with a very simple API doc

I got this error:
JsonReaderException: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
Newtonsoft.Json.JsonTextReader.ParseValue()
Newtonsoft.Json.JsonTextReader.Read()
Newtonsoft.Json.Linq.JObject.Load(JsonReader reader, JsonLoadSettings settings)
Newtonsoft.Json.Linq.JObject.Parse(string json, JsonLoadSettings settings)
Newtonsoft.Json.Linq.JObject.Parse(string json)
AccountGoWeb.Controllers.AccountController.SignIn(LoginViewModel model, string returnUrl) in AccountController.cs
+
Newtonsoft.Json.Linq.JObject resultSignIn = Newtonsoft.Json.Linq.JObject.Parse(responseSignIn.Content.ReadAsStringAsync().Result);
Microsoft.