Cannot take allOf a non-object

[ahuang11]

Describe the bug

Thanks for creating this; I love how explicit it is on errors and warnings!

I am currently encountering this warning:

WARNING parsing POST /2.1/jobs/create within default. Endpoint will not be generated.

Cannot take allOf a non-object

Raised from here. However, I don't fully understand this error, specifically what is a non-object.

Here is the contents that it warns on:

  /2.1/jobs/create:
  ...
      requestBody:
        required: true
        content:
          application/json:
            schema:
              allOf:
                - $ref: '#/components/schemas/JobSettings'
                - $ref: '#/components/schemas/AccessControlList'

I double checked to make sure these reference schemas exist:

    JobSettings:
      properties:
        name:
          type: string
          example: A multitask job
...

    AccessControlList:
      type: object
      properties:
        access_control_list:
          type: array
          description: List of permissions to set on the job.
          items:
            $ref: '#/components/schemas/AccessControlRequest'

Any thoughts and suggestions appreciated!

To Reproduce
Steps to reproduce the behavior:

1. wget https://docs.databricks.com/_static/api-refs/jobs-2.1-aws.yaml
2. Try running openapi-python-client generate --path jobs-2.1-aws.yaml which results in multiple errors similar to:

components -> schemas -> GitSource -> properties -> git_url -> required
  value is not a valid list (type=type_error.list)

3. Manually update the offenders' required from bool to list, e.g.

required: false

to

required:
  - false

4. Rerun openapi-python-client generate --path jobs-2.1-aws.yaml
5. Results in multiple warnings:

WARNING parsing POST /2.1/jobs/create within default. Endpoint will not be generated.

Cannot take allOf a non-object


WARNING parsing POST /2.1/jobs/run-now within default. Endpoint will not be generated.

Cannot take allOf a non-object

Expected behavior
A clear and concise description of what you expected to happen.

No warnings; succesful generation. I am happy to help contribute a PR if this is indeed a bug and if we can figure out the issue.

OpenAPI Spec File
A link to your openapi.json which produces this issue.
https://docs.databricks.com/_static/api-refs/jobs-2.1-aws.yaml (needs manual modification to the yaml to make it more compliant)

Desktop (please complete the following information):

• OS: Mac M1
• Python Version: 3.9.13
• openapi-python-client version 0.11.6

Additional context
Add any other context about the problem here.

[dbanty]

A schema is only considered an object if it has type: object. It seems like JobSettings does not have that set. Maybe we should assume type: object if properties is set? I'm not sure it makes sense to have properties if not an object 🤔.

[dbanty]

Side note, that required: list should be a list of the properties that are required. (e.g., required: ["name", "git_source"]) required should also be defined on the schema itself, not on each property therein.

I think there was an older version of the spec (maybe Swagger?) which had required: set on each property? Sounds like it was not updated properly. What you probably want is to process each model, loop through the properties, find out which ones have required: true, and move those into a required array up a level.

[ahuang11]

Thanks for the quick reply!

A schema is only considered an object if it has type: object. It seems like JobSettings does not have that set. Maybe we should assume type: object if properties is set? I'm not sure it makes sense to have properties if not an object 🤔.

That makes sense to me. Would you like me to create a PR for that? If so, can you point me to the relevant code section?

And also thanks for the pointer on required; I will try these out tomorrow and see if I can generate the output fully.

[dbanty]

Sure thing, I think adding an or to line 649 of openapi_python_client/parser/properties/__init__.py (the if statement looking for type == object should do it.

The easiest way to test will be via an end-to-end test, which should be described in CONTRIBUTING.md (let me know if it's not clear so I can fix it!)

[ahuang11]

Thanks for all your pointers; adding type: object to JobSettings fixed those warnings and I was able to generate the code without any warnings (after correcting below).

When I do the same to RunParameters, I get this:

Unable to parse schema /components/schemas/RunParameters

Unable to parse this part of your OpenAPI document: : type array must have items defined

Schema(title=None, multipleOf=None, maximum=None, exclusiveMaximum=None, minimum=None, exclusiveMinimum=None, maxLength=None, minLength=None, pattern=None, maxItems=None, minItems=None, uniqueItems=None, maxProperties=None, minProperties=None, required=None, enum=None, type=<DataType.ARRAY: 'array'>, allOf=[], oneOf=[], anyOf=[], schema_not=None, items=None, properties=None, additionalProperties=None, description='An array of commands to execute for jobs with the dbt task, for example `"dbt_commands": ["dbt deps", "dbt seed", "dbt run"]`', schema_format=None, default=None, nullable=False, discriminator=None, readOnly=None, writeOnly=None, xml=None, externalDocs=None, example=['dbt deps', 'dbt seed', 'dbt run'], deprecated=None)

So I updated:

dbt_commands:
  ...
  items:
    type: string

Wondering if we should also default to string if no type is defined for an array, and warn accordingly?

[ahuang11]

Also, when I tried adding an or statement:

    if data.type == oai.DataType.OBJECT or data.allOf or (data.type is None and data.properties):

I get a cascading error effect:

Updating jobs_api_2_1_client
Warning(s) encountered while generating. Client was generated, but some pieces may be missing

WARNING parsing POST /2.1/jobs/create within default. Endpoint will not be generated.

Reference #/components/schemas/JobSettings not found


WARNING parsing GET /2.1/jobs/list within default.

Cannot parse response for status code 200 (Could not find reference in parsed models or enums), response will be ommitted from generated client

Reference(ref='#/components/schemas/Job')

WARNING parsing GET /2.1/jobs/get within default.

Cannot parse response for status code 200 (Could not find reference in parsed models or enums), response will be ommitted from generated client

Reference(ref='#/components/schemas/JobSettings', description='Settings for this job and all of its runs. These settings can be updated using the [Reset](https://docs.databricks.com/dev-tools/api/latest/jobs.html#operation/JobsReset) or [Update](https://docs.databricks.com/dev-tools/api/latest/jobs.html#operation/JobsUpdate) endpoints.')

WARNING parsing POST /2.1/jobs/reset within default. Endpoint will not be generated.

Could not find reference in parsed models or enums

Reference(ref='#/components/schemas/JobSettings', description='The new settings of the job. These settings completely replace the old settings.\n\nChanges to the field `JobSettings.timeout_seconds` are applied to active runs. Changes to other fields are applied to future runs only.')

WARNING parsing POST /2.1/jobs/update within default. Endpoint will not be generated.

Could not find reference in parsed models or enums

Reference(ref='#/components/schemas/JobSettings', description='The new settings for the job. Any top-level fields specified in `new_settings` are completely replaced. Partially updating nested fields is not supported.\n\nChanges to the field `JobSettings.timeout_seconds` are applied to active runs. Changes to other fields are applied to future runs only.')

WARNING parsing GET /2.1/jobs/runs/list within default.

Cannot parse response for status code 200 (Could not find reference in parsed models or enums), response will be ommitted from generated client

Reference(ref='#/components/schemas/Run')

WARNING parsing GET /2.1/jobs/runs/get within default.

Cannot parse response for status code 200 (Reference #/components/schemas/Run not found), response will be ommitted from generated client


WARNING parsing GET /2.1/jobs/runs/get-output within default.

Cannot parse response for status code 200 (Could not find reference in parsed models or enums), response will be ommitted from generated client

Reference(ref='#/components/schemas/Run', description='All details of the run except for its output.')

Unable to parse schema /components/schemas/Job

Unable to parse this part of your OpenAPI document: : Could not find reference in parsed models or enums

Reference(ref='#/components/schemas/JobSettings', description='Settings for this job and all of its runs. These settings can be updated using the `resetJob` method.')

Unable to parse schema /components/schemas/JobSettings

invalid data in items of array named "tasks": Could not find reference in parsed models or enums

Reference(ref='#/components/schemas/JobTaskSettings')

Unable to parse schema /components/schemas/JobTaskSettings

Unable to parse this part of your OpenAPI document: : Could not find reference in parsed models or enums

Reference(ref='#/components/schemas/DbtTask', description='If dbt_task, indicates that this must execute a dbt task. It requires both Databricks SQL and the ability to use a Serverless SQL warehouse.')

Unable to parse schema /components/schemas/RunTask

Unable to parse this part of your OpenAPI document: : Could not find reference in parsed models or enums

Reference(ref='#/components/schemas/DbtTask', description='If dbt_task, indicates that this must execute a dbt task. It requires both Databricks SQL and the ability to use a Serverless SQL warehouse.')

Unable to parse schema /components/schemas/Run

invalid data in items of array named "tasks": Could not find reference in parsed models or enums

Reference(ref='#/components/schemas/RunTask')

Unable to parse schema /components/schemas/DbtTask

Unable to parse this part of your OpenAPI document: : type array must have items defined

Schema(title=None, multipleOf=None, maximum=None, exclusiveMaximum=None, minimum=None, exclusiveMinimum=None, maxLength=None, minLength=None, pattern=None, maxItems=None, minItems=None, uniqueItems=None, maxProperties=None, minProperties=None, required=None, enum=None, type=<DataType.ARRAY: 'array'>, allOf=[], oneOf=[], anyOf=[], schema_not=None, items=None, properties=None, additionalProperties=None, description='A list of dbt commands to execute. All commands must start with `dbt`. This parameter must not be empty. A maximum of up to 10 commands can be provided.', schema_format=None, default=None, nullable=False, discriminator=None, readOnly=None, writeOnly=None, xml=None, externalDocs=None, example=['dbt deps', 'dbt seed', 'dbt run --models 123'], deprecated=None)

[ahuang11]

Interestingly, all I had to do was fix one, and then the cascading warnings all disappeared.

Specifically:

Unable to parse schema /components/schemas/DbtTask

Unable to parse this part of your OpenAPI document: : type array must have items defined

I added:

          items:
            type: string

Should array default to type string if it's not specified?