Flask-RESTX Models Re-Design

[SteadBytes]

For quite some time there have been significant issues around data models, request
parsing and response marshalling in flask-restx (carried over from
flask-restplus). The most obvious of which is the deprecation warning
about the reqparse module in the documentation that has been in place for far
too long. These changes have been put off for various reasons which I won't
discuss here, however now the new fork is steadily underway I (and no doubt others) would like
to start addressing this.

Since this digs quite deep into the architecture of flask-restx there will be
significant (and likely breaking) changes required. As such, this issue is to
serve as a discussion around the API we would like to provide and some initial
ideas of how to best proceed. This is not intended to be the starting point of
hacking something together which makes things worse!

I will set out my current thoughts on the topic, please contribute by adding
more points and expanding on mine with more discussion.

High Level Goals:

• Uniform API for request parsing and response marshalling
  • e.g. remove the separation between reqparse and models
• Generate correct and valid Swagger/OpenAPI Specifications
• Validation of input and output data should conform to the generated
  Swagger/OpenAPI Specifications
  • e.g. If the Swagger/OpenAPI spec considers a value valid, the model should too.
• Define models using JSON Schema
  • Supported already, but with numerous issues (@j5awry has been battling for some time)
• OpenAPI 3 support

General Issues/Discussion Points

• What should the API look like?
  • Continue with the api.marshal , api.doc decorator style?
  • How to define models?
    • Do we force direct usage of another library e.g. Marshmallow or wrap in
      some other API and use the library for the "under the hood" work?
• Model validation
  • External libraries e.g. Marshmallow
• Schema Generation
  • External libraries e.g. Marshmallow
• Backwards compatibility
  • Continue to support reqparse and existing models interface?
  • Swagger 2.0 vs OpenAPI 3.0
    • IMO generating both should be a goal if possible

Resources/Notable Libraries

• https://marshmallow.readthedocs.io/en/stable/
• https://github.com/fuhrysteve/marshmallow-jsonschema
• https://pydantic-docs.helpmanual.io/
• Faust Models, Serialization and Codecs https://faust.readthedocs.io/en/latest/userguide/models.html
  • Faust is not a Flask or even REST library but I have found it's Models to be
    a nice interface to use.
• https://github.com/apryor6/flask_accepts
• Swagger 2.0 https://github.com/OAI/OpenAPI-Specification/blob/master/versions/2.0.md
• OpenAPI 3.0 https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.2.md

[jessekv]

I'm curious about the "High Level Goals". With the exception of defining models using JSON Schema, fastapi https://fastapi.tiangolo.com was created to meet the high level goals. Is RESTX is aiming to become a synchronous version of fastapi?

[j5awry]

@vdwees There are many frameworks working toward those high level goals. marshmallow has a setup built on Flask already. there are similar frameworks out there on top of Django, Bottle, Hug, etc (actually, Hug might be compliant out of the box...it's been a little bit since I used it). Some more (ha) examples include marshmallow-code/flask-smorest, connexion, pyramid w/ pyramid_openapi3 plugin, etc.

So, no, i don't think we're a synchronous version of fastapi. If anything, we're closest to flask-smorest, especially if we decide to use marshmallow for generating models. That's part of what makes the Python community so great and frustrating all at once. See a problem? Make a framework or library, and put it out there! did someone else do it something similar? Probably! But that's ok, cause maybe yours does it a little bit differently!

On topic now, we should probably split up doing some research on different modeling libraries, coming up with good pros/cons of each, and then coming to a decision. Some things we should consider:

1. does it already generate openapi specs (by itself or with an additional library)
2. does it already generate jsonschema (by itself or with an additional library)
3. speed!
4. maintainability -- is it well maintained? Can we trust it'll be there in a year? five years?
5. open nature (can I say i'm partial to the Marshmallow folks cause someone came knocking? Yeah, I can)

We should also consider the "write it ourselves" approach, taking into account the time it'd take to do well, and if we can personally maintain it to the level of other libraries.

[SteadBytes]

Agreed with @j5awry on the fastapi point, these are two different projects and I'm fairly certain flask-restplus existed before fastapi so it anything it's the other way round 😉

I agree with your dissesction there @j5awry , I'm hoping to take a first step in looking at 5. and/or write it ourselves approach this weekend (I was planning to last week but got dragged into the failing CI issues instead 🤷‍♀️)

[ziirish]

Like I said already, I trust you folks on the swagger/openapi part. I believe you know about this subject way more than me.

Anyway, in my view, we will soon be facing a dilemma:

• on one hand, we want to move forward and stop relying on the soon to be deprecated reqparse
• on the other hand, a huge part of the community is still using restplus and we will need to remain "compatible" for some time

So I'd suggest we release v1.0.0 with the removal of python <= 3.5 support and a clear deprecation notice for the requparse usage, but we should keep maintaining a 1.x branch for let's say 1 year where we would ship bugfix and compatible improvements.

And in parallel, we can go ahead and release a v2.0.0 where we simply drop support for reqparse and we start working on an alternative.

[goranvrbaski]

My personal opinion is that we should use an external package like Marshmallow for model validation and schema generation. Community around Marshmallow is really good and you can see day to day activity on that project.

I flask-restx since it was forked from the original flask-restplus. You also have a really good community, but I think there are not enough people to cover creating a "new marshmallow", integration with existing Marshmallow should be enough for now.

I'm open for the discussion and I also have a little bit of free time so I can help you with the integration if you choose to go that way.

[SteadBytes]

I've been experimenting a little bit with what API we could provide using. This is a modified version on the todo_blueprint.py example aiming to explore a possible user-facing API for the models/validation with Marshmallow (not tied to Marshmallow, just where I've decided to start).

The idea is to keep the existing @api.marshal (for responses) and @api.expect (for requests) API but use Marshmallow Schema. Behind the scenes, the validation, marshalling and OpenAPI spec generation will take place.

Disclaimer: This is not meant to be a good example of implementing a REST API, just exploring the model definition/marshalling/request parsing API.

Any thoughts on the API? (not so much the underlying implementation)

from uuid import uuid4

from flask import Blueprint, Flask
from marshmallow import Schema, fields

from flask_restx import Api, Resource


api_v1 = Blueprint("api", __name__, url_prefix="/api/1")

api = Api(api_v1, version="1.0", title="Todo API", description="A simple TODO API")

ns = api.namespace("todos", description="TODO operations")


# Class-style schema


class Task(Schema):
    description = fields.String(required=True, description="The task details")


class Todo(Schema):
    id = fields.String(required=True, description="The todo ID")
    task = fields.nested(Task, required=True)
    completed: fields.Boolean(required=True)
    created_at = fields.DateTime(required=True, format="iso")
    completed_at = fields.DateTime(format="iso")


# Dictionary-style schema (unchanged from user API perspective)

Todo = api.model(
    "Todo",
    {
        "id": fields.String(required=True, description="The todo ID"),
        "task": fields.nested(Task, required=True),
        "complete": fields.Boolean(required=True),
        "created_at": fields.DateTime(required=True, format="iso"),
        "completed_at": fields.DateTime(format="iso",),
    },
)


# Dummy definitions for demonstration purposes


def db_get_todo_or_abort(todo_id: str) -> dict:
    """ Fetch Todo from DB by id, abort with `404` if not found. """
    pass


@ns.route("/<string:todo_id>")
@api.doc(responses={404: "Todo not found"}, params={"todo_id": "The Todo ID"})
class Todo(Resource):
    """Show a single todo item and lets you delete them"""

    @api.marshal_with(Todo)
    def get(self, todo_id):
        """Fetch a given resource"""
        # Dict returned here is passed through Marshmallow schema for validation
        # and marshalling to JSON in the response
        data = db_get_todo_or_abort(todo_id)
        return data

    @api.expect(
        Task, location="form",
    )
    # Could also define inline as a Dict
    # @api.expect(
    #     {"task": fields.String(...)}, location="form",
    # )
    @api.marshal_with(Todo)
    def put(self, todo_id, task):
        """Update the task of a Todo"""
        # args automatically validated and passed in as an instance of Task -
        # no parser.parse_args
        data = db_update_todo(todo_id, task)  # Use the Marshmallow schema directly
        # Todo Marshmallow schema returned is automatically marhsalled into JSON
        # in the response
        return Todo(**data)


@ns.route("/")
class TodoList(Resource):
    """Shows a list of all todos, and lets you POST to add new tasks"""

    # Define some query params for filtering
    @api.expect(
        {"completed": fields.Boolean(), "completed_at": fields.DateTime(format="iso")},
        location="query",
    )
    # Marshal a list of a given schema
    @api.marshal_list_with(Todo)
    def get(self, args):
        """List all todos"""
        # Imagine this uses the filters in args properly to filter the result set
        # from the DB...
        return db_get_all("todo", filters=args)

    # Validate args with NewTodo schema then dump to a dict before passing
    # into handler
    @api.expect(Task, location="json", as_dict=True)
    @api.marshal_with(Todo, code=201)
    def post(self, args):
        """Create a todo"""
        # Just an example, I don't want to start a flame war around which values
        # should be used for IDs in a DB!
        todo_id = uuid4()
        # Do some work with args
        task = sanitise_input(args["task"])
        data = db_create_todo(str(todo_id), task)
        return data, 201


if __name__ == "__main__":
    app = Flask(__name__)
    app.register_blueprint(api_v1)
    app.run(debug=True)

[SteadBytes]

Any comments on that initial API example? I've got some ideas about the implementation however it's going to be a significant amount of work to do properly so I don't want to get started unless we're generally happy with the user-facing API 😊

The core Namespace/Api/Swagger classes are tightly couple to the Model implementation at the moment so I imagine it's not going to be a simple process 😅 Current extensions to integrate Marshmallow (e.g. flask-accepts) generallt convert the Marshmallow schema to flask-restx models/reqparse objects. However this work will be largely replacing models/reqparse so that we can fix a whole host of issues! 😁

[ziirish]

I like the idea though I would need more details on this part:

        # args automatically validated and passed in as an instance of Task -
        # no parser.parse_args

Do you have an idea of how we can split this work to help you out?

I also have a concern about one specificity of restplus/restx: the Wildcard field.
It is documented here. @sloria do you know if there is an equivalence in marshmallow?

[sloria]

In marshmallow, the handling of unknown fields is configurable.
https://marshmallow.readthedocs.io/en/stable/quickstart.html#handling-unknown-fields

There isn't a built-in way to do the globbing feature of wildcard, but using the above setting you can accept unknown fields.

Also, there's fields.Dict for nested, unstructured data: https://marshmallow.readthedocs.io/en/stable/api_reference.html#marshmallow.fields.Dict, but I'm not sure that meets the same use case.

[ziirish]

Thanks for the quick reply.
My understanding is that unknown fields don't get parsed/validated.
The fields.Dict look interesting but like you said, I don't think it is what we need either.

I'll run some tests though.

[sloria]

You might also might be interested in Schema.from_dict, which can be useful for validating fields that are only known at runtime. I wrote a bit about it here. It could also be useful for the dictionary-style API in @SteadBytes 's comment.