Excluding Response Fields and Huge Performance Hits

[rdegges]

Hey dudes,

Love Flask-Restless, but wanted to leave an issue here as I'd love to use the library, but a showstopper in my use case (and possibly that of other people) is that Flask-Restless really has no way (currently) to properly exclude database relationship queries with the pre and postprocessor stuff.

For instance, I have an application which has several models, namely:

State and Person. Each person has a state relationship (via ForeignKey).

My API clients need to frequently grab a list of all states in the system, so they'll do something like:

GET /api/states

And will immediately get their screen flooded with results (because Flask-Restless is traversing the backref and generating JSON for the millions of people in each state).

Even if I were to explicitly write a postprocessor for the GET_MANY hook, this wouldn't really solve the problem as each request to my API endpoint /api/states would still result in Flask-Restless spending an enormous amount of time loading my people backref and eventually discarding that data and returning a response.

I think a good solution to this problem is what issue #161 is suggesting: have Flask-Restless only load foreign key relationships when they're explicitly eagerly loaded by Flask-SQLAlchemy.

I'm documenting this here for discussion purposes. Thank you.

[gpopovic]

I agree with this, maybe the better solution would be to somehow be able to decide on how deep (how many levels) should flask-restless traverse

[rdegges]

I think another good idea would be to just handle URIs instead of navigating ForeignKeys.

For instance, instead of doing a ForeignKey lookup and serializing all results, restless could simply return a URI instead: "people": "/people/+18002223333". This way your client can then make an additional HTTP request to the desired resource URIs and get that data explicitly if they want it.

[jfinkels]

Hmm, this is a difficult problem to solve. I don't know how to provide a simple way for users of this library to specify how relationships are loaded. The problem with loading only eagerly loaded relationships is that it is unexpected: if a client makes a GET request to /api/person/1, I believe it is reasonable for the client to expect that it will get a list of all the computers related to that person.

Here are some possible solutions:

1. Add back in the exclude_columns keyword arguments which I so foolishly removed.
2. Let the user specify a custom serialization function (replacing the default views._to_dict function), as suggested by @klinkin in Add support custom serialization function #167.
3. Let the user specify the (global) relationship depth for serialization (and have a reasonable default [a depth of one, say]), as suggested by @goranek.
4. Replace the actual data for each related instance with a link to the related instance, as suggested by @rdegges.

Solution 1 is simple, but inflexible, and creates somewhat complex code. Solution 2 would force the user to fix the problem and so would definitely solve the problem in that sense, but it may be complicated for the user; one of the goals of this library should be to take serialization out of the hands of the user. Solution 3 is simple, but again may be too inflexible. Solution 4 is a bit awkward: in the response's JSON object would include a mixture of some real information ("name": "Jeff") and some metainformation ("computers": [{"url": "/api/computers/1"}, {"url": "/api/computers/2"}]).

I think I am currently most interested in solution 3. I might try it out to see how well it works. Any other comments or thoughts?

[rdegges]

Hey, so I think a combination of 1 + 4 would be ideal.

Here's my reasoning:

I think that having include_columns and exclude_columns options are really awesome, as they allow you to do do a lot of work really simply (for the user). For instance, not many APIs will want to expose every field, and defining your own postprocessor hooks is a lot more tedious than simply specifying which fields you'd like to use (or exclude).

I think that my suggestion (4) is really important, however, as it makes things a lot simpler (globally) for all Flask-Restless users. Since REST is all about resources and URIs, I think that having foreign key relationships loaded via a separate API URI is the right thing to do, especially since that makes it such that:

• Users hitting the new URI will have pagination / etc. per their settings.
• Users will instantly know that all relationships need to be clearly defined as APIs, forcing them to think about how many results they want to return, how it'll work, etc. I think this encourages 'good practices' for users by forcing them to think about what data they may be (accidentally) exposing with an API that eagerly loads related data.

I'm certainly open to all the other suggestions, and this is a really great topic. Thank you for spending some time reviewing this! <3

[jfinkels]

That is a fairly convincing argument. I'll start by replacing exclude_columns and include_columns for now...

[bjourne]

+1 Specifying which columns to load also solves the problem with flask-restless needlessy loading large deferred columns which is another performance drawback.

[klinkin]

Recently i discovered an interesting project https://github.com/danielholmstrom/dictalchemy.
Maybe we should use it with flask-restless?
Then Solution 2 is not going to be very difficult for users!

[michaelfillier]

Linking to related resources seems to be the widely used approach to this type of issue. If you look at other APIs, like Instagram, for example, the User endpoint does not include a list of the user's media, rather a link to the resource.

Generally, if the related object list is more than 5, I would think that it is a prime candidate for exclusion.

[jfinkels]

@michaelfillier The Instagram example you gave is not convincing. Their sample response for GET /users/user-id/media/recent shows that much of the related information is present. Also, the URLs you see there are links to the plain-old web view of the photo, not the API URL for that resource.

However, if you can find other convincing examples, then I may be swayed into that way of thinking.

[michaelfillier]

Sorry, I wasn't clear and Instagram probably wasn't the best example and I was wrong about the href to the media endpoint.

Point being, when you hit the user endpoint you don't get the media for that user, in the same sense that the OP didn't want to include the computers when hitting the people endpoint.

Actually, Instagram's API was a really bad example, since a lot of the design decisions don't follow basic HATEOS principals, even with my little exposure to these principals I can see a lot of semantic gaps in their api.