Support conditional link serialization format

[csantero]

This would be a really killer feature for my project. I have some ember routes that fetch a decently complex graph of data, and there's really no good way to inform the server that this particular route needs to side-load deeply, while maintaining the ability to return sparse results at other times (say, when loading a list).

For a simple example. I have a resource called Organization, which has many Notes. Notes have a User that entered the note. When I load an organization detail record, I want to load all the notes for that organization, as well the user that entered each note, preferably all in a single AJAX request. However if I am on a route where I need to fetch a list of Organizations, I only want to serialize an href to fetch the related records. As far as I can tell, this kind of conditional link formatting isn't yet possible. The spec has a solution for this, by allowing the client to specify resource paths to include.

I'm happy to put in the legwork toward implementing this feature, but I want to make sure I get the architecture right. The first step is to add a parameter to the Get methods on ApiController to gain access to the include query param. After that I'm not sure, I guess we have to change the IMaterializer interface somehow to allow the ApiController to tell the provider what to include. Then here comes the hard part: How do we tell the formatter to override its own strategy for resource linking behavior, and instead use what the client sent? Remember, MediaTypeFormatter does not have access to the request context.

Two potential solutions occur to me:

1. Make the formatter not be in charge of deciding what relationships to serialize at all. We add another action filter that runs after our other filters (filter, sort, enumerate). This filter would consult the IModelManager, and then modify the object content to return an ExpandoObject, and all the formatter has to do is serialize that to JSON.
2. Still add another action filter, but instead of using the model manager, have it wrap the content in some sort of object that contains hints for the formatter to tell it what relationships to serialize. The formatter would be changed to handle this new object if it sees it, and fall back to the existing behavior if it is passed something else.
   add gitignore file #2 is less of a drastic change and I think I prefer it, just wanted to get your thoughts. Am I missing something obvious?

[SphtKr]

Thanks for submitting this, this has definitely been on my mental roadmap, I just hadn't submitted an issue yet. I've wanted the whole time to allow for runtime inclusion of related objects. (The primordial version of this library had this feature, but I had to take it out--that version always returned a Payload object from every ApiController, and you could include whatever you wanted there. That had to go, however, for various reasons, some probably obvious.) I had in mind primarily letting the user programmatically specify what to include, but either way that would make it easy to add support for the include parameter of the spec.

I do have some thoughts on this but they're quite different from your proposals. I've thought so far that the way to do this would be by leveraging the RelationAggregator and the MetadataManager, with logic within the user's overriden ApiController methods having control over what's included and what's not.

I think a certain degree of onus for the decision making about what to include should be placed on the user in their ApiController code. First because the application developer should be in a good position to intelligently figure out what related objects are going to be needed (at least for the 1st-party client's use case), as your example demonstrates. But second, because allowing the include parameter to operate unchecked very quickly allows for pathological behavior, even to the point of being a significant DoS concern. For example, I have relationships on some of my models that are computed at runtime, and the computation is expensive--I serialize these as a link, but if someone specified include on that and retrieved all the records, it wouldn't only suck up a lot of bandwidth it would bring my DB server to its knees. This is a concern for me with sorting and filtering parameters too, by the way. So handing that job off to an ActionFilter seems like it would be difficult to control easily from that perspective.

My somewhat undeveloped thoughts so far are something like this:

• Give the user a method to explicitly pass arbitrary (already materialized) objects to RelationAggregator for inclusion in the return result.
• Add a method to MetadataManager where the user can find out what include directives the client specified, if any. Based on my work so far on ModelManager, it may make sense for this method to return a list of PropertyInfos or something like it (even Linq expressions?) instead of the raw query string values--this would be one fewer places where the format details would have to leak to the .NET layer.
• Maybe provide a convenience mechanism (somewhere) to help retrieve the objects for a given include parameter or set of parameters.
• Let the user pick and choose which include parameters to honor, either by letting them filter the list before passing it on or explicitly using the convenience mechanism to resolve each allowed include parameter.

It's worth noting that at the moment IMaterializer isn't intended to do much in Get methods and is mainly intended for Put and Post...unless I'm forgetting something. Generally in your Get methods you're probably using your persistence layer more directly, and in JSONAPI.EntityFramework.Http.ApiController it pretty much just carries around the DbContext in the Get code paths.

In whatever case, yes, absolutely we need to do this. I think it would be beneficial to come back to this after we've cleared some of the other issues which are impacting the API, after the dust settles it may be more clear what way we should go. But that's been my thinking on this so far.

[SphtKr]

Oh, I didn't address one part: You specifically want to have the ability to serialize the relationship property as a link, unless the related objects are decided to be included at runtime, correct? Or, more generally at least, to be able to have a way at runtime to change between serializing as a link and not--this would require the actual property containing an array of the ids in addition to the related objects being included in linked, correct?

Yeah, that's tricky and I'm not sure exactly how to pass that up either. Possibly again by using MetadataManager, and providing a method for the user to (within their ApiController) specify an instance of SerializeAsAttribute(...) for a given property path (again, as a string? PropertyInfo? Linq expression? unsure). MetadataManager was created with the idea of providing an interface to information going in both directions (hence its somewhat general and possibly poor name), including spec-provided querystring parameters, presence or absence of JSON keys in the request (it's only currently implemented function), and incoming and outgoing values in the meta key.

I use parentheses way to much (I think).

[SphtKr]

I should clarify/ask a question about why I keep mentioning using Linq expressions, it may be misguided:

Is there something like Apple's KVC in C#/.NET? That is, is there a built-in mechanism that would let me specify the string "Post.Author.Name" and give me the value at that property path? To my knowledge there is not. That's a pain because the JSON API spec uses that kind of thing quite a bit (not surprising given its roots) but we don't have a good way to implement that. I.e. how do we easily pass around the fact that the include or filter path Post.Author was provided? Building a Linq expression is the only thing I can think of that comes close.

[csantero]

Is there something like Apple's KVC in C#/.NET? That is, is there a built-in mechanism that would let me specify the string "Post.Author.Name" and give me the value at that property path?

Yeah I would build a method that does this using LINQ. It's not too difficult, I can easily whip that up when we need it. We can either let the user pass around the string or a LINQ expression, a la Expression<Func<Post, object>> expr = p => p.Author.Name. The LINQ expression can then be decomposed into a path string when we need to consume it, at which point I would build up an expression to access the value at the given path.

[csantero]

Ok, I never knew about MetadataManager before. Of course you know what I think about the singleton pattern by now =).

I think actually we're not in too different mental places. We both want something like the MetadataManager, but you currently have it as a singleton, and I would set it up in an action filter.

[SphtKr]

Right, I thought about if MetadataManager should be refactored when I started working on ModelManager. There's at least one big difference, MetadataManager has to be easily accessible at runtime to user code in an ApiController. ModelManager is more a set-it-and-forget-it thing that happens when the JsonApiFormatter is instantiated. So there has to be an entry point, and we can't easily pass it in as a parameter to the ApiController methods without changing its interface from what is upstream. I imagine it would be possible to make The MetadataManager a readonly property on the formatter, but I think it's fairly clumsy to access the formatter from within an ApiController verb method. I'm not glued to it, but at the moment I think it's a good design. Where would you move it?

[csantero]

Oh, I didn't address one part: You specifically want to have the ability to serialize the relationship property as a link, unless the related objects are decided to be included at runtime, correct? Or, more generally at least, to be able to have a way at runtime to change between serializing as a link and not--this would require the actual property containing an array of the ids in addition to the related objects being included in linked, correct?

Yes, I want a default conservative no-expansion that merely sends down a link, but the option to override that and expand the relationship, either at the api controller level or due to a prompt from the user.

My main pain point with the library at the moment is that I want to minimize the number of AJAX requests when visiting any particular ember route, but I have no way to accomplish that. Depending on what part of my application the user is in, deciding what relationships to expand in the initial GET can drastically affect the number of requests. Static attribute assignments just aren't good enough.

I do understand your arguments about wanting to safeguard against abuse at the server level. I am not opposed to an api controller solution, I just figured that since the spec mentions this topic and offers a client-based approach, we should take its lead. But there's really no reason not to do both. We could start with controller-level configuration, and that would give us the infrastructure to modify it with query params later.

[SphtKr]

Been thinking about this and I think I have a workable idea, pretty much what I described. If we refactor MetadataManager for some reason it'll probably just change the entry point, whatever object still needs to pass around the same information.

What I've got in mind will also let us override any attribute-defined default on a property--e.g. you could switch JsonIgnore on or off if you wanted to.

I need to finish that ModelManager stuff first though...I'll need some of the methods.

[csantero]

Ok, cool. I was trying to come up with a solution that lets us escape the MediaTypeFormatter ghetto... that thing has no access to any sort of per-request context. I agree, let's get ModelManager sorted out first.

[SphtKr]

Believe me I know what you mean. I probably shouldn't even admit this, but my first attempt at this library did everything, including the IMaterializer stuff and entity framework integration inside the mediaformatter. It sounded cool at the time because all you had to do was use the formatter and you're done; but I found out how dumb it was to try and wrap everything in there.