[RFC]Thoughts on an explicit way to define nested relationships?

[NullVoxPopuli]

I've been working on sideloading (#1107) for the json adapter, and needed recursive relationship building of the json object.

The problem with doing a strictly recursive approach (independent of sideloading or not), is that if you have enough references on your objects to other objects, a single request (of any kind) is basically going to return the whole database.
Example: Post belongs_to an Author which has_many Posts, and each Post has_many Comments, and each Comment belongs_to an Author.... it and would just keep going. The code in #1107 at least doesn't allow for things to be serialized multiple times (like if a post has the same author, that author only gets serialized once).

So, a potential solution that I'm working on prototyping right now is something like this:

PostSerializer < #...
  has_many :comments, include: :author
end

So, instead of recursively serializing the comments, we'd serialize them like we do today, but because we have the author key in the include hash, we'd also serialize each of the coments' authors.

BlogSerializer < #...
  has_many :posts, include: { author: [:bio], comments: [:author]] }
end

resulting json may look like:

{
  blog: {
    posts: [
      {
        author: {
          bio: {...}
        },
        comments: [
          {  author: {...} }
        ]
      }
    ]
  }
}

or

AuthorSerializer < #...
  has_many :blogs, include: { posts: [:author, comments: :author] }
end

resulting json may look like:

{
  author: {
    blogs: [
      {
        posts: [
          {
            author: {..}
            comments: [
              {
                author: {..}
              }
            ]
          }
        ]
      }
    ]
  }
}

this gets in to a problem of potentially having redundant data in the resulting json, so caching would be super important here (which I think is implemented by default? or does a cache key need to be specified in the controller? if not, should there be a default? - or would this kind of cacheing be separate, and more like fancy memoization?)

Note, I found that there was a previous attempt at solving this problem here: #952, but I think it's not clear in #952 how deep the nested relationships go / it could imply that the same recursive problem exists.

[iMacTia]

Hi @NullVoxPopuli, just to answer your last question (I'm the author of #952) the nesting can actually bring the recursive problem, but that's intended. It is actually a very similar feature to your side-loading, as you're actually asking to AMS to side-load all associations without explicitly declaring them. That's allow you to keep your code easier to maintain (if you add a new relationship later, that will be taken automatically).
The only situation a recursive loop could happen is when you have 2 serialisers like this:

TeamSerializer < #...
  has_many :players, include_nested_associations: true
end

PlayerSerializer < #...
  has_many :teams, include_nested_associations: true
end

This can actually fire the recursive problem, actually all many-to-many associations share this risk. But again, that's intended, because it depends on how you use the include_nested_associations option.
Moreover, remember that you can always define multiple serialisers for the same model to fit your needs.
Now, addressing your problem, that how I would solve it using my feature:

BlogSerializer < #...
  has_many :posts, include: include_nested_associations: true
end

PostSerializer < #...
  has_many :comments, include_nested_associations: true
  belongs_to :author, include_nested_associations: true
  # In this case, I want to have more info about the author because I want to display them after the blog post
end

AuthorSerializer < #...
  has_many :blogs, include_nested_associations: true
  has_one :bio
end

CommentSerializer < #...
  belongs_to :author
  # In this case, instead, I don't want anything else apart from basic informations (username, email, etc...) so no need for nested_associations
end

As I said before, you can always make even better defining different serialisers based on your needs.
So in this case, could be a good idea to have a BlogAuthorSerializer and a CommentAuthorSerializer :)

[NullVoxPopuli]

thanks for explaining your your technique.

But yeah, I'm trying out this solution to the recursive problem: #1114
So, lets say you have these serializers:

CommentSerializer < #...
  belongs_to :author
end

PostSerializer < #...
  has_many :comments, include: :author
end

BlogSerializer < #...
  has_many :posts, include: [:comments]
end

when rendering a blog, you'd get posts and comments for each post, but not the author for the comments.

I think this is handy, in that you have a more explicit control of how deep the recursion goes

[iMacTia]

Yes, that definitely makes sense, and if you want the author too you just need to change the BlogSerializer like this:

BlogSerializer < #...
  has_many :posts, include: [comments: :author]
end

Recursive loop problem should be fixed as well, the problem is that if you want a standard behaviour (like you always want the comment author when you get a comment), you have to copy-and-paste the same "include" option in all serialisers, where with the include_nested_associations you delegate this to the serializer itself and you just need to manage special cases.
Different solutions to the same problem, at the end. Having both of them allows you to choose the one that better fits your needs

[NullVoxPopuli]

Yeah, I don't think either should conflict with each other (functionality-wise anyway, I'm sure there are code conflicts, as I refactored a bit too)

[NullVoxPopuli]

@iMacTia so, lets say someone wants to always include the author on their comments, how do you feel about the following?

CommentSerializer < #....
  belongs_to :author, always_include: true
end

to notate that no matter who is rendering a comment, they also get the comment's author.

This makes for some interesting notation nuances for other seralizers:

BlogSerializer < #...
  has_many :posts, include: [comments: :author]
end

Here, the specification of author on comments is redundant and doesn't matter.

BlogSerializer < #...
  has_many :posts, include: [:comments]
end

I think this is what would be the most ideal for this scenario, even though it's not explicit as to what's being included in the resulting json.

[iMacTia]

@NullVoxPopuli, you actually raised another interesting point.
I agree on this as it could actually address situations that both our solutions can't address without a copy-and-paste.
I think it could be a nice addition to the set of options for associations.
Maybe not needed in the first place, but if we want to have even greater flexibility, we can also later introduce an exclude option to override the always_include option from outside :)
But again, that comes later, let's do one thing at a time.
My PR for #952 is now obsolete and would need to be redesigned on top of your PR and the 0.10 version (I think it was still 0.9 when I started).
I would suggest we go by step and we start improving the gem with your PR, that will introduce the include option.
After that, we can start thinking together about all other options:

• include_all, like a super include that automatically include all associations (similar to my PR)
• always_include
• exclude

@joaomdmoura, do you like the schedule?
Would really like to hear your thoughts on this guys :)

[jfelchner]

@iMacTia @NullVoxPopuli glad I found this. 😀 I was just about to open an issue. This is exactly what we need out of AMS right now. I read through the issues. I think that include_nested_associations could be replaced by doing something like:

BlogSerializer < #...
  has_many :posts, include: :all_nested_associations
end

IMO this:

BlogSerializer < #...
  has_many :posts, always_include: true
end

Reads like "always include the posts in this output from the blog serializer". What that really means is "include all of the nested associations specified in this serializer". Also always_include sounds like it shouldn't be able to overridden, however the entire point of this is that the serializer's associations can be overridden.

Cons to this approach would mean that no one could have an association called all_nested_associations which I think is fairly safe.

@joaomdmoura we also need to think about how this would work for applying the JSON API spec which allows nested includes to be specified. See here.

[joaomdmoura]

Hey everyone, I've being following it along and have just read through the last comments.
In really in favor of this implementation, really looking forward to get #1127 merged.

@iMacTia the schedule look great, indeed, let's take some baby steps, have the include will be awesome already and solve may of the existing needs. Then we can move forward with the other options.

Indeed @jfelchner, but the good news is that we already support neste associations following json-api conventions 😁.

I think we all agree that this is aiming for the right direction, at least the first step on #1127. Let's get it merged then close this issue and open a new one to deal with always_include 😄

[beauby]

One common pattern to avoid infinite loops in this case is to have a "light" serializer for non-primary resources (relevant for Json/FlattenJson, but not JsonApi). We could possibly make it so that primary resources are serialized according to ResourceSerializer, and by default, non-primary resources get serialized according to ResourcePreviewSerializer, if such a serializer is defined, and falls back to ResourceSerializer otherwise, with a warning/exception in case of infinite loop.

An other solution is to define the whole tree of included resources at the adapter level, JsonApi-style.