Eager-loading of associations without using TableInfo structs

[tcwalther]

I understand that GRDB doesn't have lazy-loading associations, which I think is a great design choice. However, I find the concept of writing AuthorInfo structs (name is an example) not that elegant. When I use such *Info structs, it feels that my code is very closely tied to my database structure, and that I am moving away from my business domain.

I wonder whether it would be possible to eager-load associations via optionals. In particular, I propose the following structure:

struct Author {
    var id: Int64
    var name: String
    var books: [Book]?
    static let books = hasMany(Book.self)
}

struct Book { ... }

The books variable is an optional that is nil by default. If, however, I fetch a record via Author.including(all: Author.books), the books variable becomes a list.

That way, in my code, I can simply check the optional to know whether I eager-loaded the associations. And then I can still decide to fetch the association if I feel like it, but it will be explicit. And it would remove a lot of authorInfo.author in my code.

As always, thanks in advance for your help :-).

[groue]

Hello again @tcwalther,

I understand the request. I meet difficulties on the way, maybe you can help solving them.

---

[SOLVED] Decoding the books: [Book]? Author property from Author.including(all: Author.books): this one can be solved with a little modification to the lib. I could make it run.

---

Saving an Author: if Decodable decodes books, then Encodable encodes books as well. Encodable instructs GRDB to store a json payload into a column named "books": we have an SQLite error "column does not exist".

JSON columns is a feature that is there to stay.

The SQLite error is useful: it tells developers that they made a typo somewhere.

I don't see any way to guess that "books" should get a special treatment.

---

If this technique works with to-many associations, then it should work with to-one associations as well. Since associations are a difficult topic, it's very important to prevent exceptions to sneak in.

So let's suppose the feature exists, and that I can do the same for to-one associations. BTW, the database accepts anonymous books (the book.authorId column is nullable). Here we go:

struct Book {
    var author: Author? // Yeah!
    static let author = belongsTo(Author.self)
}

let books = Book
    .including(optional: Book.author) // accept anonymous books
    .fetchAll(db) // [Book]

So far, so good! Now let's write a function that accepts a Book:

/// Return the name of the author of the book
func authorName(book: Book) -> String {
    if let author = book.author {
        return author.name
    } else {
        // Is it an actually anonymous book, or just a book
        // that was fetched without its author?
        // It is impossible to tell.
        return "Anonymous" // Correct, right?
        fatalError("Please eager load author so that I can do my job properly kthx") // just as correct
    }
}

Nil is ambiguous. We have no way to reliably take a decision based on database state.

This contrasts with your Author.books property: as you said, either author's books were eagerly loaded (not nil, with full database information that can be processed), either they were not (nil, no database information).

Now we can tell a story:

1. The GRDB user Alice learns about the technique with a to-many association. It works quite well! 👍
2. Alice reuses the technique with an optional to-one association. At first it works well (the "Correct, right?" option above). Then bugs appear. Alice fixes them by trying to control the Book instances (with or without eager-fetched author). But bugs keep coming back. Alice does not know how to fix them for good. 👎

This horror story has one single origin: a tragic exception. The technique only works correctly with to-many and non optional to-one associations. The user should have never infered that Author.books translates into Book.author just as well. But can we blame the user?

So let's remove this exception, right? For example, let's remove the feature for optional to-one associations? First, I don't quite know what "remove" really means, here. And isn't it another exception (not as bad, ok)?

---

Now you understand why the doc is full of XXXInfo structs. Their blandness and uniformity are one of their best assets. One single technique to rule them all, always with full database information.

_{See for example the Google library Room: it does not use the XXXInfo naming convention, but XXXAndYYY, e.g. BookAndAuthor. What's important here is that the documentation uses a convention. I'm afraid they're all terrible.}

I hope I could make you understand that I don't close the door. But there are tough problems to solve in order to address your request.

[groue]

Can property wrappers help?

struct Author: Codable, FetchableRecord, PersistableRecord {
    var id: Int64
    var name: String
    
    static let books = hasMany(Book.self)
    @AssociatedToMany(books) var books
}

struct Book: Codable, FetchableRecord, PersistableRecord {
    var id: Int64
    var authorId: Int64?
    var title: String
    
    static let author = belongsTo(Author.self)
    @AssociatedToOne(author) var author
}

author.books // [Book], empty when not loaded
author.$books.loaded // Bool

book.author // Author?, nil when not loaded or non-existent
book.$author.loaded // Bool

These property wrappers can solve the Encodable issue (empty implementation of encode(to encoder: Encoder)).

They can tell the developer if association was eager-loaded or not, for all kinds of associations.

Unfortunately, they can not be implemented. In their init(from decoder: Decoder) initializer, they don't have access to the underlying association. They don't know which key they have to decode in the database rows.

This is the fundamental problem of property wrappers in their current incarnation: they are fundamentally tied to instances, and don't have access to any static information (the level at which associations ought to be defined). 🤷‍♂️

[tcwalther]

Yes, I completely understand your arguments. Using nil as a signal whether an association was loaded doesn't generalise to optional associations.

At this point, I might actually start to like the pattern with the structs. I need to find a good naming convention - AuthorInfo isn't that nice, but maybe AuthorWithAssociations or sth is better. I'm still curious to see whether this programming model scales well with our app, or whether "hardcoding" our queries into structs makes it hard to pass data around.

Thanks for your fast and thorough reply!

[groue]

Thanks for your answer!

I need to find a good naming convention [...] AuthorWithAssociations

This really is a pain point, I fully agree.

So we have, so far:

• AuthorWithAssociations
• AuthorWithBooks (Room convention)
• AuthorInfo
• ExtendedAuthor
• CompleteAuthor (I've used this one with some success)
• Career (escape convention, and use a specific name for the author-books pair)

An extra difficulty is that most naming conventions do not scale well when an app does not always fetch the same set of associated records:

• author with their books
• author with their awards
• author with their books and awards
• author with their novels (some strict apps may want to express that books are filtered in the type system)
• ...

I may have to eat my cake and switch to Room convention, the single one that scales.

Please share your experience!

[groue]

(Issue is closed because it does not require extra work, but discussion is open)

[groue]

Maybe a future Swift version will make it possible to grant Codable and database powers to tuples?

Some apps will then avoid naming anything, and just decode (author: Author, books: [Book]).

[tcwalther]

I don't think tuples are a good way of going forward - they aren't particularly handy for being passed around. You now need to remember that $0 is the author and $1 is for books.

AuthorWithBooks becomes very long as soon as there is more than one association involved - hence AuthorWithAssociations. But of course, what about the associations' associations? It remains tricky.

I'll see how much this will bite me in reality.

[groue]

[SOLVED] Decoding the books: [Book]? Author property from Author.including(all: Author.books): this one can be solved with a little modification to the lib. I could make it run.

This modification will ship in the next release (53c662d).

struct AuthorInfo: Decodable, FetchableRecord {
    // Now properly decoded if the association is eager-loaded, nil otherwise
    var books: [Book]?
}