Normalising state and garbage collection

[olalonde]

The way my state is structured right now is a bit messy.

I have multiple top level keys which are related to the currently logged in user and some keys
which aren't related to the logged in user, e.g.:

{
  // logged in user
  me: {
    id: 'olalonde',
    username: 'olalonde',
    name: 'Olivier Lalonde',
    email: 'olalonde@gmail.com',
    token: 'some-json-webtoken',
  },
  // lgoged in user's files
  myFiles: {
    pagination: { total: 6, limit: 3, page: 1 },
    data: [
      { id: 'a', filename: 'myfile.txt' },
      { id: 'b', filename: 'myfile.doc' },
      { id: 'c', filename: 'myfile.jpg' },
    ],
  },
  // lgoged in user's events
  myEvents: {
    pagination: { total: 2, limit: 3, page: 1 },
    data: [
      { id: 'a', text: 'Some event...' },
      { id: 'b', text: 'Some other event...' },
    ],
  },
  // loaded when user browses a user's profile
  activeUserProfile: {
    id: 'billgates',
    username: 'billgates',
    name: 'Bill Gates',
  },
  // loaded when user browses a user's profile
  activeUserFiles: {
    pagination: { total: 2, limit: 3, page: 1 },
    data: [
      { id: 'd', filename: 'somefile.txt' },
      { id: 'e', filename: 'somefile.doc' },
    ],
  },
}

It's a bit messy and I was looking for suggestions to refactor.

When logging out I'd have to check for a LOGOUT action in the me, myFiles
and myEvents reducers which is not really maintainable if I keep adding
stuff. Another option would be to put everything related to the logged in user
under a me key/reducer and clear me when LOGOUT is dispatched, e.g.:

{
  // logged in user
  me: {
    user: {
      id: 'olalonde',
      username: 'olalonde',
      name: 'Olivier Lalonde',
      email: 'olalonde@gmail.com',
      token: 'some-json-webtoken',
    },
    files: {
      pagination: { total: 6, limit: 3, page: 1 },
      data: [
        { id: 'a', filename: 'myfile.txt' },
        { id: 'b', filename: 'myfile.doc' },
        { id: 'c', filename: 'myfile.jpg' },
      ],
    },
    events: {
      pagination: { total: 2, limit: 3, page: 1 },
      data: [
        { id: 'a', text: 'Some event...' },
        { id: 'b', text: 'Some other event...' },
      ],
    },
  },
  activeUser: {
    // logged in user's events
    // loaded when user browses a user's profile
    user: {
      id: 'billgates',
      username: 'billgates',
      name: 'Bill Gates',
    },
    files: {
      pagination: { total: 2, limit: 3, page: 1 },
      data: [
        { id: 'd', filename: 'somefile.txt' },
        { id: 'e', filename: 'somefile.doc' },
      ],
    },
  }
}

A bit cleaner... But there will be some duplication in my files and user
reducers. Another option would be to normalise the whole thing:

{
  me: 'olalonde',
  users: {
    olalonde: {
      id: 'olalonde',
      username: 'olalonde',
      name: 'Olivier Lalonde',
      email: 'olalonde@gmail.com',
      token: 'some-json-webtoken',
      files: {
        pagination: { total: 6, limit: 3, page: 1 },
        entitites: ['a', 'b', 'c'],
      },
      events: {
        pagination: { total: 2, limit: 3, page: 1 },
        entitites: ['a','b'],
      },
    },
    billgates: {
      id: 'billgates',
      username: 'billgates',
      name: 'Bill Gates',
      files: {
        pagination: { total: 2, limit: 3, page: 1 },
        entities: ['d','e'],
      },
    },
  },
  files: {
    a: { id: 'a', filename: 'myfile.txt' },
    b: { id: 'b', filename: 'myfile.doc' },
    c: { id: 'c', filename: 'myfile.jpg' },
    d: { id: 'd', filename: 'somefile.txt' },
    e: { id: 'e', filename: 'somefile.doc' },
  },
  events: {
    a: { id: 'a', text: 'Some event...' },
    b: { id: 'b', text: 'Some other event...' },
  }
}

Now this kind of normalisation leads to new problems. How do I garbage collect
the state? e.g. If a user browses thousands of profiles, I don't want to end up
with a state that has thousands of entities. I could issue an { type: 'RESET_PROFILE_DATA', user_id: 'billgates' } action when a user's profile component is
unmounted. But how does my reducer know which piece of data can be safely removed (e.g. the logged in user state shouldn't be cleared if the user navigates outside its own profile page). We
could add special cases everywhere but it doesn't feel maintainable. Another problem is that there
could be two separate components which are using the same piece of state and it'd be hard to tell.

I guess we could keep a garbage collection counter for every entity which gets incremented when the data is needed and decremented when the data can be garbage collected, etc. But that can get quite complex.

Also, what if I need to display a user's file list simultaneously in two different components and both have their own pagination. I could have filesForComponentA and filesForComponentB instead of files?

Anyways, I've got more ideas but would be curious how people typically handle this situation.

[gaearon]

If a user browses thousands of profiles

Is this a real use case in your app?

[olalonde]

@gaearon the profiles/files thing was a made up example. But let's take something like Twitter of Facebook for example. It's not unreasonable to think a user might browse through tons of stuff before closing the browser window, right? Maybe it's not something I should worry about but it feels to me like it could eventually become an issue.

[olalonde]

Do you suggest I should go with the normalised approach and worry about "garbage collection" if/when it becomes an issue?

[aweary]

I think avoiding premature optimization is generally good advice, do some stress tests on your app first and see if performance suffers, and if so you can consider optimizations then.

[naw]

I think it's worth considering the perspective that denormalization itself can be a premature optimization.

In the examples above, @olalonde was potentially concerned about storing files and events in more than one place in the state tree. Storing the same kind of thing in more than one place is not intrinsically a poor choice. If you're just displaying the data (but not performing any operations on it), it's likely fine to store it without denormalizing it.

When you add operations against the data (say renaming a file, for example), then you begin to cross the line into duplicating logic in more than one place, at which point you could consider denormalization as a strategy to remove the duplication of logic.

If I had a phone number object that was stored inside two different kinds of things (say, a phone number for a main business profile, as well as phone numbers for each employee within an array of employees), then there wouldn't necessarily be any immediate value to denormalizing business phone numbers and employee phone numbers into a combined phone_numbers hash.

On the other hand, going back to the file example, if you wanted to rename a file, and you wanted such a feature to operate identically for that file regardless of whether it happened to be residing in the me key or the activeUser key, then that's perhaps a good time to denormalize in order to prevent duplication of logic in a reducer. Of course, you can also reduce duplication by building a reducer with a factory, so that you can have identical logic running in multiple places without actually repeating your code. In that case, you can eliminate duplication without needing to denormalize.

I think you have to look at the big picture of how you intend to use the data on the page before you decide when and where to denormalize. I wouldn't necessarily see your 2nd refactoring as superior to your 1st refactoring.

[gaearon]

@naw solid advice

[gaearon]

Do you suggest I should go with the normalised approach and worry about "garbage collection" if/when it becomes an issue?

Normally, yes.

[gaearon]

(If you’re certain it’ll become an issue you might want to use something like Relay instead which I think handles this for you. Or you can roll your own normalized reducer factory based on schema that would keep track of what’s referenced where—but it sounds complicated.)

[olalonde]

@naw thanks, good advice.

@gaearon thanks I'll have a look at Relay as well though it looks like it would require a bigger refactor.

My app state is mostly "read only" so I think I'll keep going with the denormalised approach and maybe name my keys more explicitly to make it clear which views owns which piece of state (e.g. dashboard, profileView, etc.).

[olalonde]

Would be interesting to see different redux state shapes examples... I went through the examples/ directory but the apps are all relatively simple, even the "real world" one. Some links to a few "complex Redux app" repos that use different state shapes would be immensely useful. :)

[markerikson]

Yeah, that's right along with the kind of stuff I'm hoping to cover in my "Structuring Reducers" recipe ( #1784 ). Which is actually kind of an issue - there's three or four overlapping topics, and I'm not yet sure how to approach them yet. State shape, normalizing data, various ways to define reducers, ...

[markerikson]

That said, I do have links to a number of real and example apps over at https://github.com/markerikson/redux-ecosystem-links/blob/master/apps-and-examples.md. Looking through those apps might be instructive.

[woniesong92]

I have a question that might be related to "normalized state." Let's assume that we're trying to build Reddit (just like in your tutorial example) and the following comes from the API server.

subreddits: [{
  title: "food"
  posts: [{
    id: "",
    body: "..",
    comments: [{
      id: ".."
      body: "..",
    }]
    ..morePosts
  },
  title: "culture"
  posts: [{
    id: "",
    body: "..",
    comments: [{
      id: ".."
      body: "..",
    }]
    ..morePosts
  },
]

What's the conventional way of using reducers for the above response? I am generally following the idea of having a look-up table (e.g. subredditByTitle) but I remember seeing from somewhere that I shouldn't nest PostsReducer and CommentsReducer under the same SubredditsReducer. I first started out by having only subreddits reducer in the top-level, but I realized that once I start nesting the reducers, I have to pass down the actions related to Posts and Comments through subreddits reducer.

Should it look like this?

subredditByTitle: {
  food: {
    id: subreddit_1,
    title: "food"
    posts: [post_1, post_2]
  }
  culture: {
    id: subreddit_2,
    title: "culture"
    posts: [post_3, post_4]
  }
}

postsById: {
  post_1: {
    body: ".."
    comments: [comment_1, comment_2]
  },
  post_2: {
    body: "..",
    comments: [comment_3, comment_4]
  }
}

commentsById: {
  comment_1: {
    body: ".."
  },
  comment_2: {
    body: ".."
  }
}

[markerikson]

First, the typical layout for an app with normalized data would be something like:

{
    someAppData1 : { ..... },
    someAppData2 : { ..... },
    entities : {
        EntityType1 : {
            byId : {
                et1id1 : {},
                et1id2 : {},
                // etc
            },
            items : ["et1id1", "et1id2"]
        },
        EntityType2 : {
            byId : {
                et2id1 : {},
                et2id2 : {},
                // etc
            },
            items : ["et2id1", "et2id2"]
        },
    }
}

So, put all the normalized data under a parent key, have a key for each type of item, and then those keys keys contain the lookup tables and ID arrays.

The next issue is how to structure the logic for managing those entities. Some of your reducer logic could be fairly generic, like "look up an item based on its type name and its ID, and update its attributes". Some may be specific to a certain entity type. Some may require access to multiple entity keys at once. This is where you need to start thinking outside the box of combineReducers, and use that in conjunction with other approaches. For example, you can use combineReducers to write your "strictly per-domain" logic, write another function that takes the entire entities state and does cross-entity-type behavior, and use the reduceReducers utility to run the two in sequence and produce your overall "entities" reducer.

[woniesong92]

@markerikson Thank you, that's helpful. One more question: where should isFetching fit in the normalized state? If I am fetching EntityType1 I can probably do the following:

EntityType1 : {
    byId : {
        et1id1 : {},
        et1id2 : {},
        // etc
    },
    items : ["et1id1", "et1id2"],
    isFetching: true
},

But what if I am updating/creating a single entity? For example, when I am fetching posts I will update isFetching of posts entity. But when I am updating, creating, or deleting a single post where should the state of isFetching go?