Externalize the State Tree (or alternatives)

[sebmarkbage]

React provides the notion of implicitly allowing a child component to store state (using the setState functionality). However, it is not just used for business logic state. It is also used to remember DOM state, or tiny ephemeral state such as scroll position, text selection etc. It is also used for temporary state such as memoization.

This is kind of a magic black box in React and the implementation details are largely hidden. People tend to reinvent the wheel because of it, and invent their own state management systems. E.g. using Flux.

There is still plenty of use cases for Flux, but not all state belongs in Flux stores.

Manually managing the adding/removing of state nodes for all of this becomes a huge burden. So, regardless you're not going to keep doing this manually, you'll end up with your own system that does something similar. We need a convenient and standard way to handle this across components. This is not something that should be 100% in user space because then components won't be able to integrate well with each other. Even if you think you're not using it, because you're not calling setState, you still are relying on the capability being there.

It undermines the ecosystem and eventually everyone will reconverge on a single external state library anyway. We should just make sure that gets baked into React.

We designed the state tree so that the state tree data structure would be opaque so that we can optimize the internals in clever ways. It blocks many anti-patterns where external users breaks through the encapsulation boundaries to touch someone else's state. That's exactly the problem React's programming model tries to address.

However, unfortunately this state tree is opaque to end users. This means that there are a bunch of legitimate use cases are not available to external libraries. E.g. undo/redo, reclaiming memory, restoring state between sessions, debugging tools, hot reloading, moving state from server to the client and more.

We could provide a standard externalized state-tree. E.g. using an immutable-js data structure. However, that might make clever optimizations and future features more difficult to adopt. It also isn't capable of fully encapsulating the true state of the tree which may include DOM state, it may be ok to treat this state differently as a heuristic but the API need to account for it. It also doesn't allow us to enforce a certain level of encapsulation between components.

Another approach is to try to add support for more use cases to React, one-by-one until the external state tree doesn't become useful anymore. I've created separate issues for the ones we we're already planning on supporting:
#4593 Debugger Hooks as Public API
#4594 Hibernating State (not the serialized form)

What else do we need?

Pinging some stake holders:

@leebyron @swannodette @gaearon @yungsters @ryanflorence

[jimfb]

I don't (yet) buy the argument that this shouldn't be solved in userland. As an alternative, I think it might be a better to teach about state hoisting, rather than trying to implement something as part of the core. Personally, I'm still a fan of the state-hoisting pattern (similar but different from #4594):

class View {
  state = { childStates: [ new ChildViewInternalState(), new ChildViewInternalState(), new ChildViewInternalState()] }
  render() {
    return <ChildView state={this.state.childStates[this.props.index]} />;
  }
}

Component implementations provide a black-box state object which may be retained by the parent. If the parent does not provide the component's state, the component will initialize one internally (in which case, the parent forfeits the option to preserve that child component's internal state).

This also solves all three use cases described in #4594, without any special logic within the core and without increasing our API surface area. It has added benefits (for instance, allowing the parent to query a component's internal state without getting a ref to the component, if the implementor chooses to expose that internal information as part of their public API - useful for things like form input). Serializability is decided by the implementor of ChildView.

If we recommend this pattern, it can become the defacto standard, and then you can preserve entire subtrees by saving the root's internal state.

IMO, we should more aggressively utilize this design pattern to solve this problem. If this state-hoisting pattern becomes the defacto standard, many of these problems just evaporate.

It doesn't require any universal library/framework; it only requires that components provide a way of passing in an internal state, which may be treated as a black-box by the parent.

[sebmarkbage]

@jimfb I disagree that it is simple. Everyone doing this is building a library around it because of the boilerplate to deal with diffing lists etc. Edit: If this was the case, it would've been a solved problem in user space by now.

[jimfb]

@sebmarkbage Yeah, diffing lists sucks, but that can be solved with clever diffing libraries. The implementation remains internal to the component and doesn't need a global external state library. Components can swap out a diffing implementation without breaking other components/libraries. Various diffing libraries can coexist perfectly without stepping on each other's toes and without requiring conformance to a global data loading interface (therefore, not harming the ecosystem). We could even release an officially supported/recommended list-diffing implementation. IMHO, people are developing their own frameworks because they don't know of anything better and we haven't sufficiently taught them about data hoisting (the pattern is never mentioned in any docs).

[mjackson]

This is not something that should be 100% in user space because then components won't be able to integrate well with each other

This is a great point, and one that we can already see taking shape in the current Flux landscape. Every Flux library is its own little ecosystem.

[gaearon]

What I'm interested in (and one of the reasons I started Redux) is being able to keep the state tree a computation, but let the app think it's a static value.

If you look at how Redux DevTools work, we keep a "lifted state" (initial app state + debug UI + "staged" actions which will be replayed on hot reload + current state index useful for slider monitors), but the app only sees "unlifted state". (Which makes it behave normally as it's unaware that the state is backed by computation.)

Another observation is that I use Flux/Redux not just as means of "keeping state alive", but rather to:

1. Keep state shape normalized and like a database. Don't worry if it doesn't match the view hierarchy, use memoized computed data in view layer if I need it.
2. Action log that lets me understand what and why happened in the app, and potentially implement that "lifted state" approach I described above for time travel.

Right now it's hard to imagine how React could be solving that problem for me. I want my state logic to be composable, just like my UI is composable, but these are different trees. UI tree uses abstractions like "list view" to show paginated items, model tree may use abstractions like "paginated reducer factory" that creates reducers managing pagination data for ongoing requests. They do not map 1:1 though.

Here's a new interesting thing. A form component mounted to the Redux state tree, but having no dependency on Redux per se. It's nice because it works with record/replay/DevTools/etc, has prop only interface, and only needs a "state provider" in terms of current state and a dispatch function. It doesn't care if it's provided by Redux or something else. It doesn't care where the state is mounted, as long as it's mounted somewhere and dispatch changes it. In fact it could work just fine if dispatch was async and talked to server and the state would be transferred over the wire.

Not sure if it helps, but these are my thoughts. I'm not sure how to fit this into React.

[sebmarkbage]

@gaearon We can implement hot reload and time travel for the purpose of devtools / debugging with any model. People even do that with mutation. Reducers simplifies the implementation but isn't a feature in itself. The use case is solved by #4593.

To put it another way, some of the features you get is purely because you have your own framework. If you maintained React (or sent a PR :P ) you would have the same flexibility to add those features.

Do you have any other examples of use cases for reducers in production environment? We see a use case for declarative programming to solve timing issues. E.g. allowing execution to be deferred based on the tree being in an inconsistent state. https://github.com/reactjs/react-future/blob/master/09%20-%20Reduce%20State/01%20-%20Declarative%20Component%20Module.js#L32 However, this allow us to defer the tree resolution (and therefore props) before invoking the dispatch. This is not a use case that is solved by state reducers per say since any transformation of props is still synchronous. So what is the use case that make them valuable?

#4594 Solves the rollback scenario, I think.

The big difference between Flux patterns and the React component state tree is the divergence between the two hierarchical models. They seem inherently incompatible since one is optimizing for componentization and the other is optimizing for domain modeling. Perhaps, the top of an app should be modeled as a domain and then the leaves are opaque component state somehow? Except, how do you then deal with the same "user" form being open twice in different dialogs with different state. You have to refactor your domain model to model the view concepts which loses the appeal of the domain model to begin with. So perhaps they should remain parallel?

[brandonbloom]

Forgive me for quasi-ignoring the above comments. I'd like to focus on a particular design avenue.

If you're going to make this part of the public API, you shouldn't expose any concrete data types (such as immutable.js) for the state values themselves. Instead, you should specify an interface to be implemented by state stores. However, you'll probably want to expose a concrete data type for the state keys.

The case against concrete state value data types is simple: There's no benefit to restricting what can be state (other than maybe immutable) as the abstract data type of "state store" doesn't need any such restrictions.

Keys on the other hand, have some critical requirements that can influence how you'd implement the state store protocol. You'll need to decide if you want the keys to be opaque or to have some exposed structure to them. Opaque means that a single-level map is basically the only realistic option as the basis of an implementation. With eq/hash, the map can be a hash map. With a defined ordering, it can be a red/black tree, b-tree or similar. Transparent keys, on the other hand, can facilitate tree-shaped storage. Either by using a prefix of the key to delete a range in a sorted-map, or by structuring the store as a nested tree. Note that exposing tree path-style keys closes some other potential future avenues, such as reparenting stateful components. The path-style tradeoff may be wise for React as it exists today.

Another big concern is atomic/batch updates of the store. With tree path keys, you can atomic/batch remove a subtree as I mentioned above. But a state store may still choose to iterate all deleted state to do something like decrement ref-counts or implement other such cleanup.

Ultimately, the specified interface should be some variation of init/get/put/del plus potential extensions. This interface leaves all of the lifecycle management out of the store. get and put are pretty self-explanatory. init is like put, but the value (potentially a thunk of a value) is ignored if the key already exists. del is subject to the potential recursive nature discussed above (given path style keys). All of these methods can (should?) take collections, rather than operate on individual keys to enable atomicity and batch update efficiencies (eg using a transient data structure). There are possible variations on init/get/put like a unified update, but that's a whole separate topic.

Rollback/undo/etc can be optionally achieved by extending the store interface with two methods: capture():object and restore(object). The object can be completely opaque and offer only immutability of the object as a guarantee.

[sebmarkbage]

Let me clarify a situation that I really want to avoid. I often hear arguments that goes something like:

Expose A, B, C. Then let the user implement however they want. It's totally flexible.

Except of course, once you only have A, B and C, there is only really one reasonable way to implement it. I think you alluded to this, @brandonbloom.

One data structure becomes defacto the implementation, or you have many small ones that are effectively the same. Then the only value they provide is for those authors to understand the inner workings.

We're not short on experimental implementations that can by-pass any React logic anyway. Especially with the decoupled renderers. Allowing room for experimentation isn't necessarily a primary goal. We already have that.

So, what would optional data structures for the state store try to solve? Better language interop? Are there app specific perf tradeoffs?

I'm leaning towards the idea that they don't really matter.

I don't think we'll restrict what can be put as values into state. For user space content there is obviously a lot of different data structures that might be useful. (Although immutable and serializable would be nice.)

[brandonbloom]

there is only really one reasonable way to implement it

That's exactly what I want. There shouldn't be a lot of these things and normal users shouldn't be writing them.

what would optional data structures for the state store try to solve? Better language interop?

Yes: It would enable regular js objects for React.js out-of-box. immutable-js for "frameworks" built on React.js. ClojureScript data structures for Om. etc.

Alternatively, you can insist on a standard, such as immutable-js, and make the conscious decision to force frameworks such as Om to either translate state values to immutable-js, or to go-outside the system to implement their on state stores. Insisting on a standard would enable global undo/snapshot/etc, but it would also be a massive breaking change, since people store plain old javascript objects in state now. At best, you could strongly encourage immutable-js and not disallow storage of stateful, unserializable objects. My proposal assumes heterogenous state storage. If this were a new framework, I'd propose just forcing immutable-js and ignore higher-level frameworks like Om.

[jlongster]

I haven't thought about this as much as others above, but the use case I really want want is full snapshotting (I'm sure that's obvious, but just want to say that). I don't even think undo is a strong enough use case for this, because usually when you want undo you take special care to only track the state that you want to undo, and you also need to perform an action on the server to revert the change. Usually it's fine to just use a userland library for this stuff. (of course, with an immutable database, dumb snapshotting is possible but not many people use immutable dbs)

But full snapshotting is something that really begs for controlling where the entire state tree lives, even if child components don't know that their state is not actually internal.

Snapshotting is really useful for debugging. Even in production, you can keep track of the last N app states, and when something goes wrong, serialize them all and send them somewhere a developer can resume and watch to see what happened. It's also just useful for showing off parts of any site statically (for example, go to http://jlongster.com, press cmd+shift+k and paste this in: https://gist.githubusercontent.com/jlongster/55b1f54f0a29ea235dc3/raw/8643c538e836a96eaa9e7b0c5e05d998d0363268/gistfile1.txt)

I plan on researching this more, as I don't have much else to contribute right now. I do understand the difficulty though after implementing my own library: https://github.com/jlongster/blog/blob/master/src/lib/local-state.js. It's really bad though, and only works with a single React tree mounted. I don't even know how you are going to be able to track multiple mounted trees.

[Aetet]

I think base of all data structures, must be Cursor-like. Because it has simple API: get and set. And you can extend them with your own functionality. I think it must be something like https://github.com/zerkalica/immutable-di/#working-with-state . Here's an interface. And of course baobab can be provided as example.

I think separation between container and cursor that navigates on data off the container can help achieve modularity. Because we can have ImmutableCursor or NativeCursor if we need them, but container stay the same.