Non-deterministic nodes

[RunDevelopment]

Motivation
Some features we would like to add as nodes are inherently non-deterministic. This includes relatively simple things like "what's the current time?", "how many times has the user run this chain?", and "give me a random number." These example, while very simple, are very useful, so it would be nice if we could add them.

The problem with non-deterministic nodes is that they break chaiNNer. Currently, pretty much everything assumes that nodes are deterministic, and even more systems will make that assumption in the future. Here are some things that non-deterministic nodes break right now or will break in the future:

• The current chain executor assumes that all nodes are deterministic and uses this assumption to cache nodes.
• Trivial optimizations (such as iterator outlining) aren't correct anymore.
• Type broadcasts aren't correct, because a static type system requires determinism to propagate types.
• Live execution (planned feature) relies on caching intermediate results from previous runs, and non-deterministic nodes can't be cached.

Description
Find a way to support non-deterministic nodes without breaking chaiNNer. This includes answering the following questions:

• What are the semantics of the outputs of non-deterministic nodes?
  This is mostly about what down-stream nodes see. E.g. R be a non-deterministic node that generates a random number, and let R be connected to 2 nodes A and B like so:

  R ─> A
  └──> B
  

  Do A and B see the same random number? If so, why would this change if B is inside an iterator?
• How do these semantics interact with caching and optimization?
• Are there non-deterministic nodes that are acceptable to cache, e.g. for live execution? If so, what property do they have that makes it acceptable?

[jackcarey]

1. Semantics: nodes could implement non-deterministic behaviour, but the outputs should still be predictable. IMO, each output should pass the same value to all connected nodes. If more than one random number is needed for example, then the extendable list pattern could be taken from 'Text Append' to add more outputs with different values. For iterators: if the non-deterministic node isn't part of the iteration scope itself, I'm of the opinion that it shouldn't recalculate each time, because that's the flow would essentially have to either "work backwards" or "pull the non-deterministic node forward" to create a new output.

2. Caching & Optimizations: in a similar position to the toggle for enabling and disabling a node at the moment, how would you feel about a button to 'freeze' node outputs? Or making that toggle handle freezing as well? This could be part of the base node definition as a 'freezable' attribute? I think, with some refinement, this could be a workable solution. It'd have to be intuitive as to how it works though, which will take a bit more thought.

[RunDevelopment]

Semantics

I agree. To make the semantics I'm thinking of more precise, I'll make a few illustrative examples. In the following examples, R will be a non-deterministic node that outputs a random int, A and B will be simple deterministic nodes, and I will be an iterator (e.g. Image File Iterator).

R ─> A
└──> B

This should behave like the following python code:

r: int = R()
A(r)
B(r)

Now iterators, with R being outside the iterator I:

R ─> A
│
│  ┌─[ I ]─┐
│  │       │
└─────> B  │
   └───────┘

r: int = R()
A(r)
for item in I():
    B(r)

And inside the iterator I:

┌──[ I ]──┐
│ R ──> A │
│ └───> B │
└─────────┘

for item in I():
    r: int = R()
    A(r)
    B(r)

Those semantics around iterators also make it clear why deterministic nodes can be outlined while non-deterministic nodes cannot.

I think this fully solves how non-deterministic nodes should behave absent any optimizations and caching. Optimizations simply have to ensure that the above example still have the same. Caching will also have to follow the above semantics.

Remaining issues

• Data broadcasts are still immediately outdated. Imagine a node that generates a random image and we pass the result into View Image. The image preview would be immediately outdated.

  So we have to redefine the data sent to the frontend as the result of the current execution instead of the unique result of the current chain. This means that we carefully have to review all frontend code that touches broadcast data.

• Type broadcasts are still incorrect. Since types are about the chain and not the last execution of it, outdated types are just incorrect. We should probably just not send type broadcasts for nodes that are or are downstream of non-deterministic nodes.

• Live execution is still a mess. But we also haven't specified the semantics of live execution, so maybe we shouldn't worry about it too much and just deal with non-deterministic nodes then.

Freezing

This is an interesting feature for sure. It nicely complements non-deterministic nodes, but it would also be very useful for deterministic nodes. This feature essentially gives users control over which nodes are cached between runs. Very useful.

I also want to point out that freezing is not like disabling. It's not that the node is frozen, but that a specific output of the node is frozen. The frozen output is ephemeral and lost after restarting chainner, while the disabled state can be saved.

Implementation wise, chainner's executor uses a hierarchy of caches, so we just have to put the outputs of frozen nodes into a higher-level cache to keep them alive between runs. But we should hammer out the exact semantics of frozen nodes in a separate issue.

[jackcarey]

Semantics

I appreciate the code examples, thanks. I agree re. type broadcasts, but don't know your codebase well enough (yet??) to speak on the other remaining issues.

Freezing

Yeah, freezing only outputs is better than the whole node. Just to clarify do you mean freezing all outputs at once, instead of individual outputs? For more complex nodes a startFrozen boolean property could be useful either way, so that only the first output is used in subsequent runs until the user explicitly "thaws" the node. It'll be some time before I know your codebase enough to look into this more by the way