Update dependent cells for functions modifying their arguments

[lungben]

Currently, dependencies introduced by functions modifying their arguments in-place are not considered for notebook updates.

Example (each row in a separate Pluto cell):

a = [4, 3, 2, 1]
a # prints [4, 3, 2, 1]
sort!(a) # prints [1, 2, 3, 4], but does not update previous cell

[fonsp]

Unfortunately that's not possible to do correctly.

But if it were, then this notebook would error, just like

a = 1
a
a += 1

Because it creates an ambiguous notebook

[fonsp]

One thing we can do is look for the ! in function names, and show a hint to the user to rewrite the modification and definition into a single cell.

[lungben]

Unfortunately there is (to my knowledge) no way to automatically check which input argument is mutated in Julia.
According to Julia Style Guide (https://docs.julialang.org/en/v1/manual/style-guide/#Write-functions-with-argument-ordering-similar-to-Julia-Base-1), the argument being mutated is usually the first, but not always (functions and IO streams are preceeding, see e.g. Base.filter!(f, a)). And it even does not have to be an argument which is mutated, e.g. plot!. Furthermore, functions may be defined not in agreement to the Style Guide...
Thus, I agree with you that a hint is probably the best what can be done.

In the following cases, in-place functions should be safe:

• If the modified arguments are defined in the same cell - to be on the safe side, check for all arguments?
• E.g. for plot and plot! - this is safe if plot! is called in the same cell as plot.

[fonsp]

I didn't think of that - good that you mention it.

So in that case, do you think that there is still something special that we can do for ! functions? I think it would be helpful if Pluto could hint the author to do definitions and mutations in the same cell, but not if that hint pops up in the wrong places.

[lungben]

I suggest the following:

1. Add a discussion of this subject to the documentation
2. If there is a function ending with ! alone in a cell it is very likely that this is problematic. In this case, Pluto could raise an error.

Although point 2. would not capture all potentially problematic cases, it would still capture a large portion of them without raising too many false positives (I have not found a case yet where this would not create a not-well-defined state).

What do you think?

[fonsp]

Good ideas

[lungben]

Some kind of "solution" / workaround for this and #417 would be to add the possibility to manually specify which variables have been updated in a cell. The execution order would be

1. definition of the variables
2. update of variables (only allowed in one cell)
3. reading of variables

Possible syntax:

x
##
x = [1, 2, 3, 4]
##
f(x) = x % 2 == 0
##
@updates x filter!(f, x)

This should give x = [2, 4] in the first cell.

@fonsp what do you think?

[fonsp]

Creative solution! But if I realize this situation, and I am willing to write some extra code, wouldn't it be easier to just combine the definition and modification into one cell?

[lungben]

Yes, you are right ;-)

It would be best if Pluto "automagically" detects which variables have been modified in each cell, independent of the source of modification. This information could then be used for the execution order, if this is uniquely defined (e.g. 1. setting of x, 2. modification of x, 3. output of x) or for throwing an error.

One possibility to do this would be to fetch all variables in global scope after execution of each cell and calculate their checksum. This is compared to the variable checksums before cell execution (i.e. the result of the previously computed cell). This way, any changes could be detected.

Performance should not be an issue for "small" data structures. For large arrays, etc. this may be in issue. Maybe some kind of "cutoff" could be used to track only changes in sufficiently small data structures?

[RaulDurand]

I am getting a lot of occurrences of this problem and I am not sure if some dependent cells were effectively run and if their results are correct. Meanwhile, I would suggest to add a shortcut to run the whole notebook to make sure that all cells are run with the last changes.

[fonsp]

You can run all cells with Ctrl+A + Shift+Enter: #556 (comment)