Enabling deletion mutations

[mmikita95]

Providing __delitem__ method for StateProxy and StreamsyncState, and dict-like remove() API referencing the same method. Adding a "-" prefix for state keys meant to be removed at the frontend side when applying for mutated set of StateProxy. Enabling frontend to distinguish deletion mutations in ingestMutations function of core/index.ts.

[ramedina86]

This works for explicitly deleted items. But are we covering the case when someone regenerates a new StateProxy like Fabien showed in his code (when he used a dictionary comprehension)?

For example:

state["animals"] = { "cat": 1, "dog": 2 }

and then in the second run:

state["animals"] = { "dog": 3 }

Are we somehow detecting this and notifying the front end of the deletion?

I don't think it is, please let me know. Unfortunately I think a more complex and less efficient approach will be needed.

[mmikita95]

Well noticed. I'll verify it and post an update.

[mmikita95]

I've tested this case, and it seems that currently mutations for nested dictionaries are processed such that they're "nullified" each time and then reconstructed with "new" content. I'm not really sure that this behavior is efficient, but it allows to account for the case you've described. So, yeah, non-explicit removal technically works too.

Though, an interesting edge case that I've noticed to not work here is when we try to "non-explicitly remove" the whole state at once, i.e.

def very_strange_handler(state):
    state = {}

This simply does not produce any mutations at all, seemingly due to not invoking __setitem__ on neither StreamsyncState nor StateProxy.

UPD: Testing with the specific code that Fabien provided indeed causes some sort of problem, though I'm not sure what is it exactly. I'll continue to investigate.

[ramedina86]

Thanks for looking into this in depth; I appreciate it. It's a complex mechanism and it's easy to miss the edge cases. Let me know whenever you have an update, feel free to take your time. Also if you want to look at it together I'm happy to.

[ramedina86]

As you suggested, can you please add some tests to this PR? Mainly to test deletion and the "Fabien code" for key exclusion.

[mmikita95]

It was a lengthy investigation indeed, but finally a fruitful one! As it turned out, the problem with executing the code snippet Fabien provided was "desynchronization" of initial_assignment in children StateProxy instances.

I.e., when we are looking at:

items = state["items"].state

# This results in a dictionary containing existing StateProxy,
# with initial_assignment set to False
items = {k: v for k, v in items.items() if k != context["itemId"]}

state["items"] = items  # ← __setitem__ triggered on state, passing a dictionary of StateProxy instances

What this amounts to is that we are given with a "newly born" StateProxy, but all its children StateProxy instances were already existing ones, with initial_assignment set to False. And, when extracting mutations:

# src/streamsync/core.py:267-268
# '+' flag is prepended to escaped_key as per latest update
if value.initial_assignment:
    serialised_mutations[f"+{escaped_key}"] = serialised_value

this important operation is not happening, and serialized mutations passed to frontend look like this:

{"+items": null}

which effectively resulted in "state nullification" bug as described by Joel here – "newly-added" items property was meant to be a dictionary with children, but with no children provided in mutations, it just gets "nullified".

The re-initialization of passed StateProxy instances allows to avoid this issue, and as far as I can see, is the most efficient solution for this. As always, I'd greatly appreciate feedback or suggestions for improving.

[ramedina86]

Finally got the opportunity to look into this in depth. Splendid work. You now understand the mechanism much better than I do!