Optionally disable merge box in Meteor for publish functions

[mitar]

_2 Upvotes_ This segment is a great example why sometimes it would be beneficial for some publish functions to disable merge box. So currently for publish functions there is a trade-off. Do you want to minimize data send over the wire (then you have to remember what each client has) or do you want to minimize data stored on the server (then you have to send the whole change every time and leave it to the client side to resolve the diff).

I think it would be great if this would be officially supported in Meteor. I think it can be easily done without much changes needed. Simply every added, changed, removed would be send as-is directly to the client or clients. No diffing, no computation, no memory storage.

I think this would better align with a broader Meteor ecosystem than current hack of using meteor-streams (which are sending two DDP messages, added and removed to get merge box to forget the content).

cc @rodrigok

[tmeasday]

I think this sounds like a reasonable request but I would expect it to be difficult to unwind from the rest of the DDP stack easily. I think future changes to the data layer in Meteor will address the core issue however.

[tmikoss]

I'd very much like to see this feature.

I'm dealing with a case where the published data structure is deeply nested and changing very often. Since merge box seems to operate on top level keys only, and in my case nearly always something is changing under every top level key... it does nothing. Except spike the CPU usage on the servers.

[nathan-muir]

I'm also very keen to see this. I was quite excited after it was mentioned in the meteor dev shop talk from Rocket.Chat, but there haven't been any concrete details thus far.

In my case, I'm publishing a large qty of documents that are often inserted/removed, but infrequently modified. Keeping a copy of every published document in memory seems like a huge waste.

[mitar]

Rocket.Chat is using Meteor Streams for now to address this.

[rhyslbw]

+1

[fabiodr]

I believe the workaround now is using the Streamy package and loosing some publish goodness. It is very simple and only sends data throught the same current socket connection.

Maybe a complementary Channels like data transfer pattern is better than hassling the current publish mentality, that the main purpose is sync client with the same data structure (Minimongo).

One question: if i publish to a local only collection, the doc is stored and processed by Mergebox too (everything sended with this.added...)??

[mitar]

I made a package which provides this functionality: https://github.com/peerlibrary/meteor-control-mergebox

[nathan-muir]

Awesome @mitar! I look forward to testing it out.

[egoldblum]

+1 to this feature request. Couldn't really agree more with

Do you want to minimize data send over the wire (then you have to remember what each client has) or do you want to minimize data stored on the server (then you have to send the whole change every time and leave it to the client side to resolve the diff).

[mitar]

In fact I think there are three levels here:

• we diff per top-level fields and send only those changes (current Meteor default)
• we do deep diff and really send only changes (especially useful for arrays)
• or we do not do any diffs, and always simply change the whole thing (what is this ticket about)

Each of those has its advantages and disadvantages. And not everything suits all use cases, so some configuration is needed.

My proposal would be to switch to JSON patch (#4483) instead of custom DDP, and then we can move between the first and the second as needed. And the third is now available through the package.

[nathan-muir]

Just a thought - What about having a capped list of recently published/modified documents per collection.

It can then be combined with an expiry system, to reduce memory usage of documents that are unmodified with a certain time frame.

That way we get the benefits of merge-box for frequently changing documents, but with a limit on memory usage.