Broadcast error "You are operating multiple instances"

[Gummikavalier]

Description:

We got an error message You are operating multiple instances... some features will not behave as designed. some time after upgrading our test system to RC 5.1.0 in the Admin Info view as well as system logs.

We also get an error in the system log about

"name":"StreamBroadcast","msg":"You are operating multiple instances... some features will not behave as designed."}

You don't get this error if you are using USE_NATIVE_OPLOG=true variable in your environment settings.

The error does not go away if we run only one instance, so it seems to be related to streams themselves.

Has anyone else spotted the same issue?

Steps to reproduce:

1. Upgrade to RC 5.1.0
2. Ensure that you are not running with USE_NATIVE_OPLOG=true
3. Check logs and Info page for the error

Expected behavior:

All should be ok.

Actual behavior:

You get an error message:

Server Setup Information:

• Version of Rocket.Chat Server: 5.1.0
• Operating System: RHEL8
• Deployment Method: tar
• Number of Running Instances: 2
• DB Replicaset Oplog: Yes
• NodeJS Version: 14.19.3
• MongoDB Version: 4.4

Client Setup Information

• Desktop App or Browser Version:
• Operating System:

Additional context

We recently restored a clone of the server to troubleshoot other unrelated issues with our original test instance.

[emikolajczak]

Hi,
It is probably preparation to move run Rocket.Chat as multiple instances Enterprise only. May be only microservices will be an option in future?
There is reference https://docs.rocket.chat/quick-start/scaling-rocket.chat/running-multiple-instances

[heilerich]

We are hitting the same issue. This also seems to cause troubles with online status and message notifications (everything only arrives after a user reloads the page).
Does anyone know what the implications of setting USE_NATIVE_OPLOG are?

[ankar84]

That warning comes from this PR #26667
And yes, I guess @emikolajczak is right and that is a preparation to changes where multiple instances will be only in microservices and with EE.

[Gummikavalier]

Interesting. Thanks for the heads up guys! :)
I kind of expect micro services be the way into future anyway, but I hoped the future would not be here just yet. 😁

[Gummikavalier]

We are hitting the same issue. This also seems to cause troubles with online status and message notifications (everything only arrives after a user reloads the page). Does anyone know what the implications of setting USE_NATIVE_OPLOG are?

Currently the there hasn't been any functionality differences for users from using native oplog. Resource wise it moves CPU load more to the nodes from the database, as the nodes become responsible for tracking the changes themselves.

[Gummikavalier]

I'll leave this open because the error is active even with one node/instance.

[Gummikavalier]

Not getting the error anymore with RC 5.1.1 on our test system. Closing for now at least.

Note: You do get the error with two running instances but not with one. However, if you start the second one but stop it afterwards, the error just doesn't go away anymore until you refresh, or possibly stop all instances in between the refreshes.

[nmagedman]

I have confirmation of what @emikolajczak and @Gummikavalier hypothesized above. I wrote RC Tech Support about this issue and was told:

The message is only a warning at the moment (not an issue). In the future … this [i.e. support for running multiple instances] will be supported and available for enterprise users only, and [may not] be available for community users.

[emikolajczak]

I have confirmation of what @emikolajczak and @Gummikavalier hypothesized above. I wrote RC Tech Support about this issue and was told:

The message is only a warning at the moment (not an issue). In the future … this [i.e. support for running multiple instances] will be supported and available for enterprise users only, and [may not] be available for community users.

Interesting.
Sad there is no in advance communication from Rocket.chat about plans.
When scaling will not be available in CE version this solution will be totally not usable for installations greater than several dozen active users.

[Gummikavalier]

We are getting the error message on RC 5.1.2 now with native oplog enabled on and Enterprise instance. So the toaster seems to be point blank for everybody regardless of the license currently.

I'm worried too about what kind of setups are possible in the future with the Enterprise, and whether the setups are enforced to use microservices for instance. Which makes things more complex IMHO until on-premises world actually is ready for those, and not just Cloudflare and other really big names.