Rocket.Chat stability issue when many users disconnected and connected again

[emikolajczak]

Description:

Hi,
From time to time we have observing really serious Rocket.Chat stability issues. Duo to network problem, about half of our 3000 connected users are disconnected and connected again to Rocket.Chat. This situation caused under 100% processors load on Rocket docker containers and unavailability (messages greyed out, client disconnects). Also high containers RAM and Garbage Collector usage (screens below). It was probably caused by presence broadcast storm, because when we switched on "Disable Presence Broadcast", after couple of minutes everything back to normal. Switch off it again caused above problems and unavailability.
Permanent switch on "Disable Presence Broadcast" is not an option because user presence is not correct.

Steps to reproduce:

1. Server with about 3000 users online
2. In example restart reverse proxy (HAproxy, Nginx...)
3. Users are disconnected and connected again
4. Rocket.Chat is unavailable due to high load

Expected behavior:

Disconnect and connect again a lot of users same time not cause Rocket unavailable

Actual behavior:

Rocket is unavailable

Server Setup Information:

• Version of Rocket.Chat Server: 3.9.7
• Operating System: Centos7
• Deployment Method: docker-compose
• Number of Running Instances: 33
• DB Replicaset Oplog: YES
• NodeJS Version: v12.18.4
• MongoDB Version: 4.0.14

Client Setup Information

• Desktop App or Browser Version: All
• Operating System: All

Additional context

Docker contaniers CPU usage

Users sessions

Garbage collector

Heavy rocketchat.users collection updates

Relevant logs:

[ankar84]

It was probably caused by presence broadcast storm, because when we switched on "Disable Presence Broadcast", after couple of minutes everything back to normal.

That is definitely true!
If I need to restart HAProxy during high daily load I always disable user presence monitor first.

Users presence still most harmful to performance feature of Rocket Chat

[emikolajczak]

It was probably caused by presence broadcast storm, because when we switched on "Disable Presence Broadcast", after couple of minutes everything back to normal.

That is definitely true!
If I need to restart HAProxy during high daily load I always disable user presence monitor first.

Users presence still most harmful to performance feature of Rocket Chat

Hi,
When you enable presence monitor again? During working hours or when users load is low?

[ankar84]

When you enable presence monitor again? During working hours or when users load is low?

You better enable Presence Monitor Broadcast again only on a low load, because enable itself do a big load.
Couple of hours users can work without current statuses, IMHO. But if RC on a very high load - users definitely wont be happy - because no ability to chat - main chat feature.

[magicbelette]

Hi,
Remember to enable sticky sessions on your reverse proxy to avoid unecessary broadcast between instances :
#17473 (comment)
Cheers

[emikolajczak]

Hi,
We have configured sticky session on haproxy from begin.

[bbrauns]

we observe the same issues:

mongodb slow log shows an extremely high number of updates of the user collection like the following:

Apr 16 10: 10: 00 chat rocketchat_mongo_1[2662]: 2021 - 04 - 16T08: 10: 00.419 + 0000 I COMMAND[conn431346]command rocketchat.$cmd command: update {
    update: "users",
    updates: [{
            q: {
                _id: "xyz",
                $or: [{
                        status: {
                            $ne: "online"
                        }
                    }, {
                        statusConnection: {
                            $ne: "online"
                        }
                    }
                ]
            },
            u: {
                $set: {
                    status: "online",
                    statusConnection: "online",
                    _updatedAt: new Date(1618560598626)
                }
            },
            upsert: false,
            multi: false
        }
    ],
    ordered: true,
    lsid: {
        id: UUID("xyz")
    },
    $clusterTime: {
        clusterTime: Timestamp(1618560598, 161),
        signature: {
            hash: BinData(0, 0000000000000000000000000000000000000000),
            keyId: 0
        }
    },
    $db: "rocketchat"
}
numYields: 0 reslen: 229 locks: {
    Global: {
        acquireCount: {
            r: 2,
            w: 2
        }
    },
    Database: {
        acquireCount: {
            w: 2
        }
    },
    Collection: {
        acquireCount: {
            w: 2
        }
    }
}
protocol: op_msg 548ms

[emikolajczak]

Hi @bbrauns , could you share your online users and Rocket instances numbers?
Today we have similar issue due to cascade containers restart caused by memory leak. So numbers of clients were disconnected and try to connect again.

[bbrauns]

nothing fancy, max 2500 online users, 18 instances

[emikolajczak]

@bbrauns could you also share your version of Rocket.Chat server?

[ankar84]

Today we have similar issue due to cascade containers restart caused by memory leak. So numbers of clients were disconnected and try to connect again.

My advice to implement everyday instances restart one-by-one in 5 minutes between restart via crontab.
RC in that case much faster and stable during business day.

[johncrisp]

Just discussing this in https://open.rocket.chat with @emikolajczak in #support - do please join in the chat.

This is really a load balancing and support issue rather than an actual bug at the moment. Rocket.Chat is not throwing any errors, and the original post occurs due to network issues.

v3.9.7 is out of any sort of support so this won't get attention from developers until it is reproduced on the latest version as per this:

https://docs.rocket.chat/getting-support

I'll leave this open for now, but unless you reproduce it on latest and give us some in depth analysis including your own specific setups, please continue to chat on openrocket.chat - there are devs there who will engage with you about it and try to help.

https://open.rocket.chat/channel/support?msg=zSJYejLJhLjGt8snb

Thanks

[emikolajczak]

Yes, but it still not resolve problem during disconnect. I.e switch to BACKUP HAproxy due to MASTER failure.

[johncrisp]

Please, continue this on open.rocket.chat as per the reasons above.

[benedikt-wegmann]

@bbrauns could you also share your version of Rocket.Chat server?

We upgrade to 3.13.2 yesterday. The mentioned issue occurred today.