[3.7.0] OPLOG / REPLICASET IS REQUIRED

[binboum]

Description:

I tried an update on 3.7.0, the oplog is functional in 3.4.2, and when I go to 3.7.0 rocket does not launch anymore.

The database is working, the status is ok, I tried to rollback and it is functional.

Steps to reproduce:

Upgrade 3.4.2 to 3.7.0

Expected behavior:

Actual behavior:

Server Setup Information:

• Version of Rocket.Chat Server: 3.7.0
• Operating System: linux
• Deployment Method: docker
• Number of Running Instances: 1
• DB Replicaset Oplog: 1
• NodeJS Version: 12.18.4 - x64
• MongoDB Version: 4.0

Client Setup Information

• Desktop App or Browser Version:
• Operating System:

Additional context

Relevant logs:

Change Stream is available for your installation, give admin permissions to your database user to use this improved version.
self signed certificate in certificate chain
LocalStore: store created at 
LocalStore: store created at 
LocalStore: store created at 
Setting default file store to GridFS
{"line":"120","file":"migrations.js","message":"Migrations: Migrating from version 197 -> 206","time":{"$date":1601421782427},"level":"info"}
{"line":"120","file":"migrations.js","message":"Migrations: Running up() on version 198","time":{"$date":1601421782429},"level":"info"}
{"line":"120","file":"migrations.js","message":"Migrations: Running up() on version 199","time":{"$date":1601421782493},"level":"info"}
Changing schema of User records, this may take a long time ...
Migrating 0/2
Changing schema of User records finished.
{"line":"120","file":"migrations.js","message":"Migrations: Running up() on version 200","time":{"$date":1601421782789},"level":"info"}
{"line":"120","file":"migrations.js","message":"Migrations: Running up() on version 201","time":{"$date":1601421782797},"level":"info"}
{"line":"120","file":"migrations.js","message":"Migrations: Running up() on version 202","time":{"$date":1601421782825},"level":"info"}
{"line":"120","file":"migrations.js","message":"Migrations: Running up() on version 203","time":{"$date":1601421782832},"level":"info"}
{"line":"120","file":"migrations.js","message":"Migrations: Running up() on version 204","time":{"$date":1601421782879},"level":"info"}
{"line":"120","file":"migrations.js","message":"Migrations: Running up() on version 205","time":{"$date":1601421782887},"level":"info"}
{"line":"120","file":"migrations.js","message":"Migrations: Running up() on version 206","time":{"$date":1601421782893},"level":"info"}
{"line":"120","file":"migrations.js","message":"Migrations: Finished migrating.","time":{"$date":1601421782905},"level":"info"}
Not migrating, control is locked. Attempt 1/30. Trying again in 10 seconds.
{"line":"120","file":"migrations.js","message":"Migrations: Not migrating, already at version 206","time":{"$date":1601421792914},"level":"info"}
ufs: temp directory created at "/tmp/ufs"
Using GridFS for custom sounds storage
Using GridFS for custom emoji storage
Updating process.env.MAIL_URL
Browserslist: caniuse-lite is outdated. Please run next command `npm update`
➔ server.js:204 System ➔ error
➔ +---------------------------------------------------------------------------+
➔ |                                SERVER ERROR                               |
➔ +---------------------------------------------------------------------------+
➔ |                                                                           |
➔ |  Rocket.Chat Version: 3.7.0                                               |
➔ |       NodeJS Version: 12.18.4 - x64                                       |
➔ |      MongoDB Version: 4.0.7                                               |
➔ |       MongoDB Engine: wiredTiger                                          |
➔ |             Platform: linux                                               |
➔ |         Process Port: 3000                                                |
➔ |             Site URL: xxxxxxxxxxxxxxxxxxxxxxxxx                    |
➔ |     ReplicaSet OpLog: Disabled                                            |
➔ |          Commit Hash: 2c82cd9cee                                          |
➔ |        Commit Branch: HEAD                                                |
➔ |                                                                           |
➔ |  OPLOG / REPLICASET IS REQUIRED TO RUN ROCKET.CHAT, MORE INFORMATION AT:  |
➔ |  https://go.rocket.chat/i/oplog-required                                  |
➔ |                                                                           |
➔ +---------------------------------------------------------------------------+

[aleadco]

Hi @binboum please check out https://docs.rocket.chat/installation/manual-installation/ubuntu#configure-the-rocket-chat-service

[binboum]

Hi @binboum please check out https://docs.rocket.chat/installation/manual-installation/ubuntu#configure-the-rocket-chat-service

Thanks for your answer, but I have been using Rocket.Chat for a while, and the oplog works on older versions, this version is causing the problem.

[fdellwing]

What is your MONGO_URL and MONGO_OPLOG_URL environment variable?

[binboum]

MONGO_OPLOG_URL=mongodb+srv://oploguser:xxxxxxx@rocket-mongodb-headless.stg-chat.svc.cluster.local/local?replicaSet=rs0&authSource=admin

MONGO_URL=mongodb+srv://rocketchat:xxxxxxxxxxxxxx@rocket-mongodb-headless.stg-chat.svc.cluster.local/rocketchat?replicaSet=rs0&w=majority

I specify that the configuration works below 3.7.0

[fdellwing]

Can you try setting USE_NATIVE_OPLOG=true? This sounds a lot like #18987 which should be fixed with #18993, but maybe there is still a problem.

[binboum]

Thank you for your feedback, indeed with USE_NATIVE_OPLOG = true it works.

➔ System ➔ startup
➔ +-----------------------------------------------------------+
➔ |                       SERVER RUNNING                      |
➔ +-----------------------------------------------------------+
➔ |                                                           |
➔ |  Rocket.Chat Version: 3.7.0                               |
➔ |       NodeJS Version: 12.18.4 - x64                       |
➔ |      MongoDB Version: 4.0.7                               |
➔ |       MongoDB Engine: wiredTiger                          |
➔ |             Platform: linux                               |
➔ |         Process Port: 3000                                |
➔ |             Site URL: xxxxxx  |
➔ |     ReplicaSet OpLog: Enabled                             |
➔ |          Commit Hash: 2c82cd9cee                          |
➔ |        Commit Branch: HEAD                                |
➔ |                                                           |
➔ +-----------------------------------------------------------+

[fdellwing]

Hey @sampaiodiego @rodrigok,

sorry for the ping, but please take a look at this issue. This might affect a wider user range (mongodb > 3.6).

[geekgonecrazy]

@binboum so you were on 3.7.0 final release right? Not a release candidate?

Can you confirm more details of how mongo is running? What OS Rocket.Chat is running on? Just anything we can use to maybe narrow down the variables some

[binboum]

It is deployed on Kubernetes from the Docker images found here: https://hub.docker.com/r/rocketchat/rocket.chat 3.7.0

The mongodb is deployed from Bitnami docker images: https://github.com/bitnami/bitnami-docker-mongodb

I took a look at PR #18993 , @ankar84 asked an interesting question that needs to be clarified, does the user need admin permissions now?

By setting the "clusterManager" permissions on the MONGO_URL user, I no longer get the message: Change Stream is available for your installation, give admin permissions to your database user to use this improved version.

In the documentation it is written that it is necessary to have the role "clusterMonitor"

However I still get the message :

➔ +---------------------------------------------------------------------------+
➔ |                                SERVER ERROR                               |
➔ +---------------------------------------------------------------------------+
➔ |                                                                           |
➔ |  Rocket.Chat Version: 3.7.0                                               |
➔ |       NodeJS Version: 12.18.4 - x64                                       |
➔ |      MongoDB Version: 4.0.7                                               |
➔ |       MongoDB Engine: wiredTiger                                          |
➔ |             Platform: linux                                               |
➔ |         Process Port: 3000                                                |
➔ |             Site URL: xxxxxxxxxxxxxxxxxxxxx                  |
➔ |     ReplicaSet OpLog: Disabled                                            |
➔ |          Commit Hash: 2c82cd9cee                                          |
➔ |        Commit Branch: HEAD                                                |
➔ |                                                                           |
➔ |  OPLOG / REPLICASET IS REQUIRED TO RUN ROCKET.CHAT, MORE INFORMATION AT:  |
➔ |  https://go.rocket.chat/i/oplog-required                                  |
➔ |                                                                           |
➔ +---------------------------------------------------------------------------+

With oplog user :

rs0:PRIMARY> rs.status()
{
	"set" : "rs0",
	"date" : ISODate("2020-09-30T20:22:08.543Z"),
	"myState" : 1,
	"term" : NumberLong(3),
	"syncingTo" : "",
	"syncSourceHost" : "",
	"syncSourceId" : -1,
	"heartbeatIntervalMillis" : NumberLong(2000),
	"majorityVoteCount" : 1,
	"writeMajorityCount" : 1,
	"optimes" : {
		"lastCommittedOpTime" : {
			"ts" : Timestamp(1601497320, 3),
			"t" : NumberLong(3)
		},
		"lastCommittedWallTime" : ISODate("2020-09-30T20:22:00.387Z"),
		"readConcernMajorityOpTime" : {
			"ts" : Timestamp(1601497320, 3),
			"t" : NumberLong(3)
		},
		"readConcernMajorityWallTime" : ISODate("2020-09-30T20:22:00.387Z"),
		"appliedOpTime" : {
			"ts" : Timestamp(1601497320, 3),
			"t" : NumberLong(3)
		},
		"durableOpTime" : {
			"ts" : Timestamp(1601497320, 3),
			"t" : NumberLong(3)
		},
		"lastAppliedWallTime" : ISODate("2020-09-30T20:22:00.387Z"),
		"lastDurableWallTime" : ISODate("2020-09-30T20:22:00.387Z")
	},
	"lastStableRecoveryTimestamp" : Timestamp(1601497310, 5),
	"lastStableCheckpointTimestamp" : Timestamp(1601497310, 5),
	"electionCandidateMetrics" : {
		"lastElectionReason" : "electionTimeout",
		"lastElectionDate" : ISODate("2020-09-30T08:24:06.902Z"),
		"electionTerm" : NumberLong(3),
		"lastCommittedOpTimeAtElection" : {
			"ts" : Timestamp(0, 0),
			"t" : NumberLong(-1)
		},
		"lastSeenOpTimeAtElection" : {
			"ts" : Timestamp(1601454074, 1),
			"t" : NumberLong(2)
		},
		"numVotesNeeded" : 1,
		"priorityAtElection" : 5,
		"electionTimeoutMillis" : NumberLong(10000),
		"newTermStartDate" : ISODate("2020-09-30T08:24:06.905Z"),
		"wMajorityWriteAvailabilityDate" : ISODate("2020-09-30T08:24:06.933Z")
	},
	"members" : [
		{
			"_id" : 0,
			"name" : "rocket-mongodb-0.rocket-mongodb-headless.stg-chat.svc.cluster.local:27017",
			"health" : 1,
			"state" : 1,
			"stateStr" : "PRIMARY",
			"uptime" : 43093,
			"optime" : {
				"ts" : Timestamp(1601497320, 3),
				"t" : NumberLong(3)
			},
			"optimeDate" : ISODate("2020-09-30T20:22:00Z"),
			"syncingTo" : "",
			"syncSourceHost" : "",
			"syncSourceId" : -1,
			"infoMessage" : "",
			"electionTime" : Timestamp(1601454246, 1),
			"electionDate" : ISODate("2020-09-30T08:24:06Z"),
			"configVersion" : 1,
			"self" : true,
			"lastHeartbeatMessage" : ""
		}
	],
	"ok" : 1,
	"$clusterTime" : {
		"clusterTime" : Timestamp(1601497320, 3),
		"signature" : {
			"hash" : BinData(0,"xxxxxxxxxxxxxx"),
			"keyId" : NumberLong("6878052643193749508")
		}
	},
	"operationTime" : Timestamp(1601497320, 3)
}

rs0:PRIMARY> use rocketchat
switched to db rocketchat
rs0:PRIMARY> db.getUsers( {    showCredentials: false, } )
[
	{
		"_id" : "rocketchat.rocketchat",
		"userId" : UUID("a41a8a04-598d-4ad1-905e-5a4e9b291efd"),
		"user" : "rocketchat",
		"db" : "rocketchat",
		"roles" : [
			{
				"role" : "readWrite",
				"db" : "rocketchat"
			},
			{
				"role" : "clusterManager",
				"db" : "admin"
			},
			{
				"role" : "clusterMonitor",
				"db" : "admin"
			}
		],
		"mechanisms" : [
			"SCRAM-SHA-1",
			"SCRAM-SHA-256"
		]
	}
]

rs0:PRIMARY> use admin
switched to db admin
rs0:PRIMARY> db.getUsers( {    showCredentials: false, } )
[
	{
		"_id" : "admin.oploguser",
		"userId" : UUID("0e69ffe5-227d-414a-aaba-1e3e5e3f5504"),
		"user" : "oploguser",
		"db" : "admin",
		"roles" : [
			{
				"role" : "read",
				"db" : "local"
			}
		],
		"mechanisms" : [
			"SCRAM-SHA-1",
			"SCRAM-SHA-256"
		]
	},
	{
		"_id" : "admin.root",
		"userId" : UUID("f98bbebd-de29-400e-ac5a-cf36d6c72bb7"),
		"user" : "root",
		"db" : "admin",
		"roles" : [
			{
				"role" : "root",
				"db" : "admin"
			}
		],
		"mechanisms" : [
			"SCRAM-SHA-1",
			"SCRAM-SHA-256"
		]
	}
]

If you want me to do some testing, I'll make myself available.

[ankar84]

@geekgonecrazy @rodrigok what user and what role need I grant to be able use change streams?
Here is my DB users:

rs0:PRIMARY> use rocketchat
switched to db rocketchat
rs0:PRIMARY> db.getUsers( {    showCredentials: false, } )
[ ]
rs0:PRIMARY> use admin
switched to db admin
rs0:PRIMARY> db.getUsers( {    showCredentials: false, } )
[
        {
                "_id" : "admin.admin",
                "userId" : UUID("3704438b-f36c-43f3-9e53-77cc582d4e33"),
                "user" : "admin",
                "db" : "admin",
                "roles" : [
                        {
                                "role" : "root",
                                "db" : "admin"
                        }
                ],
                "mechanisms" : [
                        "SCRAM-SHA-1",
                        "SCRAM-SHA-256"
                ]
        },
        {
                "_id" : "admin.local",
                "userId" : UUID("58fa2722-1b39-422a-88c4-68b83848132b"),
                "user" : "local",
                "db" : "admin",
                "roles" : [
                        {
                                "role" : "readWrite",
                                "db" : "local"
                        }
                ],
                "mechanisms" : [
                        "SCRAM-SHA-1",
                        "SCRAM-SHA-256"
                ]
        },
        {
                "_id" : "admin.rocket",
                "userId" : UUID("15656d61-6502-47eb-a22d-af7763a4a294"),
                "user" : "rocket",
                "db" : "admin",
                "roles" : [
                        {
                                "role" : "clusterMonitor",
                                "db" : "admin"
                        },
                        {
                                "role" : "readWrite",
                                "db" : "rocketchat"
                        }
                ],
                "mechanisms" : [
                        "SCRAM-SHA-1",
                        "SCRAM-SHA-256"
                ]
        }
]

[rodrigok]

@ankar84 check this article https://hevodata.com/learn/mongodb-change-streams-real-time-data/#ac and follow the option to open change stream for the database.

[binboum]

@rodrigok Can you give more information about the new test that is now being performed to detect the ReplicaSet OpLog ?

I would like to debug, by running commands by hand, understand why it is displayed disabled on the new system, because I have set the permissions well but possible that a few things are missing.

ReplicaSet OpLog: Disabled

[rodrigok]

@binboum we are doing the same thing we were doing before, the only extra thing is that we try to use change streams first and if it's not available we fallback to the old method.

Do you have any log before the green box?

[ankar84]

Do you have any log before the green box?

I think it is a good idea to add oplog type (change streams or legacy)to that green box with storage engine and node version, since that improvement (change streams implementation)

[binboum]

self signed certificate in certificate chain
LocalStore: store created at 
LocalStore: store created at 
LocalStore: store created at 
Setting default file store to GridFS
{"line":"120","file":"migrations.js","message":"Migrations: Not migrating, already at version 206","time":{"$date":1602774498144},"level":"info"}
ufs: temp directory created at "/tmp/ufs"
Using GridFS for custom sounds storage
Using GridFS for custom emoji storage
Browserslist: caniuse-lite is outdated. Please run next command `npm update`
Updating process.env.MAIL_URL
➔ server.js:204 System ➔ error
➔ +---------------------------------------------------------------------------+
➔ |                                SERVER ERROR                               |
➔ +---------------------------------------------------------------------------+
➔ |                                                                           |
➔ |  Rocket.Chat Version: 3.7.1                                               |
➔ |       NodeJS Version: 12.18.4 - x64                                       |
➔ |      MongoDB Version: 4.0.7                                               |
➔ |       MongoDB Engine: wiredTiger                                          |
➔ |             Platform: linux                                               |
➔ |         Process Port: 3000                                                |
➔ |             Site URL: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx                  |
➔ |     ReplicaSet OpLog: Disabled                                            |
➔ |          Commit Hash: e37d7ba8ed                                          |
➔ |        Commit Branch: HEAD                                                |
➔ |                                                                           |
➔ |  OPLOG / REPLICASET IS REQUIRED TO RUN ROCKET.CHAT, MORE INFORMATION AT:  |
➔ |  https://go.rocket.chat/i/oplog-required                                  |
➔ |                                                                           |
➔ +---------------------------------------------------------------------------+

WITH USE_NATIVE_OPLOG=true

LocalStore: store created at 
LocalStore: store created at 
LocalStore: store created at 
Setting default file store to GridFS
{"line":"120","file":"migrations.js","message":"Migrations: Not migrating, already at version 206","time":{"$date":1602774656578},"level":"info"}
ufs: temp directory created at "/tmp/ufs"
Using GridFS for custom sounds storage
Using GridFS for custom emoji storage
Browserslist: caniuse-lite is outdated. Please run next command `npm update`
Updating process.env.MAIL_URL
➔ System ➔ startup
➔ +-----------------------------------------------------------+
➔ |                       SERVER RUNNING                      |
➔ +-----------------------------------------------------------+
➔ |                                                           |
➔ |  Rocket.Chat Version: 3.7.1                               |
➔ |       NodeJS Version: 12.18.4 - x64                       |
➔ |      MongoDB Version: 4.0.7                               |
➔ |       MongoDB Engine: wiredTiger                          |
➔ |             Platform: linux                               |
➔ |         Process Port: 3000                                |
➔ |             Site URL: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  |
➔ |     ReplicaSet OpLog: Enabled                             |
➔ |          Commit Hash: e37d7ba8ed                          |
➔ |        Commit Branch: HEAD                                |
➔ |                                                           |
➔ +-----------------------------------------------------------+