file upload fails when multiple distinct instances run on same server caused by non local /tmp/ufs directory

[Jerome-mi]

Description:

the aim is not increase performance but providing diffrent rocket.chat on same machine, so I don't want to replicate databases :
when installing 2 distinct instances (distinct domaine names, distinct databases), on same server, (same mongod server)
file upload fails in second instance
file download still works fine on both
I'm using GridFS

Steps to reproduce:

any attempt to upload a file into a channel, progress bar remains at 0

Expected behavior:

correct upload as databases are different
why don't use /tmp/ufs/<user> instead of /tmp/ufs ? same for rocket.chat_importer
the question is : is it possible to install different isolated instances of rocketchat on a same machine ???

Actual behavior:

Error_RocketChat_requires_oplog_tailing_when_running_in_multiple_instances

Error_RocketChat_requires_oplog_tailing_when_running_in_multiple_instances_details

Server Setup Information:

• Version of Rocket.Chat Server: 0.74.2
• Operating System: ubuntu 16.04
• Deployment Method: tar / ansible role
• Number of Running Instances: 2 instances listening, 2 distinct databases on different ports behind same nginx
• DB Replicaset Oplog: None
• NodeJS Version: 8.11.4

Additional context

:

Relevant logs:

[reetp]

Please make sure you are on latest 0.74.2 and check the supported software versions here (just checking the basics here!):

https://github.com/RocketChat/Rocket.Chat/releases/tag/0.74.2

Node: 8.11.4
NPM: 6.4.1
MongoDB: 3.2, 3.4, 3.6, 4.0

Do you have some rocket chat logs to show the errors?

[Jerome-mi]

I updated rocket.chat to latest, same issue (edited the post with version and NPM version)
here is the log
[34mI20190207-11:22:54.684(1) ➔ System ➔ startup
�[34mI20190207-11:22:54.684(1) ➔ +---------------------------------------------------+
�[34mI20190207-11:22:54.684(1) ➔ | SERVER RUNNING |
�[34mI20190207-11:22:54.685(1) ➔ +---------------------------------------------------+
�[34mI20190207-11:22:54.685(1) ➔ | |
�[34mI20190207-11:22:54.685(1) ➔ | Rocket.Chat Version: 0.74.2 |
�[34mI20190207-11:22:54.685(1) ➔ | NodeJS Version: 8.15.0 - x64 |
�[34mI20190207-11:22:54.685(1) ➔ | Platform: linux |
�[34mI20190207-11:22:54.686(1) ➔ | Process Port: 3001 |
�[34mI20190207-11:22:54.686(1) ➔ | Site URL: XXX |
�[34mI20190207-11:22:54.686(1) ➔ | ReplicaSet OpLog: Disabled |
�[34mI20190207-11:22:54.686(1) ➔ | Commit Hash: 5fb5e57 |
�[34mI20190207-11:22:54.687(1) ➔ | Commit Branch: HEAD |
�[34mI20190207-11:22:54.687(1) ➔ | |
�[34mI20190207-11:22:54.687(1) ➔ +---------------------------------------------------+
�[34mI20190207-11:22:54.690(1) SyncedCron ➔ info Scheduled "version_check" next run @FRI Feb 08 2019 02:00:00 GMT+0100 (CET)
�[34mI20190207-11:22:54.692(1) SyncedCron ➔ info Scheduled "aggregate-sessions" next run @FRI Feb 08 2019 02:00:00 GMT+0100 (CET)
�[34mI20190207-11:22:54.695(1) SyncedCron ➔ info Scheduled "Generate download files for user data" next run @thu Feb 07 2019 11:30:00 GMT+0100 (CET)
�[34mI20190207-11:22:54.697(1) SyncedCron ➔ info Scheduled "Remove users from Token Channels" next run @thu Feb 07 2019 12:00:00 GMT+0100 (CET)
�[34mI20190207-11:22:54.699(1) SyncedCron ➔ info Scheduled "Generate and save statistics" next run @thu Feb 07 2019 12:22:00 GMT+0100 (CET)
�[34mI20190207-11:22:54.701(1) SyncedCron ➔ info Scheduled "Cleanup OEmbed cache" next run @FRI Feb 08 2019 11:22:00 GMT+0100 (CET)
�[34mI20190207-11:22:54.704(1) SyncedCron ➔ info Scheduled "version_check" next run @FRI Feb 08 2019 02:00:00 GMT+0100 (CET)
�[34mI20190207-11:22:54.707(1) SyncedCron ➔ info Scheduled "aggregate-sessions" next run @FRI Feb 08 2019 02:00:00 GMT+0100 (CET)
�[34mI20190207-11:22:54.709(1) SyncedCron ➔ info Scheduled "Generate download files for user data" next run @thu Feb 07 2019 11:30:00 GMT+0100 (CET)
�[34mI20190207-11:22:54.711(1) SyncedCron ➔ info Scheduled "Remove users from Token Channels" next run @thu Feb 07 2019 12:00:00 GMT+0100 (CET)
�[34mI20190207-11:22:54.714(1) SyncedCron ➔ info Scheduled "Generate and save statistics" next run @thu Feb 07 2019 12:22:00 GMT+0100 (CET)
�[34mI20190207-11:22:54.716(1) SyncedCron ➔ info Scheduled "Cleanup OEmbed cache" next run @FRI Feb 08 2019 11:22:00 GMT+0100 (CET)

[reetp]

No errors in rocketchat, mongo or webserver logs when you try and upload ?

What file storage system are you using?

[Jerome-mi]

I'm using GridFS and every parameters by default except file types blanked to allow any type.
Nothing in mongodb log, (/var/log/mongodb/mongod.log)
Nothing in nginx errors
Nginx access :
XXX - - [07/Feb/2019:16:46:01 +0100] "POST /ufs/GridFS:Uploads/9kDK3FpSi7RRXgh8a?token=eAf939b949&progress=0.12270820850808868 HTTP/2.0" 500 161 "https://XXX/group/testerome" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0"

after activating debug log in Rocket.chat, I have this:
[34mI20190207-16:36:47.302(1) API ➔ debug Success { statusCode: 200, body: { apps: [], success: true } }
�[34mI20190207-16:36:47.462(1) Meteor ➔ method public-settings/get -> userId: PyWERy9TzuxHePc37 , arguments: [ 2019-02-07T11:10:29.677Z ]
�[34mI20190207-16:36:47.489(1) Meteor ➔ method subscriptions/get -> userId: WkgxgB8SHPrB4sZpR , arguments: [ 2019-02-07T10:26:48.046Z ]
�[34mI20190207-16:36:48.406(1) Meteor ➔ method UserPresence:online -> userId: quHiKkdEoaxa6qWa3 , arguments: [ null ]
�[34mI20190207-16:36:48.460(1) Meteor ➔ method rooms/get -> userId: PyWERy9TzuxHePc37 , arguments: [ 2019-02-07T11:13:44.256Z ]
�[34mI20190207-16:36:48.607(1) Meteor ➔ method rooms/get -> userId: quHiKkdEoaxa6qWa3 , arguments: [ 2019-02-07T15:33:38.391Z ]
�[34mI20190207-16:36:48.817(1) Meteor ➔ method public-settings/get -> userId: quHiKkdEoaxa6qWa3 , arguments: [ 2019-02-07T15:36:47.460Z ]
�[34mI20190207-16:36:49.034(1) Meteor ➔ method rooms/get -> userId: quHiKkdEoaxa6qWa3 , arguments: [ 2019-02-07T15:33:38.391Z ]
�[34mI20190207-16:36:49.235(1) Meteor ➔ method subscriptions/get -> userId: quHiKkdEoaxa6qWa3 , arguments: [ 2019-02-07T15:33:38.384Z ]
�[34mI20190207-16:36:49.459(1) Meteor ➔ method subscriptions/get -> userId: PyWERy9TzuxHePc37 , arguments: [ 2019-02-07T11:13:55.343Z ]
�[34mI20190207-16:36:49.645(1) Meteor ➔ method subscriptions/get -> userId: quHiKkdEoaxa6qWa3 , arguments: [ 2019-02-07T15:33:38.384Z ]
�[34mI20190207-16:37:05.538(1) UploadProxy ➔ debug Upload URL: /ufs/GridFS:Uploads/M8atWzhPtFBve4rpR?token=E88bDB39F9&progress=0.15186118938158089
�[34mI20190207-16:37:05.540(1) UploadProxy ➔ debug Correct instance

[Jerome-mi]

before upgrade, NGINX access was like that :

XXXX - - [01/Feb/2019:07:14:44 +0100] "POST /ufs/GridFS:Uploads/hHJRk7pAtHaXoKBP7?token=08182b98eA&progress=1 HTTP/2.0" 204 342 "https://XXX/direct/Dominique" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0"

since tomorrow, after upgrade, it's like that :

XXX - - [07/Feb/2019:16:46:01 +0100] "POST /ufs/GridFS:Uploads/9kDK3FpSi7RRXgh8a?token=eAf939b949&progress=0.12270820850808868 HTTP/2.0" 500 161 "https://XXX/group/testerome" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0

[9joshua]

@Jerome-mi
Your node version: 8.15.0
RC 74.2 requires: 8.11.4

Did you try with 8.11.4?

[Jerome-mi]

moved to 8.11.4, same issue

[9joshua]

I experienced a similar issue just yesterday on 74.2. For some users uploaded files would hang at 0%. Any image uploads, such as screen shots, would only show the user a extremely blurry preview of the image. If the user tried to clicked on a download, they would occasionally see an access denied screen. In one case logging out and back in solved the problem, but in another, I just clicked to edit the user's details and clicked save, and the problem went away.

Check to see if all users cannot upload files. If any of them can upload, while others cannot, this could be the same issue.

[Jerome-mi]

the problem is due to ownership of /tmp/ufs directory. when 2 or more instances of rocket.chat are installed on the same machine with differents users, the unique ufs directory become closed to other users, so only one instance of rocket.chat permits upload of file or avatars.
it's not possible to set TMP or TMPDIR as environment variable in service launching environment because the path seems to be hard coded
I changed the ownhership of /tmp/ufs to root with mode 0777 as a workaround but it's not perfect
another solution is to use an unique user with diffrents mongo databases, which is not perfect too

[aventrax]

Same issue here! I migrated 2 previously separated instances on the same server with 2 different users. Upload on one of the instance is not working due to a Permission Denied error.

I'm running RC 1.2.1, please fix it!

[reetp]

Same issue here! I migrated 2 previously separated instances on the same server with 2 different users. Upload on one of the instance is not working due to a Permission Denied error.

See workaround above

I'm running RC 1.2.1, please fix it!

It's Open Source. It will get fixed when it gets fixed.

Obviously if you can fix it then your Pull Request with code will be welcome.