Error when assembling chunks, status code 504 when using S3

[dronesportugal]

How to use GitHub

• Please use the 👍 reaction to show that you are affected by the same issue.
• Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.

Steps to reproduce

1. Upload a big file, in my case 9GB .zip file
2. Error will show "Error when assembling chunks, status code 504"
3. File is locked and unable to be deleted "Failed to load resource: the server responded with a status of 423 ()"

Expected behaviour

Tell us what should happen
Expected file to be upload without errors

Actual behaviour

Tell us what happens instead
Error 504 assembling chunks and file is locked, unable to move or delete

Server configuration

Operating system:
Ubuntu 18.04
Web server:

Database:
Type: pgsql

Version: PostgreSQL 10.10 (Ubuntu 10.10-0ubuntu0.18.04.1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0, 64-bit

Size: 14.4 MB
PHP version:
PHP
Version: 7.2.24

Memory Limit: 512 MB

Max Execution Time: 3600

Upload max size: 1000 MB
Nextcloud version: (see Nextcloud admin page)

Updated from an older Nextcloud/ownCloud or fresh install:

Where did you install Nextcloud from:
Professionaly install
Daniel Hansson
T&M Hansson IT AB
https://www.hanssonit.se

and installed from Unraid app, 2 different sources, same issue
Signing status:

Signing status

Login as admin user into your Nextcloud and access 
http://example.com/index.php/settings/integrity/failed 
paste the results here.

List of activated apps:

App list

If you have access to your command line run e.g.:
sudo -u www-data php occ app:list
from within your Nextcloud installation folder

Nextcloud configuration:

Config report

If you have access to your command line run e.g.:
sudo -u www-data php occ config:list system
from within your Nextcloud installation folder

or 

Insert your config.php content here. 
Make sure to remove all sensitive content such as passwords. (e.g. database password, passwordsalt, secret, smtp password, …)

Are you using external storage, if yes which one: local/smb/sftp/...

Are you using encryption: yes/no

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...

LDAP configuration (delete this part if not used)

LDAP config

With access to your command line run e.g.:
sudo -u www-data php occ ldap:show-config
from within your Nextcloud installation folder

Without access to your command line download the data/owncloud.db to your local
computer or access your SQL server remotely and run the select query:
SELECT * FROM `oc_appconfig` WHERE `appid` = 'user_ldap';


Eventually replace sensitive data as the name/IP-address of your LDAP server or groups.

Client configuration

Browser:
Google Chrome latest version
Operating system:
Windows 10

Logs

Web server error log

Web server error log

Insert your webserver log here

Nextcloud log (data/nextcloud.log)

Nextcloud log

Insert your Nextcloud log here

Browser log

Browser log

Insert your browser log here, this could for example include:

a) The javascript console log
b) The network log
c) ...

[GuyPaddock]

@Corrupto what web server are you using? What type of PHP (Apache or FPM)?

[dronesportugal]

@Corrupto what web server are you using? What type of PHP (Apache or FPM)?

PHP FPM 7.2
Nextcloud 17.0.1 stable

Hope to answer everything you asked.

[GuyPaddock]

@Corrupto We were seeing this too... because of the size of the file involved, nginx is timing-out before PHP is done assembling the final file.

With the current version of Nextcloud, you would want to adjust your NGinx configuration to increase timeouts to give NC more time to assemble files after upload:

        fastcgi_connect_timeout 60;
        fastcgi_send_timeout 1800;
        fastcgi_read_timeout 1800;

This allows up to 30 mins of time to pass. It probably doesn't need to be quite that high, but it all depends on how fast your server disk I/O is.

It'd be great if a future version of Nextcloud had a daemon to handle this kind of task out-of-proc (like Seafile does), since larger files are becoming pretty normal.

[dronesportugal]

@Corrupto We were seeing this too... because of the size of the file involved, nginx is timing-out before PHP is done assembling the final file.

With the current version of Nextcloud, you would want to adjust your NGinx configuration to increase timeouts to give NC more time to assemble files after upload:

        fastcgi_connect_timeout 60;
        fastcgi_send_timeout 1800;
        fastcgi_read_timeout 1800;

This allows up to 30 mins of time to pass. It probably doesn't need to be quite that high, but it all depends on how fast your server disk I/O is.

It'd be great if a future version of Nextcloud had a daemon to handle this kind of task out-of-proc (like Seafile does), since larger files are becoming pretty normal.

Can you let me know the location of the Nginx configuration? its not on the nextcloud folder. Maybe on /etc/php/7.2/fpm? wish file?

[GuyPaddock]

@Corrupto It depends on what nginx containers you are using... In our case, it's located in /etc/nginx/nginx.conf

[zarmhast]

Issue still reproducible in Nextcloud 20.0

I cannot set nginx/PHP-fpm timeouts to high values, as it only shifts the problem, instead of solving it, and at some point I start getting network disconnects instead of 504.

My proposed solution would be to mark files as ready to be reassembled, and handle the lengthy reassembly process in a cron task (cron.php). It seems a simple enough change, yet resulting in a way more stable file server. Especially considering that it is not that easy to add a daemon to a PHP-FPM server.

[burntoc]

Same problem here. Frustrating this has been lingering so long.

[SMH17]

It returns this error with big files upload even when actually they are correctly uploaded (tested checksum, no integrity issues, after upload in processing task it is triggered timeout event while uploaded file chucks are in PHP temp folder with assembling still in progress).
This needs to be addressed, Nextcloud should keep connection alive till the chucks are fully assembled, preventing these timeout events and related error. Setting huge timeouts isn't a good practice.

Configurations where I have reproduced this issue

[1]
OS: Debian Buster (ARM 64bit)
PHP: 7.3
Apache: 2.4
Nextcloud: 20.0.4

File size uploaded with this error 12GB, 33GB
Upload mode: Web interface drag-and-drop (Chromium browser 87.0.4280.88)

[2]
OS: Debian Buster (x86-64)
PHP: 7.4
Nginx: 1.18
Nextcloud: 20.0.4

File size uploaded with this error 40GB
Upload mode: Web interface drag-and-drop (Chromium browser 87.0.4280.88)

[reavessm]

Still happens on Nextcloud 20.06 and Pgsql 11.9. I set the fastcgi settings but that still didn't fix it

[brtptrs]

After adjusting the timeouts we have the situation that big uploads end with "Error when assembling chunks, status code 504" errors only to have the files appear "magically" in nextcloud after some time. It seems that die the feedback on screen can also be misleading.

[SMH17]

The issue is still here even with the latest Nextcloud 21.

[kiplingw]

I confirm this with 21.0.1.1. I hope we won't have to upgrade to version 22 in order to have this fixed, since some of us on shared hosting don't have access to the newer MySQL required to run it.