[Bug]: Uploading files larger than 2GB to SFTP external storage backend fails

[chaosgrid]

⚠️ This issue respects the following points: ⚠️

• This is a bug, not a question or a configuration/webserver/proxy issue.
• This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
• Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
• I agree to follow Nextcloud's Code of Conduct.

Bug description

Uploading files larger than 2GB to SFTP external storage backend fails. This is on a docker install using the official Nextcloud image version 29.0.9-apache.

Steps to reproduce

1. Connect sftp storage backend space
2. Upload or copy file larger than 2GB to it.
3. It fails.

Expected behavior

Successful file upload.

Nextcloud Server version

29

Operating system

Other

PHP engine version

None

Web server

None

Database engine version

None

Is this bug present after an update or on a fresh install?

None

Are you using the Nextcloud Server Encryption module?

None

What user-backends are you using?

• Default user-backend (database)
• LDAP/ Active Directory
• SSO - SAML
• Other

Configuration report

List of activated Apps

Nextcloud Signing status

Nextcloud Logs

The log has these entries:

[PHP] Info: Invalid HMAC at /var/www/html/3rdparty/phpseclib/phpseclib/phpseclib/Net/SSH2.php#3717
	COPY /remote.php/dav/files/redacted/test_file_over2gb.dat

[PHP] Info: Connection closed by server at /var/www/html/3rdparty/phpseclib/phpseclib/phpseclib/Net/SSH2.php#4069
	COPY /remote.php/dav/files/redacted/test_file_over2gb.dat

[no app in context] Warning: Failed to copy stream to storage
	COPY /remote.php/dav/files/redacted/test_file_over2gb.dat

Additional info

This issue is probably related to or the same as #48231

It happens on a docker container install using the official Nextcloud image with version 29.0.9-apache

According to Claude, it might be that the packaged phpseclib version is too old. In fact, checking the composer.json in my docker container, it says: "phpseclib/phpseclib": "^2.0.45"

Claude says:

That explains the issue! You're using phpseclib 2.x which has known limitations with files larger than 2GB due to how it handles HMAC calculations with 32-bit integers.

Is this plausible?

When I connect the same storage via FTPS, uploads above 2GB succeed.
I am not sure when this bug was introduced since it definitely worked in the past, we uploaded >2GB files to the storage using the SFTP backend before.

[chaosgrid]

Update: This also happens when downloading a file from sftp backend that is bigger than 2GB. It fails right at the 2GB mark and the Nextcloud client simply says "Connection closed"..

[chaosgrid]

I also found this forum post, seems related (though it is not the SFTP backend): https://help.nextcloud.com/t/unable-to-upload-files-larger-than-2gb-via-webdav/200157

[chaosgrid]

Updated to Nextcloud docker image v30.0.2

Same issues. When downloading a file over 2GB from sftp backend, right around the 2GB mark it fails and the debug output is this:

[joshtrichards]

I see you're working the issue a bit over at phpseclib, but wanted to check on this comment:

When I connect the same storage via FTPS, uploads above 2GB succeed.

Was that a typo or did you really test using FTPS instead of SFTP?

[chaosgrid]

I see you're working the issue a bit over at phpseclib, but wanted to check on this comment:

When I connect the same storage via FTPS, uploads above 2GB succeed.

Was that a typo or did you really test using FTPS instead of SFTP?

Yes, not a typo. I tested the same storage server via the FTPS Nextcloud storage backend and using that, there are no issues with files over 2GB. Also, directly connecting to the storage server using an SFTP tool (like WinSCP) also does not produce these bugs.

[chaosgrid]

@joshtrichards

Hey there, so we concluded the issue over at phpseclib: The error originated from a bug in phpseclib when talking to ProFTPd servers.
phpseclib has been updated and it would be great to update the phpseclib dependency ASAP:
2.0.48 and 3.0.43 have the fix.

Also, @terrafrost phpseclib dev said:

That said, if Nextcloud is using phpseclib 2.0, they would do well to upgrade to 3.0. 3.0 introduces support for newer best practices algorithms that phpseclib 2.0 doesn't have.

[joshtrichards]

Thanks for the update, @chaosgrid. Definitely helpful!

We should get a notification from the bot soon for 2.0.48 to bump to it in our 3rdparty repo.

Re: 3.x - already in progress, but currently stalled until someone has a chance to look into why some of the unit tests are unexpectedly failing. See #48183.

Cc: @susnux