SFTP External Storage error

[Teke-lili]

Steps to reproduce

1. Setup external storage SFTP
2. Check Connection
3. Open external storage
4. Upload File with size larger than 524.888 bytes
5. Try download the file

Expected behaviour

1. Setup external storage SFTP
2. Check Connection
3. Open external storage
4. Upload File with size larger than 524.888 bytes
5. Download the file

Actual behaviour

1. Setup external storage SFTP
2. Check Connection
3. Open external storage
4. Upload File with size larger than 524.888 bytes
5. Try to download file
6. Error Page Displayed

Server configuration

**Operating system: Linux 3.10.0-1160.6.1.el7.x86_64 x86_64

**Web server:

**Database: Type: mysql Version: 10.3.25
**PHP version: PHP Version 7.4.15

**Nextcloud version: Nextcloud 20.0.7

**Updated from an older Nextcloud/ownCloud or fresh install: Updated from previous stable version

**Where did you install Nextcloud from:

Signing status:

Signing status

No errors have been found.

List of activated apps:

Activated Apps

Accessibility 1.6.0 Activity 2.13.4 Brute-force settings 2.0.1 Calendar 2.1.3 Collaborative tags 1.10.0 Comments 1.10.0 Community Document Server 0.1.9 Contacts 3.4.3 Contacts Interaction 1.1.0 Dashboard 7.0.0 Default encryption module 2.8.1 Deleted files 1.10.1 Element for Nextcloud 0.7.2 End-to-End Encryption 1.6.2 EPUB/CBZ/PDF ebook reader 1.4.5 External storage support 1.11.1 External Storage: OneDrive 1.1.0 Extract 1.3.0 Federation 1.10.1 File sharing 1.12.2 First run wizard 2.9.0 Group folders 8.2.0 Log Reader 2.5.0 Maps 0.1.8 Monitoring 1.10.0 Nextcloud announcements 1.9.0 Notifications 2.8.0 Password policy 1.10.1 Passwords 2021.2.0 PDF viewer 2.0.1 Photos 1.2.3 Privacy 1.4.0 Ransomware protection 1.8.0 Recommendations 0.8.0 Right click 0.17.0 Share by mail 1.10.0 Support 1.3.0 Talk 10.0.6 Tasks 0.13.6 Text 3.1.0 Theming 1.11.0 Update notification 1.10.0 Usage survey 1.8.0 User status 1.0.1 Versions 1.13.0 Video player 1.9.0 Weather status 1.0.0

Nextcloud configuration:

Nextcloud configuration

{ "system": { "instanceid": "***REMOVED SENSITIVE VALUE***", "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "trusted_domains": [ "www.webserver.cloud" ], "datadirectory": "***REMOVED SENSITIVE VALUE***", "dbtype": "mysql", "version": "20.0.7.1", "overwrite.cli.url": "https:\/\/www.webserver.cloud", "dbname": "***REMOVED SENSITIVE VALUE***", "dbhost": "***REMOVED SENSITIVE VALUE***", "dbport": "", "dbtableprefix": "oc_", "mysql.utf8mb4": true, "dbuser": "***REMOVED SENSITIVE VALUE***", "dbpassword": "***REMOVED SENSITIVE VALUE***", "installed": true, "mail_smtpmode": "sendmail", "mail_smtpsecure": "tls", "mail_sendmailmode": "pipe", "mail_from_address": "***REMOVED SENSITIVE VALUE***", "mail_domain": "***REMOVED SENSITIVE VALUE***", "mail_smtpauthtype": "LOGIN", "mail_smtpport": "587", "mail_smtphost": "***REMOVED SENSITIVE VALUE***", "memcache.local": "\\OC\\Memcache\\APCu", "maintenance": false, "loglevel": 2, "theme": "", "enable_certificate_management": true } }

**Are you using external storage, if yes which one: SFTP, WebDAV

Are you using encryption: yes on WebDav, no on SFTP

Are you using an external user-backend, if yes which one: No

Client configuration

**Browser: Safari, Firefox

**Operating system:OS X 10.14.6, 10.15.7, Windows 7, Windows 10

Logs

Web server error log

Nextcloud log (data/nextcloud.log)

Nextcloud log

[PHP] Error: Error: fopen(sftpread://NextcloudExternal/_Shared_Remote/PhotoArchiv/Peter/DSC_0007.JPG): failed to open stream: "OCA\Files_External\Lib\Storage\SFTPReadStream::stream_open" call failed at /var/www/virtual/webserver/www.webserver.cloud/apps/files_external/lib/Lib/Storage/SFTP.php#378 at <>

0. <>
   OC\Log\ErrorHandler::onError(2, "fopen(sftpread: ... d", "/var/www/virtua ... p", 378, {path: "Nextclou ... l})
1. /var/www/virtual/webserver/www.webserver.cloud/apps/files_external/lib/Lib/Storage/SFTP.php line 378
   fopen("sftpread://Next ... G", "r", false, null)
2. /var/www/virtual/webserver/www.webserver.cloud/lib/private/Files/Storage/Wrapper/Wrapper.php line 300
   OCA\Files_External\Lib\Storage\SFTP->fopen("NextcloudExtern ... G", "r")
3. /var/www/virtual/webserver/www.webserver.cloud/lib/private/Files/Storage/Wrapper/Availability.php line 315
   OC\Files\Storage\Wrapper\Wrapper->fopen("NextcloudExtern ... G", "r")
4. /var/www/virtual/webserver/www.webserver.cloud/lib/private/Files/Storage/Wrapper/Encryption.php line 480
   OC\Files\Storage\Wrapper\Availability->fopen("NextcloudExtern ... G", "r")
5. /var/www/virtual/webserver/www.webserver.cloud/lib/private/Files/Storage/Wrapper/Wrapper.php line 300
   OC\Files\Storage\Wrapper\Encryption->fopen("NextcloudExtern ... G", "r")
6. /var/www/virtual/webserver/www.webserver.cloud/lib/private/Files/View.php line 1165
   OC\Files\Storage\Wrapper\Wrapper->fopen("NextcloudExtern ... G", "r")
7. /var/www/virtual/webserver/www.webserver.cloud/lib/private/Files/View.php line 1001
   OC\Files\View->basicOperation("fopen", "/webserver_cloud ... G", ["read"], "r")
8. /var/www/virtual/webserver/www.webserver.cloud/lib/private/Files/Node/File.php line 115
   OC\Files\View->fopen("/webserver_cloud ... G", "r")
9. /var/www/virtual/webserver/www.webserver.cloud/lib/private/Preview/ProviderV2.php line 86
   OC\Files\Node\File->fopen("r")
10. /var/www/virtual/webserver/www.webserver.cloud/lib/private/Preview/Image.php line 50
    OC\Preview\ProviderV2->getLocalFile(OC\Files\Node\File {})
11. /var/www/virtual/webserver/www.webserver.cloud/lib/private/Preview/GeneratorHelper.php line 63
    OC\Preview\Image->getThumbnail(OC\Files\Node\File {}, 4096, 4096)
12. /var/www/virtual/webserver/www.webserver.cloud/lib/private/Preview/Generator.php line 244
    OC\Preview\GeneratorHelper->getThumbnail(OC\Preview\JPEG {}, OC\Files\Node\File {}, 4096, 4096)
13. /var/www/virtual/webserver/www.webserver.cloud/lib/private/Preview/Generator.php line 140
    OC\Preview\Generator->getMaxPreview(OC\Files\SimpleFS\SimpleFolder {}, OC\Files\Node\File {}, "image/jpeg", "")
14. /var/www/virtual/webserver/www.webserver.cloud/lib/private/Preview/Generator.php line 109
    OC\Preview\Generator->generatePreviews(OC\Files\Node\File {}, [{width: 32,heig ... }], "image/jpeg")
15. /var/www/virtual/webserver/www.webserver.cloud/lib/private/PreviewManager.php line 190
    OC\Preview\Generator->getPreview(OC\Files\Node\File {}, 32, 32, true, "fill", null)
16. /var/www/virtual/webserver/www.webserver.cloud/core/Controller/PreviewController.php line 170
    OC\PreviewManager->getPreview(OC\Files\Node\File {}, 32, 32, true, "fill")
17. /var/www/virtual/webserver/www.webserver.cloud/core/Controller/PreviewController.php line 143
    OC\Core\Controller\PreviewController->fetchPreview(OC\Files\Node\File {}, 32, 32, false, true, "fill")
18. /var/www/virtual/webserver/www.webserver.cloud/lib/private/AppFramework/Http/Dispatcher.php line 169
    OC\Core\Controller\PreviewController->getPreviewByFileId(38764, 32, 32, false, true, "fill")
19. /var/www/virtual/webserver/www.webserver.cloud/lib/private/AppFramework/Http/Dispatcher.php line 100
    OC\AppFramework\Http\Dispatcher->executeController(OC\Core\Controller\PreviewController {}, "getPreviewByFileId")
20. /var/www/virtual/webserver/www.webserver.cloud/lib/private/AppFramework/App.php line 152
    OC\AppFramework\Http\Dispatcher->dispatch(OC\Core\Controller\PreviewController {}, "getPreviewByFileId")
21. /var/www/virtual/webserver/www.webserver.cloud/lib/private/Route/Router.php line 309
    OC\AppFramework\App::main("OC\Core\Controller\PreviewController", "getPreviewByFileId", OC\AppFramework\ ... {}, {_route: "core.P ... "})
22. /var/www/virtual/webserver/www.webserver.cloud/lib/base.php line 1008
    OC\Route\Router->match("/core/preview")
23. /var/www/virtual/webserver/www.webserver.cloud/index.php line 37
    OC::handleRequest()

GET /index.php/core/preview?fileId=38764&x=32&y=32
from 91.56.150.220 by webserver_cloud at 2021-02-24T18:51:32+00:00

SFTP Server Side Log

SSH Server Side log

<session id="9915" service="SSH" remoteAddress=«xxx.xxx.xxx.xxx:yyyyy» virtualAccount=«Name» windowsAccount=«NameW»/>
<channel type="session" id="1"/>
<sfs moduleName="FlowSfsWin" mountPath="/NextcloudExternal" code="90000" desc="Transferring file ended.">
  <parameters path="F:\_remote_nextcloud\_Shared_Remote\PhotoArchiv\Peter\DSC_0007.JPG" timeMs="238" bytesRead="524288" bytesWritten="0" readRangeOffset="0" readRangeLength="524288" writeRangeOffset="0" writeRangeLength="0" createdNewFile="false" resizedFile="false" endedBy="Client"/>
  <help message="File transfer ended by client."/>
</sfs>

Additional Information

The closed issue #20979 referred to changes in apps/files_external/lib/Lib/Storage/SFT.php. Editing the SFTP.php file to remove the special cases for read/write fixes the problem. However chunk size for transfer via SFTP is then limited to 8192bytes resulting in numerous connection attempts via SFTP to the server. Changing the definitions request_chunk(256 * 1024) setting in apps/files_external/lib/Lib/Storage/SFTPReadStream.php enlarges the chunk size but does not fix the problem.

[running-spec]

Same Issue in Version 21.0.0

[szaimen]

I suppose this is still valid in NC21.0.2?

[Teke-lili]

I did not test on NC21.0.2. However, "apps/files_external/lib/Lib/Storage/SFTP.php" in NC21.0.2 is identical to NC20.0.11, which I am using now. The issue still persists.

[Teke-lili]

Did not want to close the issue. Wrong button.

[szaimen]

Hi, please update to 24.0.9 or better 25.0.3 and report back if it fixes the issue. Thank you!

My goal is to add a label like e.g. 25-feedback to this ticket of an up-to-date major Nextcloud version where the bug could be reproduced. However this is not going to work without your help. So thanks for all your effort!

If you don't manage to reproduce the issue in time and the issue gets closed but you can reproduce the issue afterwards, feel free to create a new bug report with up-to-date information by following this link: https://github.com/nextcloud/server/issues/new?assignees=&labels=bug%2C0.+Needs+triage&template=BUG_REPORT.yml&title=%5BBug%5D%3A+