Instant upload from Android app fails

[schiessle]

I took a multiple photos today with my mobile phone and for one photo the instant upload from Android fails repeatingly.

I chose the server repository because the error in the nextcloud.log looks like a server issue:

{"reqId":"lO6ofag5CjHNz9xxnwq7","level":4,"time":"2020-03-30T17:50:48+00:00","remoteAddr":"89.247.35.166","user":"bjoern","app":"webdav","method":"PROPFIND","url":"/remote.php/dav/uploads/bjoern/4e4452a6ee37a7444aaf6649f25293c8","message":{"Exception":"OCP\\Files\\NotFoundException","Message":"/bjoern/files/4e4452a6ee37a7444aaf6649f25293c8/0000000000000000-0000000005494087","Code":0,"Trace":[{"file":"/var/www/nextcloud/lib/private/Files/Node/Folder.php","line":137,"function":"get","class":"OC\\Files\\Node\\Root","type":"->","args":["/bjoern/files/4e4452a6ee37a7444aaf6649f25293c8/0000000000000000-0000000005494087"]},{"file":"/var/www/nextcloud/apps/dav/lib/Connector/Sabre/SharesPlugin.php","line":157,"function":"get","class":"OC\\Files\\Node\\Folder","type":"->","args":["/4e4452a6ee37a7444aaf6649f25293c8/0000000000000000-0000000005494087"]},{"file":"/var/www/nextcloud/apps/dav/lib/Connector/Sabre/SharesPlugin.php","line":210,"function":"getShares","class":"OCA\\DAV\\Connector\\Sabre\\SharesPlugin","type":"->","args":[{"__class__":"OCA\\DAV\\Connector\\Sabre\\File"}]},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/PropFind.php","line":98,"function":"OCA\\DAV\\Connector\\Sabre\\{closure}","class":"OCA\\DAV\\Connector\\Sabre\\SharesPlugin","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/apps/dav/lib/Connector/Sabre/SharesPlugin.php","line":213,"function":"handle","class":"Sabre\\DAV\\PropFind","type":"->","args":["{http://nextcloud.org/ns}sharees",{"__class__":"Closure"}]},{"function":"handleGetProperties","class":"OCA\\DAV\\Connector\\Sabre\\SharesPlugin","type":"->","args":[{"__class__":"Sabre\\DAV\\PropFind"},{"__class__":"OCA\\DAV\\Connector\\Sabre\\File"}]},{"file":"/var/www/nextcloud/3rdparty/sabre/event/lib/EventEmitterTrait.php","line":105,"function":"call_user_func_array","args":[[{"__class__":"OCA\\DAV\\Connector\\Sabre\\SharesPlugin"},"handleGetProperties"],[{"__class__":"Sabre\\DAV\\PropFind"},{"__class__":"OCA\\DAV\\Connector\\Sabre\\File"}]]},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":1059,"function":"emit","class":"Sabre\\Event\\EventEmitter","type":"->","args":["propFind",[{"__class__":"Sabre\\DAV\\PropFind"},{"__class__":"OCA\\DAV\\Connector\\Sabre\\File"}]]},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":981,"function":"getPropertiesByNode","class":"Sabre\\DAV\\Server","type":"->","args":[{"__class__":"Sabre\\DAV\\PropFind"},{"__class__":"OCA\\DAV\\Connector\\Sabre\\File"}]},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":1666,"function":"getPropertiesIteratorForPath","class":"Sabre\\DAV\\Server","type":"->","args":["uploads/bjoern/4e4452a6ee37a7444aaf6649f25293c8",["{DAV:}creationdate","{DAV:}getetag","{http://owncloud.org/ns}permissions","{DAV:}getlastmodified","{http://owncloud.org/ns}id","{DAV:}getcontentlength","{http://owncloud.org/ns}favorite","{DAV:}resourcetype","{http://nextcloud.org/ns}sharees","{DAV:}displayname","{DAV:}getcontenttype","{http://nextcloud.org/ns}has-preview","{http://owncloud.org/ns}size"],1]},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php","line":355,"function":"generateMultiStatus","class":"Sabre\\DAV\\Server","type":"->","args":[{"__class__":"Generator"},false]},{"function":"httpPropFind","class":"Sabre\\DAV\\CorePlugin","type":"->","args":[{"absoluteUrl":"https://wolke.schiessle.org/remote.php/dav/uploads/bjoern/4e4452a6ee37a7444aaf6649f25293c8","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"/var/www/nextcloud/3rdparty/sabre/event/lib/EventEmitterTrait.php","line":105,"function":"call_user_func_array","args":[[{"__class__":"Sabre\\DAV\\CorePlugin"},"httpPropFind"],[{"absoluteUrl":"https://wolke.schiessle.org/remote.php/dav/uploads/bjoern/4e4452a6ee37a7444aaf6649f25293c8","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]]},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":479,"function":"emit","class":"Sabre\\Event\\EventEmitter","type":"->","args":["method:PROPFIND",[{"absoluteUrl":"https://wolke.schiessle.org/remote.php/dav/uploads/bjoern/4e4452a6ee37a7444aaf6649f25293c8","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]]},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":254,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->","args":[{"absoluteUrl":"https://wolke.schiessle.org/remote.php/dav/uploads/bjoern/4e4452a6ee37a7444aaf6649f25293c8","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"/var/www/nextcloud/apps/dav/lib/Server.php","line":319,"function":"exec","class":"Sabre\\DAV\\Server","type":"->","args":[]},{"file":"/var/www/nextcloud/apps/dav/appinfo/v2/remote.php","line":35,"function":"exec","class":"OCA\\DAV\\Server","type":"->","args":[]},{"file":"/var/www/nextcloud/remote.php","line":165,"args":["/var/www/nextcloud/apps/dav/appinfo/v2/remote.php"],"function":"require_once"}],"File":"/var/www/nextcloud/lib/private/Files/Node/Root.php","Line":203,"CustomMessage":"--"},"userAgent":"Mozilla/5.0 (Android) Nextcloud-android/3.11.0","version":"18.0.3.0"}

As you can see I get a file not found exception for /bjoern/files/4e4452a6ee37a7444aaf6649f25293c8/0000000000000000-0000000005494087 which looks strange because the right path would be bjoern/uploads/4e4452a6ee37a7444aaf6649f25293c8/0000000000000000-0000000005494087

I'm running Nextcloud 18.0.3

[kesselb]

The app does a propfind request for https://wolke.xyz.org/remote.php/dav/uploads/bjoern/4e4452a6ee37a7444aaf6649f25293c8 😕

Probably not bad to catch that exception ;) Do you know how to reproduce this request?

[kesselb]

https://github.com/nextcloud/android-library/blob/d81788b50d32f93d81f458a476edc081df081035/src/main/java/com/owncloud/android/lib/resources/files/ChunkedFileUploadRemoteOperation.java#L90-L93

Make sense. I think the server should ignore {http://nextcloud.org/ns}sharees property for such requests and the android app do not request them for chunks ;)

cc @rullzer @tobiasKaminsky

[kesselb]

<?php

define('CHUNK_SIZE', 2000000);

$davUrl = 'https://nextcloud.test/remote.php/dav/';
$data = file_get_contents(__DIR__ . '/4k-wallpaper-background-beautiful-853199.jpg');
$name = 'test-upload-' . random_int(1000, 2000);

$ch = curl_init();

// folder for chunks
curl_setopt($ch, CURLOPT_URL, $davUrl . 'uploads/admin/' . $name);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'MKCOL');
curl_setopt($ch, CURLOPT_USERPWD, 'admin:admin');
curl_exec($ch);

// destination folder we use later
curl_setopt($ch, CURLOPT_URL, $davUrl . 'files/admin/destination-' . $name);
curl_exec($ch);

$chunks = str_split($data, CHUNK_SIZE);
$start = 0;
$end = CHUNK_SIZE;

foreach ($chunks as $key => $chunk) {
	// upload chunks
	curl_setopt($ch, CURLOPT_URL, $davUrl . 'uploads/admin/' . $name . '/' . $start . '-' . $end);
	curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'PUT');
	curl_setopt($ch, CURLOPT_POSTFIELDS, $chunk);
	curl_exec($ch);
	$start = 1 + $end;
	$end = CHUNK_SIZE + $start;

	curl_setopt($ch, CURLOPT_URL, $davUrl . 'uploads/admin/' . $name . '?XDEBUG_SESSION_START=phpstorm');
	curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'PROPFIND');
	curl_setopt($ch, CURLOPT_POSTFIELDS, '<?xml version="1.0" encoding="utf-8" ?>
  <D:propfind xmlns:D="DAV:">
    <D:prop xmlns:OC="http://owncloud.org/ns" xmlns:NC="http://nextcloud.org/ns">
      <D:displayname/>
      <NC:sharees/>
      <OC:size/>
      <OC:id/>
    </D:prop>
  </D:propfind> ');
	curl_exec($ch);
}

// try to move the chunk and use the folder as destination
curl_setopt($ch, CURLOPT_URL, $davUrl . 'uploads/admin/' . $name . '/.file');
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'MOVE');
curl_setopt($ch, CURLOPT_HTTPHEADER, [
	'Destination:' . $davUrl . 'files/admin/' . $name . '.jpg', // that should work
]);
curl_exec($ch);

If someone wants to do more digging. You probably need to adjust XDEBUG_SESSION_START and place a breakpoint apps/dav/lib/Connector/Sabre/SharesPlugin.php:handleGetProperties.

Index: apps/dav/lib/Connector/Sabre/SharesPlugin.php
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
--- apps/dav/lib/Connector/Sabre/SharesPlugin.php	(revision f5e81749901ae7a5c6db91bfd61c886b2c644592)
+++ apps/dav/lib/Connector/Sabre/SharesPlugin.php	(date 1585604323570)
@@ -179,6 +179,10 @@
 			return;
 		}
 
+		if (strpos($sabreNode->getFileInfo()->getInternalPath(), 'uploads/') === 0) {
+			return;
+		}
+
 		// need prefetch ?
 		if ($sabreNode instanceof \OCA\DAV\Connector\Sabre\Directory
 			&& $propFind->getDepth() !== 0

If the chunks are always uploaded to uploads/ that should work. At least i'm not able to reproduce it anymore.

[schiessle]

the patch suggested by @kesselb solved the problem for me. After adding the three lines the Android client instantly uploaded the previously failing picture.

[ymage]

Same here.
Maybe a patch should be pushed to android client too.

[tobiasKaminsky]

Make sense. I think the server should ignore {http://nextcloud.org/ns}sharees property for such requests and the android app do not request them for chunks ;)

While it is true that Android should not ask for this on chunks, according to dav protocol, it is allowed to ask for any property and it should never fail, but show a proper response:

<d:propstat>
      <d:prop>
        <d:displayname/>
        <d:creationdate/>
        <nc:share-expiration/>
        <nc:rich-workspace/>
        <nc:X-Hash-MD5/>
        <d:getcontenttype/>
      </d:prop>
      <d:status>HTTP/1.1 404 Not Found</d:status>
    </d:propstat>

[phantomdez]

I have the same issue but it appears to only happen with large video files when recorded with my camera. Fix suggested above allowed them to upload as well Thanks.

[tobiasKaminsky]

Thanks for confirmation.
On Android I changed it (nextcloud/android-library#418) and it will be in next version.

@kesselb can you maybe create a PR for server? ❤️

[kesselb]

Ideally we add that check for every property not available for chunks. I'm not sure yet but it might be easier to add that check a bit earlier and unset the properties not available for chunks.