SD card is not synced on android 10+

[cmvizitiu]

Maybe I'm missing something but the latest version from F-Droid will only consider the "standard" folders from internal storage. Some applications (notably the camera app) can be configured to use external storage e.g. SD-card. An option to add both sources would be great; so either list the standard folders two times or put two buttons for each folder: one for "internal" and another for "SDcard".

At least the option to chose between the two before initiating back-up would be useful. Of course the ability to merge the content(s) of the standard folders (e.g. DCIM/camera) into one back-up would be ideal.

[phpbg]

Hi,
the expected behavior is:

• all files appearing in your gallery should be synced, wether they are on external SD card or internal phone storage.
• In the advanced parameters, when you select folders to be included, there is indeed no indication if it is on a sd card or internal storage. Maybe there is room for improvement here...
• If you have a DCIM/camera on both sd card and internal storage, thay will be merged on the dav server

Do you experience a different behavior? If yes, can you give more details, and provide your android version and phone brand?

[cmvizitiu]

Sorry for the delay; I ran some investigations and here are the relevant results as seen from adb and webdav

Overall storage on Android:

adb ls /storage
000041c9 00000064 671f9ce6 .
000041ed 00001000 67067323 ..
000041f8 00008000 671f9ce6 922F-18CD
000041f8 00000d7c 6327cd8e emulated
000041ed 0000003c 671f9ce0 self

On Samsung we have three folders and the resulted sync is "partial".

E.g. the most important type of content for me, the pictures, looks like this:

First, SD-card storage of pictures folders

# adb ls /storage/922F-18CD/DCIM/Camera
000081f8 001f91a3 671fa266 20241028_164038.jpg
000081f8 002413ce 672d00d6 20241107_200302.jpg
000081f8 00272756 672d00f4 20241107_200331.jpg
000081f8 001bfd1c 672f6736 20241109_154421.jpg
000081f8 001b7e19 672f673c 20241109_154428.jpg
000081f8 001a7dbd 672f67a6 20241109_154613.jpg
000081f8 001b0c04 672f67b2 20241109_154626.jpg
000081f8 0020d914 672f68a2 20241109_155024.jpg
#
# adb ls /storage/922F-18CD/Pictures
000041f8 00008000 671f89a0 .thumbnails

And the "internal" storage situation

# adb ls /storage/self/primary/Pictures
000045f8 00001000 672f68ae .thumbnails
000081b0 003401a4 66765d0e 20240622_080646.jpg
000081b0 001da829 66765d0e 20240622_080641.jpg
000081b0 001eb33f 66765d0e 20240622_080623.jpg
000081b0 001e5f14 66765d0e 20240622_080607.jpg
000081b0 00232a7d 66765d0e 20240622_080543.jpg
000081b0 0038b4d0 66765d0e 20240622_080533.jpg
000081b0 001cd31d 66765d0e 20240622_080514.jpg
000081b0 0032479a 66765d0e 20240622_080501.jpg
000081b0 0014e436 66765d0e 20240622_080455.jpg
000081b0 0016d56c 66765d0e 20240622_080452.jpg
000081b0 0015cbeb 66765d0e 20240622_080449.jpg
000081b0 0016c812 66765d0e 20240622_080447.jpg
000081b0 00490f19 66765d0e 20240622_080342.jpg
000081b0 004d485b 66765d0e 20240622_080312.jpg
000081b0 002a228f 66765d0e 20240622_080256.jpg
000081b0 00373e02 66765d0e 20240622_080228.jpg
000081b0 002f1210 66765d0e 20240622_080224.jpg
000081b0 002ebb0a 66765d0e 20240622_080221.jpg
000081b0 002e69d9 66765d0e 20240622_080219.jpg
000081b0 002d4c95 66765d0f 20240622_080217.jpg
#
# adb ls /storage/emulated/0/Pictures
000045f8 00001000 672f68ae .thumbnails
000081b0 003401a4 66765d0e 20240622_080646.jpg
000081b0 001da829 66765d0e 20240622_080641.jpg
000081b0 001eb33f 66765d0e 20240622_080623.jpg
000081b0 001e5f14 66765d0e 20240622_080607.jpg
000081b0 00232a7d 66765d0e 20240622_080543.jpg
000081b0 0038b4d0 66765d0e 20240622_080533.jpg
000081b0 001cd31d 66765d0e 20240622_080514.jpg
000081b0 0032479a 66765d0e 20240622_080501.jpg
000081b0 0014e436 66765d0e 20240622_080455.jpg
000081b0 0016d56c 66765d0e 20240622_080452.jpg
000081b0 0015cbeb 66765d0e 20240622_080449.jpg
000081b0 0016c812 66765d0e 20240622_080447.jpg
000081b0 00490f19 66765d0e 20240622_080342.jpg
000081b0 004d485b 66765d0e 20240622_080312.jpg
000081b0 002a228f 66765d0e 20240622_080256.jpg
000081b0 00373e02 66765d0e 20240622_080228.jpg
000081b0 002f1210 66765d0e 20240622_080224.jpg
000081b0 002ebb0a 66765d0e 20240622_080221.jpg
000081b0 002e69d9 66765d0e 20240622_080219.jpg
000081b0 002d4c95 66765d0f 20240622_080217.jpg
#
# adb ls /storage/self/primary/DCIM/Camera
000081b0 005563ec 666eef18 20240616_165639.jpg
000081f8 001f44c2 646d7024 20230524_050210.jpg
000081f8 001f5800 646d702d 20230524_050219.jpg
000081f8 0020b70c 646d706b 20230524_050321.jpg
000081b0 0073bab4 666f019d 20240616_181541.jpg
000081b0 006207ca 666f01ed 20240616_181700.jpg
000081b0 004ebf88 666f0203 20240616_181722.jpg
000081b0 004b2c69 666f0207 20240616_181727.jpg
#
# adb ls /storage/emulated/0/DCIM/Camera
000081b0 005563ec 666eef18 20240616_165639.jpg
000081f8 001f44c2 646d7024 20230524_050210.jpg
000081f8 001f5800 646d702d 20230524_050219.jpg
000081f8 0020b70c 646d706b 20230524_050321.jpg
000081b0 0073bab4 666f019d 20240616_181541.jpg
000081b0 006207ca 666f01ed 20240616_181700.jpg
000081b0 004ebf88 666f0203 20240616_181722.jpg
000081b0 004b2c69 666f0207 20240616_181727.jpg

Finally, here's how the folder on webdav side looks like

# ls -la ./webdav/phone/Pictures/
total 53504
drwxr-xr-x. 2 apache apache    4096 Oct 28 17:37 ./
drwxr-xr-x. 8 apache apache    4096 Oct 28 17:37 ../
-rw-r--r--. 1 apache apache 2968725 Oct 28 17:37 20240622_080217.jpg
-rw-r--r--. 1 apache apache 3041753 Oct 28 17:37 20240622_080219.jpg
-rw-r--r--. 1 apache apache 3062538 Oct 28 17:37 20240622_080221.jpg
-rw-r--r--. 1 apache apache 3084816 Oct 28 17:37 20240622_080224.jpg
-rw-r--r--. 1 apache apache 3620354 Oct 28 17:37 20240622_080228.jpg
-rw-r--r--. 1 apache apache 2761359 Oct 28 17:37 20240622_080256.jpg
-rw-r--r--. 1 apache apache 5064795 Oct 28 17:37 20240622_080312.jpg
-rw-r--r--. 1 apache apache 4787993 Oct 28 17:37 20240622_080342.jpg
-rw-r--r--. 1 apache apache 1493010 Oct 28 17:37 20240622_080447.jpg
-rw-r--r--. 1 apache apache 1428459 Oct 28 17:37 20240622_080449.jpg
-rw-r--r--. 1 apache apache 1496428 Oct 28 17:37 20240622_080452.jpg
-rw-r--r--. 1 apache apache 1369142 Oct 28 17:37 20240622_080455.jpg
-rw-r--r--. 1 apache apache 3295130 Oct 28 17:37 20240622_080501.jpg
-rw-r--r--. 1 apache apache 1889053 Oct 28 17:37 20240622_080514.jpg
-rw-r--r--. 1 apache apache 3716304 Oct 28 17:37 20240622_080533.jpg
-rw-r--r--. 1 apache apache 2304637 Oct 28 17:37 20240622_080543.jpg
-rw-r--r--. 1 apache apache 1990420 Oct 28 17:37 20240622_080607.jpg
-rw-r--r--. 1 apache apache 2011967 Oct 28 17:37 20240622_080623.jpg
-rw-r--r--. 1 apache apache 1943593 Oct 28 17:37 20240622_080641.jpg
-rw-r--r--. 1 apache apache 3408292 Oct 28 17:37 20240622_080646.jpg
# 
# ls -la ./webdav/phone/DCIM/Camera/
total 35120
drwxr-xr-x. 2 apache apache    4096 Oct 28 17:37 ./
drwxr-xr-x. 4 apache apache    4096 Oct 28 17:37 ../
-rw-r--r--. 1 apache apache 2049218 Oct 28 17:37 20230524_050210.jpg
-rw-r--r--. 1 apache apache 2054144 Oct 28 17:37 20230524_050219.jpg
-rw-r--r--. 1 apache apache 2144012 Oct 28 17:37 20230524_050321.jpg
-rw-r--r--. 1 apache apache 5596140 Oct 28 17:37 20240616_165639.jpg
-rw-r--r--. 1 apache apache 7584436 Oct 28 17:37 20240616_181541.jpg
-rw-r--r--. 1 apache apache 6424522 Oct 28 17:37 20240616_181700.jpg
-rw-r--r--. 1 apache apache 5160840 Oct 28 17:37 20240616_181722.jpg
-rw-r--r--. 1 apache apache 4926569 Oct 28 17:37 20240616_181727.jpg

As you can see, the pictures from 20241109* are missing (being stored on SD-card). I usually set the camera applications to store pictures on SD-card but the content of the SD-card is never synced... I know that because I lost some pictures: For some reasons, upon OS update Samsung insists on deleting everything on the SD-card!

It would seem as if the SD-card is not "merged" (by Samsung's version of Android) into the "standard" folder structure as seen by the user. Since I tend store all my stuff on the SD-card (in case of phone catastrophic failure it can be extracted while internal storage is a dead MMC memory chip on the phone's board), it would be great if I could specify additional folders to sync.

Additional note: The "Documents" folder sync looks a bit stranger, let me know if you need details but it's basically the same issue: certain key folders on SD-card having "official" names e.g. "Documents" or "Downloads" are not synced.

[phpbg]

Thanks for the details. What's your android version?

[cmvizitiu]

14 (patch level October 2024) by Samsung.

[KiralyCraft]

I also came across this issue, and my suggestion would be to keep separation on the WebDav server too. From my understanding, the main advantage of EasySync is that the source of media files are fetched from Android's "Media" iterator (did not check), which gets them from wherever they are. This means, it's not EasySync's particular responsibility to include folders, but it does have the option to filter or exclude them.

Correct me if I'm wrong, but I think this is why there is an option to exclude folders (default opt-in) but not to add extras. What I think would be doable in here would be to have the option to preserve the storage source, given that it would effectively be a "filter" operation.

This brings in my suggestion (which should probably be a separate issue) to separate the files on the WebDav server based on their source. So, /storage/emulated/ files would be in an emulated folder on the server, while /storage/([A-F0-9\-]+) would be in another corresponding folder. This could also "fix" the situation where two files in different storages have the same name, and by merging the folders on the server they would (probably?) be overwritten.

[cmvizitiu]

Oh, I was afraid it was something in this line; Namely that Android tries to somehow "hide" the underlying storage structure by providing an unified view of sorts and in Samsung's case it fails. Most interesting though, the gallery app shows all images from all folders including the SDcard. So they must be using another API call... or different parameters maybe?

[phpbg]

Hi guys,
thank you for all the details.
I need to do further testing and I don't have yet a similar enough phone model (I do have an older huawei phone with an SD card, but the behavior seems correct and all files are synced).
Give me a few more weeks to investigate.

@KiralyCraft thank you for your sponsorship, it's greatly appreciated

[phpbg]

Hi guys,
I've done further testing, and here are the results:

• Indeed for android 10+ devices, only primary storage is synced, so SD card is not synced at all.
• This is because we query media store using MediaStore.VOLUME_EXTERNAL_PRIMARY instead of MediaStore.VOLUME_EXTERNAL
• Using MediaStore.VOLUME_EXTERNAL will sync SD card files, but there will be some undefined behavior if we have a file with the same filename on both the SD card and the internal storage. The risk of dataloss is too high and we must handle properly this situation

As @KiralyCraft pointed out, we can have a distinct folder per storage source on webdav server to handle the situation. I see two issues with this :

• how are we going to handle the upgrade for current users
• i think people without sd card shouldn't be bothered by a single extra folder on their app

I'll try to figure out a solution

[cmvizitiu]

As @KiralyCraft pointed out, we can have a distinct folder per storage source on webdav server to handle the situation. I see two issues with this :

* how are we going to handle the upgrade for current users

* i think people without sd card shouldn't be bothered by a single extra folder on their app

For me any solution would work:

• Automatically create the new folder on the webdav server with a generic name (e.g. SD) OR with the serial-number of the card; Both options have merits.
• Upon version upgrade display a README/warning asking that the folder be created on webdav side
• If unified storage and conflicting names then maybe add a certain string in file names originating from SD e.g. SDcard_20240622_080221.jpg or use the volume's #number 922F-18CD_20240622_080221.jpg

[zerodeux]

If unified storage and conflicting names then maybe add a certain string in file names originating from SD e.g. SDcard_20240622_080221.jpg or use the volume's #number 922F-18CD_20240622_080221.jpg

I backup up that. I guess that having the same file name on both primary and SD storage should be a rare (and quite unexpected) event : how do you end up with such a situation ?

But I'd suggest to rename the conflicting file as 20240622_080221_922F-18CD.jpg : it is less suprising for the end user to have a suffix added rather than a prefix (and respects the name alphasort which is the default on most browsing tools/commands).

[KiralyCraft]

Indeed it can be quite rare to have conflicting file names, but conflicting folders can happen. I would advise against renaming files, because depending on the use case, having to rename thousands of files to remove the suffix of the card ID would be rather painful.

I primarily use EasySync as a backup solution ("Keep local copy" mode), so if my phone suddenly dies I can simply take all files from a folder and "plop" them on a new device, adb push everything and voila, everything's back together. Having the files renamed might have consequences for users who expect their media files to have the same name, and who use the app like this as well.

Migration should be quite simple, the idea of having the storage ID used somehow is quite nice. Also, internal storage will always be "Internal Storage" (or alike), and all devices have it. If going with folders, simply check:

• Is the file `Internal_Storage/.easySync_superlongfilenamehere.txt' present?
  • Yes - The user is using the new folder structure
  • No - The user may not be using the new folder structure, or it's a new sync. Are there files present in the folder?
    • Yes - This may be an upgrading sync. Ask the user if they would like to migrate to the new folder structure (and moving all files as a consequence). Do they even need to know how they're laid out on the server?
    • No - It's a new sync, create folders for the sdcard and internal storage

The problems here:

• What happens if the user syncs to a folder that is not exclusive to EasySync, and performs an upgrade?
• What happens if the user uses an upgraded version of EasySync together with an outdated version on another device?
• The outdated device may download it's own internal storage again, in a folder called "Internal Storage".

Migration should be quite simple

Oh my, why did I think that?