[Bug]: NextCloud Desktop vfs working in local cache without warning, local files deleted when virtual files environment reset.

[spikusmax]

⚠️ Before submitting, please verify the following: ⚠️

• This is a bug, not a question or a configuration issue.
• This issue is not already reported on Github (I've searched it).
• Nextcloud Server and Desktop Client are up to date. See Server Maintenance and Release Schedule and Desktop Releases for supported versions.
• I agree to follow Nextcloud's Code of Conduct

Bug description

On several occasion now directories have stopped syncing (as per bug #9393). When this happens users have been able to continue working in a local cache of the directory that has stopped syncing, without any error or notice that this is happening. Work in these directories is not reflected in the Nextcloud activity log. When the virtual environment is reset, or the account added and re-added, the work that has been complete locally is removed and irretrievable.

Most recently:

A financial spreadsheet was created by User A in Folder X. This spreadsheet did not appear in User B's sync client. User B then created a spreadsheet with the same name in the Folder X, they also created a subfolder and put numerous PDFs in it, none of the files/folders created by User B were synced to NextCloud and no errors were given. These files are completely absent in the activity logs but there is ample evidence that the work was being done so it is not a case of 'the server ate my homework'. User B worked within this cached version of the directory for several weeks before we reset the virtual environment (to resolve not-appearing files for a different directory). Once we did this the local version of User Bs folder disappeared, attempting to access from Excel's "Recent Files" gave errors that the file no longer existed/had been moved.

Steps to reproduce

Unable to reproduce because the trigger event - a directory that stops syncing - is not reproducible - there appears no pattern in which folders stop syncing.

Expected behavior

I'd expect that all files and folders sync. I'd also expect that an error would be displayed if there are discrepancies between server and client versions, and if there was some issue with the desktop client getting updates on any given directory.
...

Which files are affected by this bug

Nextcloud Desktop VFS client

Operating system

macOS

Which version of the operating system you are running.

Mac OS 15.7.3

Package

Official macOS Virtual files 12+ universal pkg

Nextcloud Server version

31.0.13

Nextcloud Desktop Client version

4.0.6

Is this bug present after an update or on a fresh install?

Updated to a major version (ex. 3.16.3 to 3.17.0)

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

Are you using an external user-backend?

• Default internal user-backend
• LDAP/ Active Directory
• SSO - SAML
• Other

Nextcloud Server logs

Additional info

So far this issue has been discovered well after the the fact and haven't been captured by the debug logs can be created in the desktop client.

[TheGranadilla]

... It mean reseting VSF in case of a stuck sync (issues #8510 #9393 #9012 it all seems to be same problem) can lead to dataloss....
This is... painfull. What could be the best temporary fix, downgrading to 3.16.6, or use daily build?

[spikusmax]

November last year I downgraded everyone to 3.16.6 because of this syncing issue. I then went through a testing period with the up to date version at the time in late December. All appeared well until last week, resetting then resulted in the discovery of another folder that had stopped syncing in early January.

It is excruciating, this is the second time major data loss for the finance staffer has occurred as a result of vfs (first time when the app was updated which was assumend to be a one off, and now with normal use. Syncing has stopped working for random files additional times for a few users, but data loss was not outcome until now, only because the fault was identified early and the environment reset immediately).

I thought maybe, givent the type of work that was being done by the staff, this might be related to this bug:

#9307

However, there was no issues syncing when I tested according to those parameters.

As for a temporary fix, if I can't replicate the problem/trigger event I have l zero optimism that it will be resolved within a useful timeframe for my client. I'll be switching the 40ppl organisation to standard sync until I see documentation that the vfs problem has been diagnosed and resolved. It's just too unreliable at the moment.

[TheGranadilla]

@spikusmax

November last year I downgraded everyone to 3.16.6 because of this syncing issue. I then went through a testing period with the up to date version at the time in late December. All appeared well until last week, resetting then resulted in the discovery of another folder that had stopped syncing in early January.

Thanks for your feedback,
I'm not sure to understand, you confirm that you had sync issue confirmed with versions After 3.16.6, but nothing to report with 3.16.6, no data-loss or desync ? Why not downgrading again?

[spikusmax]

@TheGranadilla

3.16.6, but nothing to report with 3.16.6, no data-loss or desync ? Why not downgrading again?

Two reasons:

1- They pay me by the hour, there's a lot of computers, it is quicker for me to switch than to downgrade.

2- I am not hopeful for a quick fix, 3.17 was released in August last year so it's been an unresolved issue for 6 months and two major updates already, what if it's another 6 months, or a year, or two? Switching to standard sync allows us to continue to be up to date on the client end and evade any future issues associated with old client/current servers versions, or incompatibilities between old clients and new OS releases.

[i2h3]

I only skimmed this issue now but a quick comment on the reset: we probably should add a warning to the switch and change the retention policy of unsynchronized user data. The reset by deactivation and reactivation, as the name implies, is starting over. The possibility itself must persist as a last resort of self help and not to get stuck with outdated file provider domains which cannot synchronize due to other problems (in example network which the client cannot resolve).

[spikusmax]

@i2h3 To be useful in this context, an option to retain data at reset would be useful if it was coupled with an option to reconcile cached data with the incoming data, and give the user the usual options to resolve resultant conflicts. In my case, 99.5% of the cached data was in-sync with the server - manually sorting through hundreds of directories to find discrepancies is not something the average workplace user is going to have the time to do.

[i2h3]

if it was coupled with an option to reconcile cached data with the incoming data, and give the user the usual options to resolve resultant conflicts.

That unfortunately is impossible with file provider domains which have unique identities for their individual lifetime. They are isolated containers. That change was introduced just last year to avoid a lot of other problems and to stabilize the client. I will not get into details too far here, but with a file provider extension we actually have to synchronize three states: macOS, client and server. The first one is not under our control and if messed up, hard to recover. That left a lot of users with broken setups because macOS internally reused dirty databases which did not fit together at all anymore with what our client was reporting.

The usual synchronization process in a plain folder leaves full control to our code while posing other drawbacks. In the end, both are working differently. I need to do some testing, but for now NSFileProviderManager.DomainRemovalMode.preserveDirtyUserData may suffice to at least avoid data loss. I am sorry to say that the other expectations likely would escalate into rocket science dedicated to edge cases which are economically unfeasible among all other priorities.

Hopefully I can manage to make this part of the next scheduled release.