[Bug]: Sync overwrote new versions with old versions

[jedenastka]

⚠️ Before submitting, please verify the following: ⚠️

• This is a bug, not a question or a configuration issue.
• This issue is not already reported on Github (I've searched it).
• Nextcloud Server and Desktop Client are up to date. See Server Maintenance and Release Schedule and Desktop Releases for supported versions.
• I agree to follow Nextcloud's Code of Conduct

Bug description

I have an Nextcloud server which has not been online for a week or so due to an issue with it's storage. During that time, I have done some work on files that are normally synced to the server - since the server has been offline, those were not synced.

After I brought the server back online, restored a backup and did a sync, I was surprised to find local files which I modified during that week replaced with their old copies from the remote, rather than the other way around. Newly created local files were detected and synced to remote properly. This caused me to suffer minor data loss.

I have ran occ files:scan and occ files:cleanup after restoring from backup which I don't know if changed anything.

nextcloud.log: nextcloud_censored.log.gz
Debug archive: client_censored.zip

Steps to reproduce

1. Sync some files with your Nextcloud server
2. Break your Nextcloud server
3. Do modifications to the synced files
4. Wait a week
5. Cause additional unknown factors?
6. Fix your Nextcloud server and restore data from backup (sudo rsync -avhhhAXUH --info=progress2 --partial /path/to/backup/nextcloud/data/ /path/to/nextcloud/data is what I did)
7. Run occ files:scan and occ files:cleanup
8. Perform a sync
9. Local files are reverted to their old remote versions

Expected behavior

New versions of the local files are uploaded to remote. If there are any issues/it's not possible to determine which version is the newer one, ask the user.

Which files are affected by this bug

sync/wydatki.kmy, notatki/belweder_interfejs.txt

Operating system

Linux

Which version of the operating system you are running.

Debian GNU/Linux Unstable

Package

Distro package manager

Nextcloud Server version

31.0.3, 31.0.2 at the moment of sync

Nextcloud Desktop Client version

3.16.2-1

Is this bug present after an update or on a fresh install?

Updated to a major version (ex. 3.3.6 to 3.4.0)

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

Are you using an external user-backend?

• Default internal user-backend
• LDAP/ Active Directory
• SSO - SAML
• Other

Nextcloud Server logs

Additional info

Due to privacy (Microsoft, AI scrapers, stalkers etc.) concerns, I have removed sync databases with potentially sensitive file names from the client and censored the logs. If there is a need, I can share more privately with contributors (see my website for contact options, let me know here before you contact me).

[nilsding]

Many thanks for providing the debug logs!

I could reproduce this via:

1. create a file test.txt somewhere in the sync directory, and let it sync at least once
2. create another file test.txt outside of the sync directory and change its modification time (e.g. touch -m -d'1 week ago' test.txt)
3. upload the file from (2) via the web UI, replacing the existing one
4. observe that the client has overwritten the file

From what I can tell the client only uses the file's etag during a sync to check whether a file needs to be downloaded again, without considering whether the local file has a newer modification time than the server.
IMO it'd be a good idea to check whether the modification time in the local sync db is greater than what the server tells us, and count this as as a conflict.

[camilasan]

From what I can tell the client only uses the file's etag during a sync to check whether a file needs to be downloaded again, without considering whether the local file has a newer modification time than the server.

We do check for modification time/checksum/etc during sync.
But I would expect that the client needs to trust etag changes.

restored a backup

this triggered the etag changes.

• create a file test.txt somewhere in the sync directory, and let it sync at least once

• create another file test.txt outside of the sync directory and change its modification time (e.g. touch -m -d'1 week ago' test.txt)

• upload the file from (2) via the web UI, replacing the existing one

• observe that the client has overwritten the file

If I understand the report right, what was reported was the opposite (or did you mean the client overwrote the local file?):

9. Local files are reverted to their old remote versions

And this happened because of the backup that was restored.

With that said, I have not checked the logs shared, what I wrote was just a general input.

@jedenastka have you run the run the maintenance:data-fingerprint command afterwards?

This is documented here https://docs.nextcloud.com/server/latest/admin_manual/maintenance/restore.html#synchronising-with-clients-after-data-recovery

If the recovered backup is outdated the state of the clients may be more up to date than the state of the server. In this case also make sure to run the maintenance:data-fingerprint command afterwards.

[jedenastka]

@jedenastka have you run the run the maintenance:data-fingerprint command afterwards?

This is documented here https://docs.nextcloud.com/server/latest/admin_manual/maintenance/restore.html#synchronising-with-clients-after-data-recovery

I did not (I would mention it in reproduction steps otherwise), wasn't aware of this. I guess this is on me then?

That said, the docs state that this is the case if the recovered backup is outdated, which I don't think is the case, but maybe I'm just misunderstanding what "outdated" means here (ie. how Nextcloud stores state on the server and on the client). The backup was taken after the server became unoperational, so it had the latest versions of all files at the time.

Also, should I still run it? What's lost is lost anyways, so I guess it doesn't matter.

I'll also mention that other users of the server have since reported being asked to resolve conflicts manually by the client. Not sure if it's normal that it works for some users/files but not others.

[nilsding]

If I understand the report right, what was reported was the opposite (or did you mean the client overwrote the local file?):

@camilasan Yes, I meant that the client replaced the newer local file with the one from the server containing the old timestamp

[camilasan]

I did not (I would mention it in reproduction steps otherwise), wasn't aware of this. I guess this is on me then?

I am not sure.
From the docs:

After restoring a backup of your data directory or the database, you should always call maintenance:data-fingerprint once. This changes the ETag for all files in the communication with sync clients, allowing them to realize a file was modified.

I'll also mention that other users of the server have since reported being asked to resolve conflicts manually by the client. Not sure if it's normal that it works for some users/files but not others.

That would be the expected behavior. Was it only you with the issue of files being overwritten?

My thinking here is that issues like this (after a restored backup the client not behaving correctly) have been reported before and we have fixed it. We have community and customers restoring backups all the time, I would have expected a flood of bug reports if the client was not doing the bare minimum in this scenario (recognizing a conflict).

[jedenastka]

That would be the expected behavior. Was it only you with the issue of files being overwritten?

Apparently.

My thinking here is that issues like this (after a restored backup the client not behaving correctly) have been reported before and we have fixed it. We have community and customers restoring backups all the time, I would have expected a flood of bug reports if the client was not doing the bare minimum in this scenario (recognizing a conflict).

I guess either everyone properly runs maintenance:data-fingerprint or in most cases it doesn't matter if you run it or not, but in my specific case it mattered (and I didn't run it).

It still feels weird to me that technically nothing changed between the two syncs (after all files were identical after restore), yet not syncing for a week broke something. It's probably because I don't understand the internals though.

I don't know. Feel free to close this report if you think it's on me for not running maintenance:data-fingerprint or it's simply not realistic to reproduce this specific case.