Bug: Files with newer timestamp are not chosen as version to be kept

[JC-comp]

Describe the bug

Due to the rejection of #2617 (comment), I'll assume that files with newer timestamps across remote and local should be chosen as the version to be kept. However, there are some cases where this policy is not consistently followed. If these are expected behaviors, the timestamp-first mode should be considered as a feature request. Both sides should be evaluated for changes rather than one side always replacing the other, especially if both sides could be changing.

Operating System Details

skipped

Client Installation Method

From Source

OneDrive Account Type

Personal

What is your OneDrive Application Version

v2.5.0-alpha-5 GitHub version: v2.4.25-64-g1630ae3

What is your OneDrive Application Configuration

skipped

What is your 'curl' version

skipped

Where is your 'sync_dir' located

Local

What are all your system 'mount points'

skipped

What are all your local file system partition types

skipped

How do you use 'onedrive'

skipped

Steps to reproduce the behaviour

Case 1-1

$ touch 1.txt
$ onedrive --sync
$ curl -X PUT "https://graph.microsoft.com/v1.0/me/drive/root:/1.txt:/content" \
   --data "modify first (should be ignored)"
$ echo "modify again locally (newer version should be kept)" > 1.txt
$ onedrive --sync
$ cat 1.txt
modify first (should be ignored)

Case 1-2 (resync)

$ touch 1.txt
$ onedrive --sync
$ curl -X PUT "https://graph.microsoft.com/v1.0/me/drive/root:/1.txt:/content" \
   --data "modify first (should be ignored)"
$ echo "modify again locally (newer version should be kept)" > 1.txt
$ onedrive --sync --resync
$ cat 1.txt
modify first (should be ignored)

Case 2-1

$ touch 1.txt
$ onedrive --sync --local-first
$ echo "modify first (should be ignored)" > 1.txt
$ curl -X PUT "https://graph.microsoft.com/v1.0/me/drive/root:/1.txt:/content" \ 
   --data "modify again remotely(newer version should be kept)"
$ onedrive --sync --local-first
$ cat 1.txt
modify first (should be ignored)

Case 2-2 (resync)

$ touch 1.txt
$ onedrive --sync --local-first
$ echo "modify first (should be ignored)" > 1.txt
$ curl -X PUT "https://graph.microsoft.com/v1.0/me/drive/root:/1.txt:/content" \ 
   --data "modify again remotely(newer version should be kept)"
$ onedrive --sync --local-first --resync 
$ cat 1.txt
modify first (should be ignored)

Complete Verbose Log Output

Case 1-1

Reading configuration file: /home/alan/.config/onedrive/config
Configuration file successfully loaded
Using 'user' configuration path for application state data: /home/alan/.config/onedrive
Using the following path to store the runtime application log: /var/log/onedrive
Using IPv4 and IPv6 (if configured) for all network operations
Checking Application Version ...
Attempting to initialise the OneDrive API ...
Configuring Global Azure AD Endpoints
The OneDrive API was initialised successfully
Opening the item database ...
Application Version:  v2.5.0-alpha-5 GitHub version: v2.4.25-64-g1630ae3
Account Type:         personal
Default Drive ID:     af2b5e52bea96776
Default Root ID:      AF2B5E52BEA96776!sea8cc6beffdb43d7976fbc7da445c639
Remaining Free Space: 5.00 GB (5364164153 bytes)
Sync Engine Initialised with new Onedrive API instance
All application operations will be performed in the configured local 'sync_dir' directory: /home/alan/OneDrive
Fetching /delta response from the OneDrive API for Drive ID: af2b5e52bea96776
Processing API Response Bundle: 1 - Quantity of 'changes|items' in this bundle to process: 2
Finished processing /delta JSON response from the OneDrive API
Processing 1 applicable changes and items received from Microsoft OneDrive
Processing OneDrive JSON item batch [1/1] to ensure consistent local state
Number of items to download from OneDrive: 1
The local file to replace (./1.txt) has been modified locally since the last download. Renaming it to avoid potential local data loss.
The local item is out-of-sync with OneDrive, renaming to preserve existing file and prevent local data loss: ./1.txt -> ./1-55db26f69037.txt
Downloading file ./1.txt ... done
Performing a database consistency and integrity check on locally stored data
Processing DB entries for this Drive ID: af2b5e52bea96776
Processing ~/OneDrive
The directory has not changed
Processing 1.txt
The file has not changed
Scanning the local file system '~/OneDrive' for new data to upload
New items to upload to OneDrive: 1
Total New Data to Upload:        52 Bytes
Uploading new file ./1-55db26f69037.txt ... done.
Performing a last examination of the most recent online data within Microsoft OneDrive to complete the reconciliation process
Fetching /delta response from the OneDrive API for Drive ID: af2b5e52bea96776
Processing API Response Bundle: 1 - Quantity of 'changes|items' in this bundle to process: 2
Finished processing /delta JSON response from the OneDrive API
Processing 1 applicable changes and items received from Microsoft OneDrive
Processing OneDrive JSON item batch [1/1] to ensure consistent local state

Sync with Microsoft OneDrive is complete

Case 1-2

Reading configuration file: /home/alan/.config/onedrive/config
Configuration file successfully loaded
Using 'user' configuration path for application state data: /home/alan/.config/onedrive
Using the following path to store the runtime application log: /var/log/onedrive

The usage of --resync will delete your local 'onedrive' client state, thus no record of your current 'sync status' will exist.
This has the potential to overwrite local versions of files with perhaps older versions of documents downloaded from OneDrive, resulting in local data loss.
If in doubt, backup your local data before using --resync

Are you sure you wish to proceed with --resync? [Y/N] y

Deleting the saved application sync status ...
Using IPv4 and IPv6 (if configured) for all network operations
Checking Application Version ...
Attempting to initialise the OneDrive API ...
Configuring Global Azure AD Endpoints
The OneDrive API was initialised successfully
Opening the item database ...
Application Version:  v2.5.0-alpha-5 GitHub version: v2.4.25-64-g1630ae3
Account Type:         personal
Default Drive ID:     af2b5e52bea96776
Default Root ID:      AF2B5E52BEA96776!sea8cc6beffdb43d7976fbc7da445c639
Remaining Free Space: 5.00 GB (5364164153 bytes)
Sync Engine Initialised with new Onedrive API instance
All application operations will be performed in the configured local 'sync_dir' directory: /home/alan/OneDrive
Fetching /delta response from the OneDrive API for Drive ID: af2b5e52bea96776
Processing API Response Bundle: 1 - Quantity of 'changes|items' in this bundle to process: 3
Skipping item - excluded by skip_file config: /./~tmp21_1
Finished processing /delta JSON response from the OneDrive API
Processing 1 applicable changes and items received from Microsoft OneDrive
Processing OneDrive JSON item batch [1/1] to ensure consistent local state
Local file time discrepancy detected: ./1.txt
This local file has a different modified time 2024-Feb-04 15:51:53Z (UTC) when compared to remote modified time 2024-Feb-04 15:51:49Z (UTC)
The local file has a different hash when compared to remote file hash
Local item does not exist in local database - replacing with file from OneDrive - failed download?
The local item is out-of-sync with OneDrive, renaming to preserve existing file and prevent local data loss: ./1.txt -> ./1-55db26f69037.txt
Number of items to download from OneDrive: 1
Downloading file ./1.txt ... done
Performing a database consistency and integrity check on locally stored data
Processing DB entries for this Drive ID: af2b5e52bea96776
Processing ~/OneDrive
The directory has not changed
Processing 1.txt
The file has not changed
Scanning the local file system '~/OneDrive' for new data to upload
New items to upload to OneDrive: 1
Total New Data to Upload:        52 Bytes
Uploading new file ./1-55db26f69037.txt ... done.
Performing a last examination of the most recent online data within Microsoft OneDrive to complete the reconciliation process
Fetching /delta response from the OneDrive API for Drive ID: af2b5e52bea96776
Processing API Response Bundle: 1 - Quantity of 'changes|items' in this bundle to process: 2
Finished processing /delta JSON response from the OneDrive API
Processing 1 applicable changes and items received from Microsoft OneDrive
Processing OneDrive JSON item batch [1/1] to ensure consistent local state

Sync with Microsoft OneDrive is complete

Case 2-1

Reading configuration file: /home/alan/.config/onedrive/config
Configuration file successfully loaded
Using 'user' configuration path for application state data: /home/alan/.config/onedrive
Using the following path to store the runtime application log: /var/log/onedrive
Using IPv4 and IPv6 (if configured) for all network operations
Checking Application Version ...
Attempting to initialise the OneDrive API ...
Configuring Global Azure AD Endpoints
The OneDrive API was initialised successfully
Opening the item database ...
Application Version:  v2.5.0-alpha-5 GitHub version: v2.4.25-64-g1630ae3
Account Type:         personal
Default Drive ID:     af2b5e52bea96776
Default Root ID:      AF2B5E52BEA96776!sea8cc6beffdb43d7976fbc7da445c639
Remaining Free Space: 5.00 GB (5364164153 bytes)
Sync Engine Initialised with new Onedrive API instance
All application operations will be performed in the configured local 'sync_dir' directory: /home/alan/OneDrive
Performing a database consistency and integrity check on locally stored data
Processing DB entries for this Drive ID: af2b5e52bea96776
Processing ~/OneDrive
The directory has not changed
Processing 1.txt
Local file time discrepancy detected: 1.txt
The file content has changed locally and has a newer timestamp, thus needs to be uploaded to OneDrive
Changed local items to upload to OneDrive: 1
Uploading modified file 1.txt ... done.
Scanning the local file system '~/OneDrive' for new data to upload
Fetching /delta response from the OneDrive API for Drive ID: af2b5e52bea96776
Processing API Response Bundle: 1 - Quantity of 'changes|items' in this bundle to process: 2
Finished processing /delta JSON response from the OneDrive API
Processing 1 applicable changes and items received from Microsoft OneDrive
Processing OneDrive JSON item batch [1/1] to ensure consistent local state

Sync with Microsoft OneDrive is complete

Case 2-2

Reading configuration file: /home/alan/.config/onedrive/config
Configuration file successfully loaded
Using 'user' configuration path for application state data: /home/alan/.config/onedrive
Using the following path to store the runtime application log: /var/log/onedrive

The usage of --resync will delete your local 'onedrive' client state, thus no record of your current 'sync status' will exist.
This has the potential to overwrite local versions of files with perhaps older versions of documents downloaded from OneDrive, resulting in local data loss.
If in doubt, backup your local data before using --resync

Are you sure you wish to proceed with --resync? [Y/N] y

Deleting the saved application sync status ...
Using IPv4 and IPv6 (if configured) for all network operations
Checking Application Version ...
Attempting to initialise the OneDrive API ...
Configuring Global Azure AD Endpoints
The OneDrive API was initialised successfully
Opening the item database ...
Application Version:  v2.5.0-alpha-5 GitHub version: v2.4.25-64-g1630ae3
Account Type:         personal
Default Drive ID:     af2b5e52bea96776
Default Root ID:      AF2B5E52BEA96776!sea8cc6beffdb43d7976fbc7da445c639
Remaining Free Space: 5.00 GB (5364164153 bytes)
Sync Engine Initialised with new Onedrive API instance
All application operations will be performed in the configured local 'sync_dir' directory: /home/alan/OneDrive
Performing a database consistency and integrity check on locally stored data
Processing DB entries for this Drive ID: af2b5e52bea96776
Processing ~/OneDrive
The directory has not changed
Scanning the local file system '~/OneDrive' for new data to upload
New items to upload to OneDrive: 1
Total New Data to Upload:        33 Bytes
The file we are attemtping to upload as a new file already exists on Microsoft OneDrive: ./1.txt
Skipping uploading this file as moving it to upload as a modified file (online item already exists): ./1.txt
Changed local items to upload to OneDrive: 1
Uploading modified file ./1.txt ... done.
Fetching /delta response from the OneDrive API for Drive ID: af2b5e52bea96776
Processing API Response Bundle: 1 - Quantity of 'changes|items' in this bundle to process: 3
Skipping item - excluded by skip_file config: /./~tmp21_1
Finished processing /delta JSON response from the OneDrive API
Processing 1 applicable changes and items received from Microsoft OneDrive
Processing OneDrive JSON item batch [1/1] to ensure consistent local state

Sync with Microsoft OneDrive is complete

Screenshots

No response

Other Log Information or Details

No response

Additional context

No response

[abraunegg]

@JC-comp
I will look at this in more detail - however, please understand this .. your points are extreme edge cases.

Please note:

• When using --sync or --monitor the files online are the source of truth - thus the online file is the file to trust as correct and accurate and is the source for the local file copy. This is irrespective of if --resync is being used.

  • In the case where the file exists locally (but not in-sync) , and there is a timestamp mismatch, but the file hash is the same, the offending file timestamp should be corrected
  • In the case where the file exists locally (but not in-sync) , and there is a timestamp mismatch, but the file hash is different, the local file should be renamed, and then uploaded as a new file back to OneDrive.
  • In the case where the file exists locally , and the file was previously in sync , if the local file will be replaced by the online version, as the online version is the source of truth.

• When using --local-first (irrespective of --sync or --monitor or --resync) the files locally are the source of truth - thus the local file is the file to trust as correct (content, timestamp etc) and accurate and is the source for the online file copy.

  • In the case where the file does not exist online, the file is uploaded
  • In the case where the file exists (posix match) the online file is replaced. The older item can be retrieved online via file version history function online.

For all your extreme cases above, please replicate with the Microsoft OneDrive on a Windows platform without using files-on-demand and report back as to what the Windows client is doing.

Observations

Case 1-1 & Case 1-2

• 1.txt is an empty file and in-sync, thus, the online copy, because of the mode of operation being used (--sync) the online copy is what is being brought down. If --monitor was being used, the moment the local file was edited, this would have been uploaded.
• This is not a bug but an extreme edge case
• Replicate with the Microsoft OneDrive Windows Client and report back

Case 2-1 & Case 2-2

• 1.txt is an empty file and in-sync, and because --local-first is being used, the local file (the current empty file) is the source of truth.
• The local file is modified, then the online
• The local file is uploaded, as the local file is the source of truth
• This is not a bug but an extreme edge case
• Replicate with the Microsoft OneDrive Windows Client and report back

Please can you replicate your cases using the Microsoft OneDrive Client on Windows and report back

[JC-comp]

@abraunegg

1. I believe the cases I mentioned here are not rare extreme edge cases; they can easily occur when changes are made from both sides, even in monitor mode.
2. Currently, the strategy here always selects one side to replace the other side in case of conflict. If this behavior is intentional and not a bug, it highlights the necessity for a timestamp-based first mode, which preserves the latest version across all sites, as initially requested in the rejected Feature Request: Timestamp first mode #2617.
3. The Microsoft OneDrive Client on Windows operates only in singular mode. Microsoft's conflict resolution strategy dictates that the online version will always be preserved, while any conflicting local versions will be safely backed up with an appended host name postfix. So in all cases, the outcome results in two files: the online modification in "1.txt" and the local modification in "1-hostname.txt".

[abraunegg]

@JC-comp
As per above - please provide evidence replicating your issue using Microsoft Windows - not theorising what may or may not happen.

Screen shots and visualisations of what is occurring when using Microsoft Windows & Windows OneDrive client please - not text based theory.

[JC-comp]

@abraunegg
When both the local and remote files are modified simultaneously with the OneDrive app running, the final result remains consistent regardless of which side saves first. Microsoft ensures a local backup is created and preserves the online version.

[abraunegg]

@JC-comp
Use Case 1-1 and Case 1-2 - not a bug

Use Case 2-1 and Case 2-2 resolved with fecec1b

Additionally, please read:

• https://github.com/abraunegg/onedrive/blob/onedrive-v2.5.0-alpha-5/docs/client-architecture.md
• https://github.com/abraunegg/onedrive/blob/onedrive-v2.5.0-alpha-5/docs/client-architecture.md#file-conflict-handling---default-operational-modes
• https://github.com/abraunegg/onedrive/blob/onedrive-v2.5.0-alpha-5/docs/client-architecture.md#file-conflict-handling---local-first-operational-mode