Skip to content

rclone verbose logs cannot be parsed by NPBackup #132

@GuitarBilly

Description

@GuitarBilly

npbackup-cli
W10 enterprise 64bit
confirmed on v3.0.0-rc14 and v3.0.0-rc15
This backup to OneDrive worked fine on rc12 and rc13.

I have set "Environment variables" RCLONE_VERBOSE to 2 to print extra information.
I have set "Additional backup only parameters" to --dry-run -o rclone.args="serve restic --stdio --b2-hard-delete --user-agent ISV|rclone.org|rclone/v1.65.2" in an attempt to get the --user-agent passed onto rclone.

steps to reproduce and terminal log:

C:\Users\guitarbilly\Downloads\npbackup-cli>npbackup-cli.exe -c ..\NPBACKUP_COMMON\npbackup.conf --repo-name OneDrive_via_rclone --backup
2025-02-20 22:14:10,166 :: INFO :: npbackup 3.0.0-rc15-windows-cli-x64-public-3.13-c 2025021701 - Copyright (C) 2022-2025 NetInvent running as AD\guitarbilly
2025-02-20 22:14:10,213 :: INFO :: Loaded config AA29C37A in C:\Users\guitarbilly\Downloads\npbackup-cli\..\NPBACKUP_COMMON\npbackup.conf
2025-02-20 22:14:10,219 :: INFO :: No extra information for repos x300server_laptop found
2025-02-20 22:14:10,332 :: INFO :: Searching for a backup newer than 1 day, 0:00:00 ago
2025-02-20 22:14:36,836 :: INFO :: Snapshots listed successfully
2025-02-20 22:14:36,838 :: ERROR :: Runner: Function has_recent_snapshot failed with: list indices must be integers or slices, not str
2025-02-20 22:14:36,840 :: ERROR :: Trace:
Traceback (most recent call last):
  File "C:\Users\guitarbilly\DOWNLO~1\NP14FC~1\npbackup\core\runner.py", line 669, in wrapper
  File "C:\Users\guitarbilly\DOWNLO~1\NP14FC~1\npbackup\core\runner.py", line 442, in wrapper
  File "C:\Users\guitarbilly\DOWNLO~1\NP14FC~1\npbackup\core\runner.py", line 653, in wrapper
  File "C:\Users\guitarbilly\DOWNLO~1\NP14FC~1\npbackup\core\runner.py", line 591, in wrapper
  File "C:\Users\guitarbilly\DOWNLO~1\NP14FC~1\npbackup\core\runner.py", line 512, in wrapper
  File "C:\Users\guitarbilly\DOWNLO~1\NP14FC~1\npbackup\core\runner.py", line 613, in wrapper
  File "C:\Users\guitarbilly\DOWNLO~1\NP14FC~1\npbackup\core\runner.py", line 1095, in has_recent_snapshot
  File "C:\Users\guitarbilly\DOWNLO~1\NP14FC~1\npbackup\restic_wrapper\__init__.py", line 1407, in has_recent_snapshot
  File "C:\Users\guitarbilly\DOWNLO~1\NP14FC~1\npbackup\restic_wrapper\__init__.py", line 1363, in _has_recent_snapshot
TypeError: list indices must be integers or slices, not str
2025-02-20 22:14:36,843 :: ERROR :: Backend finished with errors.
2025-02-20 22:14:36,843 :: WARNING :: Cannot get exec time from environment
2025-02-20 22:14:36,881 :: INFO :: Metrics pushed successfully.
2025-02-20 22:14:36,884 :: ERROR :: Runner: Function backup failed with: cannot unpack non-iterable bool object
2025-02-20 22:14:36,884 :: ERROR :: Trace:
Traceback (most recent call last):
  File "C:\Users\guitarbilly\DOWNLO~1\NP14FC~1\npbackup\core\runner.py", line 669, in wrapper
  File "C:\Users\guitarbilly\DOWNLO~1\NP14FC~1\npbackup\core\runner.py", line 442, in wrapper
  File "C:\Users\guitarbilly\DOWNLO~1\NP14FC~1\npbackup\core\runner.py", line 645, in wrapper
  File "C:\Users\guitarbilly\DOWNLO~1\NP14FC~1\npbackup\core\runner.py", line 591, in wrapper
  File "C:\Users\guitarbilly\DOWNLO~1\NP14FC~1\npbackup\core\runner.py", line 512, in wrapper
  File "C:\Users\guitarbilly\DOWNLO~1\NP14FC~1\npbackup\core\runner.py", line 613, in wrapper
  File "C:\Users\guitarbilly\DOWNLO~1\NP14FC~1\npbackup\core\runner.py", line 1261, in backup
TypeError: cannot unpack non-iterable bool object
2025-02-20 22:14:36,887 :: ERROR :: Cannot decode JSON from restic data: the JSON object must be str, bytes or bytearray, not bool
2025-02-20 22:14:36,887 :: ERROR :: Cannot find processed bytes: 'total_bytes_processed'
2025-02-20 22:14:36,888 :: ERROR :: Backend finished with errors.
2025-02-20 22:14:36,888 :: WARNING :: Cannot get exec time from environment
2025-02-20 22:14:36,983 :: INFO :: Metrics pushed successfully.
2025-02-20 22:14:36,984 :: ERROR :: Operation finished
2025-02-20 22:14:37,005 :: INFO :: ExecTime = 0:00:26.845940, finished, state is: errors.

C:\Users\guitarbilly\Downloads\npbackup-cli>

expected result:
npbackup should not crash but handle it gracefully with an error or warning to the user.

Note that with RCLONE_VERBOSE set to 2 the npbackup-gui hangs on listing snapshots for this OneDrive repo since rc14, hence my trial with npbackup-cli, here are the npbackup-gui messages in the GUI after loading the snapshots:
Last messages

:"d7b4e620"},{"time":"2025-02-20T11:59:51.622411+01:00","parent":"cc8c5b338923ee2dfadff7a63346d5d450f25db4a5a75952b2c66ab053755b58","tree":"9edaa6874f9dbcf82df0cd0aeb2bfa97925aee8f382ffc4fcde2836535d072c6","paths":["C:\\H"],"hostname":"MYLAPTOP","username":"AD\\guitarbilly","tags":["C_H"],"program_version":"restic 0.16.4","id":"34248f08f2093255a7ede7d8335851d269f9440ead3dad7b98e4b81ce410d516","short_id":"34248f08"},{"time":"2025-02-20T12:00:31.0987431+01:00","parent":"d7b4e6201cd5e34b3eb239050ded086f7a04437bd4a448749723347bccfc27ef","tree":"eadf8dcf458e678886274f9fc61e081a4a643db7dbf278460a6cd256881b466a","paths":["C:\\Program Files\\totalcmd\\Wciconex.dll","C:\\Program Files\\totalcmd\\plugins","C:\\Program Files\\totalcmd\\wciconex.inc","C:\\Users\\guitarbilly\\AppData\\Roaming\\GHISLER"],"hostname":"NLNIJ01NB0507","username":"AD\\guitarbilly","tags":["TC"],"program_version":"restic 0.16.4","id":"999e57510cb48cab1fc83abba1029a8e8c511b0a6609197ddcbe6df8de4fcbfe","short_id":"999e5751"}]
Snapshots listed successfully
Runner took 27.58145 seconds for snapshots

Error messages

rclone: 2025/02/20 22:30:46 DEBUG : rclone: Version "v1.65.2" starting with parameters ["rclone" "serve" "restic" "--stdio" "--b2-hard-delete" "onedrive:MyPreciousData\\abc"]
rclone: 2025/02/20 22:30:46 DEBUG : Creating backend with remote "onedrive:MyPreciousData\\abc"

rclone: 2025/02/20 22:30:46 DEBUG : rclone: Version "v1.65.2" starting with parameters ["rclone" "serve" "restic" "--stdio" "--b2-hard-delete" "onedrive:MyPreciousData\\abc"]
rclone: 2025/02/20 22:30:46 DEBUG : Creating backend with remote "onedrive:MyPreciousData\\abc"
rclone: 2025/02/20 22:30:46 DEBUG : Using config file from "C:\\Users\\guitarbilly\\AppData\\Roaming\\rclone\\rclone.conf"
rclone: 2025/02/20 22:30:47 DEBUG : fs cache: renaming cache item "onedrive:MyPreciousData\\abc" to be canonical "onedrive:MyPreciousData/abc"
rclone: 2025/02/20 22:30:47 DEBUG : file-15691317375884925585: GET request error: object not found
rclone: 2025/02/20 22:30:47 DEBUG : keys: list request

rclone: 2025/02/20 22:30:46 DEBUG : rclone: Version "v1.65.2" starting with parameters ["rclone" "serve" "restic" "--stdio" "--b2-hard-delete" "onedrive:MyPreciousData\\abc"]
rclone: 2025/02/20 22:30:46 DEBUG : Creating backend with remote "onedrive:MyPreciousData\\abc"
rclone: 2025/02/20 22:30:46 DEBUG : Using config file from "C:\\Users\\guitarbilly\\AppData\\Roaming\\rclone\\rclone.conf"
rclone: 2025/02/20 22:30:47 DEBUG : fs cache: renaming cache item "onedrive:MyPreciousData\\abc" to be canonical "onedrive:MyPreciousData/abc"
rclone: 2025/02/20 22:30:47 DEBUG : file-15691317375884925585: GET request error: object not found
rclone: 2025/02/20 22:30:47 DEBUG : keys: list request

rclone: 2025/02/20 22:30:46 DEBUG : rclone: Version "v1.65.2" starting with parameters ["rclone" "serve" "restic" "--stdio" "--b2-hard-delete" "onedrive:MyPreciousData\\abc"]
rclone: 2025/02/20 22:30:46 DEBUG : Creating backend with remote "onedrive:MyPreciousData\\abc"
rclone: 2025/02/20 22:30:46 DEBUG : Using config file from "C:\\Users\\guitarbilly\\AppData\\Roaming\\rclone\\rclone.conf"
rclone: 2025/02/20 22:30:47 DEBUG : fs cache: renaming cache item "onedrive:MyPreciousData\\abc" to be canonical "onedrive:MyPreciousData/abc"
rclone: 2025/02/20 22:30:47 DEBUG : file-15691317375884925585: GET request error: object not found
rclone: 2025/02/20 22:30:47 DEBUG : keys: list request

rclone: 2025/02/20 22:30:46 DEBUG : rclone: Version "v1.65.2" starting with parameters ["rclone" "serve" "restic" "--stdio" "--b2-hard-delete" "onedrive:MyPreciousData\\abc"]
rclone: 2025/02/20 22:30:46 DEBUG : Creating backend with remote "onedrive:MyPreciousData\\abc"
rclone: 2025/02/20 22:30:46 DEBUG : Using config file from "C:\\Users\\guitarbilly\\AppData\\Roaming\\rclone\\rclone.conf"
rclone: 2025/02/20 22:30:47 DEBUG : fs cache: renaming cache item "onedrive:MyPreciousData\\abc" to be canonical "onedrive:MyPreciousData/abc"
rclone: 2025/02/20 22:30:47 DEBUG : file-15691317375884925585: GET request error: object not found
rclone: 2025/02/20 22:30:47 DEBUG : keys: list request
rclone: 2025/02/20 22:30:51 DEBUG : locks: list request

rclone: 2025/02/20 22:30:46 DEBUG : rclone: Version "v1.65.2" starting with parameters ["rclone" "serve" "restic" "--stdio" "--b2-hard-delete" "onedrive:MyPreciousData\\abc"]
rclone: 2025/02/20 22:30:46 DEBUG : Creating backend with remote "onedrive:MyPreciousData\\abc"
rclone: 2025/02/20 22:30:46 DEBUG : Using config file from "C:\\Users\\guitarbilly\\AppData\\Roaming\\rclone\\rclone.conf"
rclone: 2025/02/20 22:30:47 DEBUG : fs cache: renaming cache item "onedrive:MyPreciousData\\abc" to be canonical "onedrive:MyPreciousData/abc"
rclone: 2025/02/20 22:30:47 DEBUG : file-15691317375884925585: GET request error: object not found
rclone: 2025/02/20 22:30:47 DEBUG : keys: list request
rclone: 2025/02/20 22:30:51 DEBUG : locks: list request
rclone: 2025/02/20 22:30:51 DEBUG : locks/55c6681445b7a516ccd2baab769ce2a9b70e5a7524a3e1d6ba23a8463dff66c9: Starting multipart upload
rclone: 2025/02/20 22:30:51 DEBUG : locks/55c6681445b7a516ccd2baab769ce2a9b70e5a7524a3e1d6ba23a8463dff66c9: Uploading segment 0/160 size 160

rclone: 2025/02/20 22:30:46 DEBUG : rclone: Version "v1.65.2" starting with parameters ["rclone" "serve" "restic" "--stdio" "--b2-hard-delete" "onedrive:MyPreciousData\\abc"]
rclone: 2025/02/20 22:30:46 DEBUG : Creating backend with remote "onedrive:MyPreciousData\\abc"
rclone: 2025/02/20 22:30:46 DEBUG : Using config file from "C:\\Users\\guitarbilly\\AppData\\Roaming\\rclone\\rclone.conf"
rclone: 2025/02/20 22:30:47 DEBUG : fs cache: renaming cache item "onedrive:MyPreciousData\\abc" to be canonical "onedrive:MyPreciousData/abc"
rclone: 2025/02/20 22:30:47 DEBUG : file-15691317375884925585: GET request error: object not found
rclone: 2025/02/20 22:30:47 DEBUG : keys: list request
rclone: 2025/02/20 22:30:51 DEBUG : locks: list request
rclone: 2025/02/20 22:30:51 DEBUG : locks/55c6681445b7a516ccd2baab769ce2a9b70e5a7524a3e1d6ba23a8463dff66c9: Starting multipart upload
rclone: 2025/02/20 22:30:51 DEBUG : locks/55c6681445b7a516ccd2baab769ce2a9b70e5a7524a3e1d6ba23a8463dff66c9: Uploading segment 0/160 size 160
rclone: 2025/02/20 22:30:52 DEBUG : locks: list request
rclone: 2025/02/20 22:30:52 DEBUG : snapshots: list request

rclone: 2025/02/20 22:30:46 DEBUG : rclone: Version "v1.65.2" starting with parameters ["rclone" "serve" "restic" "--stdio" "--b2-hard-delete" "onedrive:MyPreciousData\\abc"]
rclone: 2025/02/20 22:30:46 DEBUG : Creating backend with remote "onedrive:MyPreciousData\\abc"
rclone: 2025/02/20 22:30:46 DEBUG : Using config file from "C:\\Users\\guitarbilly\\AppData\\Roaming\\rclone\\rclone.conf"
rclone: 2025/02/20 22:30:47 DEBUG : fs cache: renaming cache item "onedrive:MyPreciousData\\abc" to be canonical "onedrive:MyPreciousData/abc"
rclone: 2025/02/20 22:30:47 DEBUG : file-15691317375884925585: GET request error: object not found
rclone: 2025/02/20 22:30:47 DEBUG : keys: list request
rclone: 2025/02/20 22:30:51 DEBUG : locks: list request
rclone: 2025/02/20 22:30:51 DEBUG : locks/55c6681445b7a516ccd2baab769ce2a9b70e5a7524a3e1d6ba23a8463dff66c9: Starting multipart upload
rclone: 2025/02/20 22:30:51 DEBUG : locks/55c6681445b7a516ccd2baab769ce2a9b70e5a7524a3e1d6ba23a8463dff66c9: Uploading segment 0/160 size 160
rclone: 2025/02/20 22:30:52 DEBUG : locks: list request
rclone: 2025/02/20 22:30:52 DEBUG : snapshots: list request

rclone: 2025/02/20 22:30:46 DEBUG : rclone: Version "v1.65.2" starting with parameters ["rclone" "serve" "restic" "--stdio" "--b2-hard-delete" "onedrive:MyPreciousData\\abc"]
rclone: 2025/02/20 22:30:46 DEBUG : Creating backend with remote "onedrive:MyPreciousData\\abc"
rclone: 2025/02/20 22:30:46 DEBUG : Using config file from "C:\\Users\\guitarbilly\\AppData\\Roaming\\rclone\\rclone.conf"
rclone: 2025/02/20 22:30:47 DEBUG : fs cache: renaming cache item "onedrive:MyPreciousData\\abc" to be canonical "onedrive:MyPreciousData/abc"
rclone: 2025/02/20 22:30:47 DEBUG : file-15691317375884925585: GET request error: object not found
rclone: 2025/02/20 22:30:47 DEBUG : keys: list request
rclone: 2025/02/20 22:30:51 DEBUG : locks: list request
rclone: 2025/02/20 22:30:51 DEBUG : locks/55c6681445b7a516ccd2baab769ce2a9b70e5a7524a3e1d6ba23a8463dff66c9: Starting multipart upload
rclone: 2025/02/20 22:30:51 DEBUG : locks/55c6681445b7a516ccd2baab769ce2a9b70e5a7524a3e1d6ba23a8463dff66c9: Uploading segment 0/160 size 160
rclone: 2025/02/20 22:30:52 DEBUG : locks: list request
rclone: 2025/02/20 22:30:52 DEBUG : snapshots: list request

rclone: 2025/02/20 22:30:46 DEBUG : rclone: Version "v1.65.2" starting with parameters ["rclone" "serve" "restic" "--stdio" "--b2-hard-delete" "onedrive:MyPreciousData\\abc"]
rclone: 2025/02/20 22:30:46 DEBUG : Creating backend with remote "onedrive:MyPreciousData\\abc"
rclone: 2025/02/20 22:30:46 DEBUG : Using config file from "C:\\Users\\guitarbilly\\AppData\\Roaming\\rclone\\rclone.conf"
rclone: 2025/02/20 22:30:47 DEBUG : fs cache: renaming cache item "onedrive:MyPreciousData\\abc" to be canonical "onedrive:MyPreciousData/abc"
rclone: 2025/02/20 22:30:47 DEBUG : file-15691317375884925585: GET request error: object not found
rclone: 2025/02/20 22:30:47 DEBUG : keys: list request
rclone: 2025/02/20 22:30:51 DEBUG : locks: list request
rclone: 2025/02/20 22:30:51 DEBUG : locks/55c6681445b7a516ccd2baab769ce2a9b70e5a7524a3e1d6ba23a8463dff66c9: Starting multipart upload
rclone: 2025/02/20 22:30:51 DEBUG : locks/55c6681445b7a516ccd2baab769ce2a9b70e5a7524a3e1d6ba23a8463dff66c9: Uploading segment 0/160 size 160
rclone: 2025/02/20 22:30:52 DEBUG : locks: list request
rclone: 2025/02/20 22:30:52 DEBUG : snapshots: list request

rclone: 2025/02/20 22:30:46 DEBUG : rclone: Version "v1.65.2" starting with parameters ["rclone" "serve" "restic" "--stdio" "--b2-hard-delete" "onedrive:MyPreciousData\\abc"]
rclone: 2025/02/20 22:30:46 DEBUG : Creating backend with remote "onedrive:MyPreciousData\\abc"
rclone: 2025/02/20 22:30:46 DEBUG : Using config file from "C:\\Users\\guitarbilly\\AppData\\Roaming\\rclone\\rclone.conf"
rclone: 2025/02/20 22:30:47 DEBUG : fs cache: renaming cache item "onedrive:MyPreciousData\\abc" to be canonical "onedrive:MyPreciousData/abc"
rclone: 2025/02/20 22:30:47 DEBUG : file-15691317375884925585: GET request error: object not found
rclone: 2025/02/20 22:30:47 DEBUG : keys: list request
rclone: 2025/02/20 22:30:51 DEBUG : locks: list request
rclone: 2025/02/20 22:30:51 DEBUG : locks/55c6681445b7a516ccd2baab769ce2a9b70e5a7524a3e1d6ba23a8463dff66c9: Starting multipart upload
rclone: 2025/02/20 22:30:51 DEBUG : locks/55c6681445b7a516ccd2baab769ce2a9b70e5a7524a3e1d6ba23a8463dff66c9: Uploading segment 0/160 size 160
rclone: 2025/02/20 22:30:52 DEBUG : locks: list request
rclone: 2025/02/20 22:30:52 DEBUG : snapshots: list request

EDIT : here is the npbackup-gui termnial log:

C:\Users\guitarbilly>Downloads\npbackup-gui\npbackup-gui.exe -c Downloads\NPBACKUP_COMMON\npbackup.conf --repo-name OneDrive_via_rclone
2025-02-20 22:29:48,671 :: INFO :: GUI: npbackup 3.0.0-rc15-windows-gui-x64-public-3.13-c 2025021701 - Copyright (C) 2022-2025 NetInvent running as AD\guitarbilly
2025-02-20 22:29:48,728 :: INFO :: Loaded config AA29C37A in C:\Users\guitarbilly\Downloads\NPBACKUP_COMMON\npbackup.conf
2025-02-20 22:29:48,734 :: INFO :: No extra information for repos x300server_MYLAPTOP found
2025-02-20 22:29:49,677 :: INFO :: Listing snapshots of repo OneDrive_via_rclone
2025-02-20 22:30:15,125 :: INFO :: Snapshots listed successfully
2025-02-20 22:30:15,125 :: INFO :: Runner took 25.448921 seconds for snapshots
2025-02-20 22:30:28,720 :: INFO :: Listing snapshots of repo OneDrive_via_rclone
2025-02-20 22:30:56,298 :: INFO :: Snapshots listed successfully
2025-02-20 22:30:56,300 :: INFO :: Runner took 27.58145 seconds for snapshots
2025-02-20 22:30:56,343 :: INFO :: Metrics pushed successfully.
2025-02-20 22:41:21,488 :: CRITICAL :: GUI Execution error list indices must be integers or slices, not str
2025-02-20 22:41:21,490 :: INFO :: Trace:
Traceback (most recent call last):
  File "C:\Users\guitarbilly\DOWNLO~1\NPAA0E~1\npbackup\gui\__main__.py", line 1238, in main_gui
  File "C:\Users\guitarbilly\DOWNLO~1\NPAA0E~1\npbackup\gui\__main__.py", line 1107, in _main_gui
  File "C:\Users\guitarbilly\DOWNLO~1\NPAA0E~1\npbackup\gui\__main__.py", line 659, in get_gui_data
  File "C:\Users\guitarbilly\DOWNLO~1\NPAA0E~1\npbackup\restic_wrapper\__init__.py", line 1363, in _has_recent_snapshot
TypeError: list indices must be integers or slices, not str
2025-02-20 22:41:21,538 :: INFO :: ExecTime = 0:11:32.869565, finished, state is: critical.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions