Error during checkpoint removal

[aminhashemi1994]

Hi,

I am using virtnbdbackup version: 1.9.48

Something happened to my VM and I was getting this error:

[2024-06-16 16:11:31] INFO root checkpoint - redefine [MainThread]: Loading checkpoint list from: [/opt/kvm_pool/backup/20240520-100050013236165-101_networkcontroller-1-prod/monthly/checkpoints]
[2024-06-16 16:11:31] INFO root checkpoint - create [MainThread]: Checkpoint handling.
[2024-06-16 16:11:31] ERROR root checkpoint - delete [MainThread]: Error during checkpoint removal: [internal error: bitmap 'virtnbdbackup.0' not found in backing chain of 'vda']
[2024-06-16 16:11:31] ERROR root virtnbdbackup - main [MainThread]: Failed to remove checkpoint.
Failed to Create monthly Backup for 20240520-100050013236165-101_networkcontroller-1-prod
monthly backup Failed for 20240520-100050013236165-101_networkcontroller-1-prod on Host: 172.30.70.1
Creating full weekly backup for 20240520-102633875500887-20_cando-neobank-13-dev-gateway.part.psg.network
/usr/local/bin/virtnbdbackup

After some research, I found that I could use virsh checkpoint-list to get the checkpoints of my VM. The issue was that virtnbd always failed to create a backup when this error occurred and was not able to create any new backup. No matter if I removed all the backups or not, this error always stopped me from creating new backups.

I was able to fix this issue by using this command without needing to shut down my VM:

i have found "virtnbdbackup.0" in the error log and in the "virsh checkpoint-list"

virsh checkpoint-delete --metadata <vm_name> virtnbdbackup.0

[abbbi]

at some point it seems your checkpoint chain was broken or got inconsistent. Search for old discussions and issues for this issue. Most of the time this happens if vm is transferred to another host (see readme for transient evironments) or something else "strange" has happened like a third party utility removing checkpoints or something. The complete history of backups is required to tell at what point the problem started.

"Something happened to my VM" is the cause for this, and something only can be analyzed with both libvirt/qemu logs and/or complete history of backup loggings.

[aminhashemi1994]

i have managed to recretate this error by shuting down the vm while it is trying to create backup.
but after that i get this error everytime, until i manually run the below command :

virsh checkpoint-delete --metadata <vm_name> virtnbdbackup.0

i can see the checkpoint has been added to my vm with this command.

virsh checkpoint-list <VM_name>

but i get the error i said.
i have created a bash script that can create schedual remote backup for me. but i have faced with this error. so if you can add a fix i can use that, else i need to handle this error in my bash sript .

this is the history of my commands and outputs, from virsh everything looks fine i think:

qemu-img info --backing-chain debian-20240520-100050013236165-101_networkcontroller-1-prod.img
image: debian-20240520-100050013236165-101_networkcontroller-1-prod.img
file format: qcow2
virtual size: 15 GiB (16106127360 bytes)
disk size: 13.4 GiB
cluster_size: 65536
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
    extended l2: false
root@hv212:/opt/kvm_pool# virsh domblklist 20240520-100050013236165-101_networkcontroller-1-prod
 Target   Source
------------------------------------------------------------------------------------------
 vda      /opt/kvm_pool/debian-20240520-100050013236165-101_networkcontroller-1-prod.img
 vdb      /opt/kvm_pool/kk2-20240520-100050013236165-101_networkcontroller-1-prod.qcow2

root@hv212:/opt/kvm_pool# virsh checkpoint-delete --metadata 20240520-100050013236165-101_networkcontroller-1-prod virtnbdbackup.0

[abbbi]

i have managed to recretate this error by shuting down the vm while it is trying to create backup.

i see, in this situation all kind of strange things can happen depending at what point the vitual machine is shutdown during the backup. If the shutdown happens during active data-transfer the backup will fail and subsequent backups with different modes like inc/diff will also fail as it detects a partial transferred backup in the target directory.

From your output its seen that the checkpoint was created on libvirt, but was not written to the qcow image, the reason is unclear for me but i really guess its the shutdown that came in between here. Now if virtnbdbackup attempts to re-create or delete the checkpoint it fails (because it doesnt use the --metadata option). It could retry with --metadata BUT still its a corner case here ..

Other than that, there is no logic in virtnbdbackup that checks if virtual machine is altering its state during the backup, also, i cannot block shutdown operations on libvirt level or something. I dont think i can implement something REAL sophisticated to handle these situations in all cases. You should guarantee your virtual machines are up and running while creating backups or use the option -S, then virtnbdbackup will at least start the virtual machine in non-op mode in case it is stopped before the backup.

Option -S was added in version 2.3.

That should be the current state (with the LATEST version 2.10 used):

1. vm is shutdown before the backup starts: virtnbdbackup will create only copy backups and no checkpoint is created
2. vm is shutdown before the backup starts and virtnbdbackup is used with option -S: it will start the virtual machine in non-operational mode to be able to create full/inc/diff backups: the checkpoint will be created correctly
3. vm changes operational state in between backup operation: not handled at all and all kind of strange situations can appear