virtnbdbackup and virtnbdrestore: differential backups

[skleijkers]

Hi abbbi,

I opened the issue regarding the differential backup, I experimented a little further with the differential backup and I found a way to make differential backups, although it isn't a nice way.
I first made a full backup (checkpoint virtnbdbackup.0) and secondly a incremental backup (checkpoint virtnbdbackup.1). I now remove the virtnbdbackup.1 checkpoint with "virsh checkpoint-delete X virtnbdbackup,1". Furthermore I had to remove the checkpoint virtnbdbackup.1 from the checkpoint directory, else virtnbdbackup complaints it can't redefine the .1 checkpoint. Now I can make another incremental backup with "virtnbdbackup -d X -l inc -o Y -c virtnbdbackup.0" where Y is the dataset directory used for the full and first incremental. Virtnbdbackup now creates a .2 checkpoint (it iterates from the .cpl file) and the parent becomes .0, because libvirt doesn't have another checkpoint.
I have tested it with adding and deleting files in the VM and result is as expected, I used virtnbdrestore to recreate the disk files and mounted them after connecting with qemu-nbd.

I have three problems with this method:

1. virtnbdbackup tries to redefine the deleted checkpoint if it finds a definition of the checkpoint in the checkpoint directory. If a parent checkpoint is defined it should redefine starting from that parent up.
2. virtnbdrestore doesn't deal with the child parent relation, it always builds the restore from the first checkpoint to the checkpoint that's defined with --until. It should also check the parent and only restore the branch from parent to parent. This is a bit more complicated, because the restore works from the other side, first the full and then the children, so by defining the checkpoint it should first check how the branch connects to the last parent, the full backup.
3. You have to delete the child checkpoints if libvirt doesn't use the parent option.

So it's possible to make differential backups, although as long as libvirt doesn't have a way to use the parent option it won't be a very nice way.

Kind regards,
Stefan

[abbbi]

Quick shot implementation for backup in this branch: https://github.com/abbbi/virtnbdbackup/tree/diffbackup
Not tested for data integrity.

[skleijkers]

Thanks! I'll give it a try tomorrow!

Well I think mixing incrementals and differentials shall be a problem, because after deleting a checkpoint, the bitmap will also be removed and there isn't a method for saving and restoring the bitmap. So as long as there isn't a way to redefine the parent of a checkpoint mixing incrementals and differentials isn't possible.

I found this information on https://libvirt.org/formatcheckpoint.html:
"Attributes of libvirt checkpoints are stored as child elements of the domaincheckpoint element. At checkpoint creation time, normally only the name, description, and disks elements are settable. The rest of the fields are ignored on creation and will be filled in by libvirt in for informational purposes by virDomainCheckpointGetXMLDesc(). However, when redefining a checkpoint, with the VIR_DOMAIN_CHECKPOINT_CREATE_REDEFINE flag of virDomainCheckpointCreateXML(), all of the XML fields described here are relevant on input, even the fields that are normally described as readonly for output."

This is also what you wrote about the parent attribute. So in QEMU 5.6.0 it will be possible to redefine a checkpoint with all attributes. This makes it possible to rebuild the tree from the information in the checkpoint directory. But I don't know if and how the bitmaps will be recreated, so maybe this still doesn't work.

The other thing I don't get is why they have the option --tree in virsh checkpoint-list, you would assume that it would be possible to have different branches from checkpoints. Maybe it's something they are working on. They also keep track of number of children and descendants. But at the moment I see no way how to create siblings...

[abbbi]

Well I think mixing incrementals and differentials shall be a problem, because after deleting a checkpoint, the bitmap will also be removed and there isn't a method for saving and restoring the bitmap. So as long as there isn't a way to redefine the parent of a checkpoint mixing incrementals and differentials isn't possible.

might well be, but its easy to implement a check and fail backup if the target directory already has either of one existing.
I think what now happens would be that the diff backup would allways just include the differences which happened since
the last incremental backup, from that point on until another incremental backup is issued. Which is Ok, too.

This is also what you wrote about the parent attribute. So in QEMU 5.6.0 it will be possible to redefine a checkpoint with all attributes. This makes it possible to rebuild the tree from the information in the checkpoint directory. But I don't know if and how the bitmaps will be recreated, so maybe this still doesn't work.

the bitmap information is stored in the qcow images. It is possible to remove a checkpoint in libvirt with the --metadata option
and the bitmap is kept in the qcow file. Using the VIR_DOMAIN_CHECKPOINT_CREATE_REDEFINE option its possible to
re-create a checkpoint within libvirt which then points to the bitmap already existant in the qcow file (wihtout that option creating the checkpoint would fail because the bitmap already exists)

The current approach for the diff backup does remove both bitmap and checkpoint after the diff backup, not sure if thats the right approach, but it looks promising:

Quick test:

 1) full backup: ./virtnbdbackup -l full -o /tmp/DIFFTEST -d vm1
 2) destroy vm, guestmount, create 5k file:
     guestmount -d vm1 -m /dev/sda1 /mnt
     dd if=/dev/urandom of=/mnt/diff1 bs=1K count=5
     umount /mnt
     virsh start vm1
  3) diff backup: /virtnbdbackup -l diff -o /tmp/DIFFTEST -d vm1
  4) repeat step 2 with target file "diff2"
  5) diff backup: /virtnbdbackup -l diff -o /tmp/DIFFTEST -d vm1
  6) restore and mount image, it shows both files recovered:
ls -1 /mnt/
datafile
diff1
diff2

I think it might be good to consult the libvirt guys (namely @jnsnow or eric blake in the #virt channel) for details on this topic.

[skleijkers]

Hi,

I tried the 0.44 and the 0.45 from the diffbackup branch and I have the following problems:

• It just makes a full backup, no differential
• It removes the checkpoint, also from the cpt file. Now it overwrites the last diff backup, because it uses the same name (virtnbdbackup.1)

The relevant logs:

FULL:
[2022-02-21 11:23:50] INFO common - printVersion [MainThread]: Version: 0.45 Arguments: /usr/bin/virtnbdbackup -d X -l full -o /X/backupset.99
[2022-02-21 11:23:50] INFO virtnbdbackup - main [MainThread]: Backup level: [full]
[2022-02-21 11:23:50] INFO virtnbdbackup - main [MainThread]: Domain has 1 disks attached which support changed block tracking.
[2022-02-21 11:23:50] INFO virtnbdbackup - main [MainThread]: Concurrent backup processes: [1]
[2022-02-21 11:23:50] INFO libvirthelper - redefineCheckpoints [MainThread]: Loading checkpoint list from: [/X/backupset.99/checkpoints]
[2022-02-21 11:23:50] INFO virtnbdbackup - main [MainThread]: Checkpoint handling
[2022-02-21 11:23:50] INFO virtnbdbackup - main [MainThread]: Using checkpoint name: virtnbdbackup.0
[2022-02-21 11:23:50] INFO virtnbdbackup - main [MainThread]: Temporary scratch file target directory: /var/tmp
[2022-02-21 11:23:50] INFO virtnbdbackup - main [MainThread]: NDB Endpoint socket: /var/tmp/virtnbdbackup.47091
[2022-02-21 11:23:50] INFO virtnbdbackup - main [MainThread]: Starting backup job.
[2022-02-21 11:23:51] INFO libvirthelper - fsFreeze [MainThread]: Freeze filesystems.
[2022-02-21 11:23:51] INFO libvirthelper - fsThaw [MainThread]: Thawed filesystems.
[2022-02-21 11:23:51] INFO virtnbdbackup - main [MainThread]: Started backup job with checkpoint, saving information.
[2022-02-21 11:23:51] INFO libvirthelper - backupCheckpoint [MainThread]: Saving checkpoint config to: /X/backupset.99/checkpoints/virtnbdbackup.0.xml
[2022-02-21 11:23:51] INFO nbdhelper - version [vda]: libnbd version: 1.6.1
[2022-02-21 11:23:51] INFO nbdhelper - waitForServer [vda]: Waiting until NBD server on socket [/var/tmp/virtnbdbackup.47091] is up.
[2022-02-21 11:23:52] INFO nbdhelper - getBlockInfo [vda]: Using Maximum Block size supported by NBD server: [33554432]
[2022-02-21 11:23:52] INFO nbdhelper - waitForServer [vda]: Connection to NBD backend succeeded.
[2022-02-21 11:23:53] INFO virtnbdbackup - backupDisk [vda]: Got 14379 extents to backup.
[2022-02-21 11:23:53] INFO virtnbdbackup - backupDisk [vda]: 68719476736 bytes disk size
[2022-02-21 11:23:53] INFO virtnbdbackup - backupDisk [vda]: 5062221824 bytes of data extents to backup
[2022-02-21 11:23:53] INFO virtnbdbackup - backupDisk [vda]: Write data to target file: /X/backupset.99/vda.full.data.partial
[2022-02-21 11:23:53] INFO virtnbdbackup - backupDisk [vda]: Creating thin provisioned stream backup image
[2022-02-21 11:24:25] INFO virtnbdbackup - backupConfig [MainThread]: Saving VM config to: /X/backupset.99/vmconfig.virtnbdbackup.0.xml
[2022-02-21 11:24:25] INFO virtnbdbackup - main [MainThread]: Backup jobs finished, stopping backup task.
[2022-02-21 11:24:25] INFO virtnbdbackup - main [MainThread]: Finished

FIRST DIFF:
[2022-02-21 11:24:39] INFO common - printVersion [MainThread]: Version: 0.45 Arguments: /usr/bin/virtnbdbackup -d X -l diff -o /X/backupset.99
[2022-02-21 11:24:39] INFO virtnbdbackup - main [MainThread]: Backup level: [diff]
[2022-02-21 11:24:39] INFO virtnbdbackup - main [MainThread]: Domain has 1 disks attached which support changed block tracking.
[2022-02-21 11:24:39] INFO virtnbdbackup - main [MainThread]: Concurrent backup processes: [1]
[2022-02-21 11:24:39] INFO libvirthelper - redefineCheckpoints [MainThread]: Loading checkpoint list from [/X/backupset.99/checkpoints]
[2022-02-21 11:24:39] INFO virtnbdbackup - main [MainThread]: Checkpoint handling
[2022-02-21 11:24:39] INFO virtnbdbackup - main [MainThread]: Next checkpoint id: 1
[2022-02-21 11:24:39] INFO virtnbdbackup - main [MainThread]: Parent checkpoint name virtnbdbackup.0
[2022-02-21 11:24:39] INFO virtnbdbackup - main [MainThread]: Using checkpoint name: virtnbdbackup.1
[2022-02-21 11:24:39] INFO virtnbdbackup - main [MainThread]: Temporary scratch file target directory: /var/tmp
[2022-02-21 11:24:39] INFO virtnbdbackup - main [MainThread]: NDB Endpoint socket: /var/tmp/virtnbdbackup.47120
[2022-02-21 11:24:39] INFO virtnbdbackup - main [MainThread]: Starting backup job.
[2022-02-21 11:24:39] INFO libvirthelper - fsFreeze [MainThread]: Freeze filesystems.
[2022-02-21 11:24:39] INFO libvirthelper - fsThaw [MainThread]: Thawed filesystems.
[2022-02-21 11:24:39] INFO nbdhelper - version [vda]: libnbd version: 1.6.1
[2022-02-21 11:24:39] INFO nbdhelper - waitForServer [vda]: Waiting until NBD server on socket [/var/tmp/virtnbdbackup.47120] is up.
[2022-02-21 11:24:40] INFO nbdhelper - getBlockInfo [vda]: Using Maximum Block size supported by NBD server: [33554432]
[2022-02-21 11:24:40] INFO nbdhelper - waitForServer [vda]: Connection to NBD backend succeeded.
[2022-02-21 11:24:41] INFO virtnbdbackup - backupDisk [vda]: Got 14379 extents to backup.
[2022-02-21 11:24:41] INFO virtnbdbackup - backupDisk [vda]: 68719476736 bytes disk size
[2022-02-21 11:24:41] INFO virtnbdbackup - backupDisk [vda]: 5062221824 bytes of data extents to backup
[2022-02-21 11:24:41] INFO virtnbdbackup - backupDisk [vda]: Write data to target file: /X/backupset.99/vda.diff.virtnbdbackup.1.data.partial
[2022-02-21 11:24:41] INFO virtnbdbackup - backupDisk [vda]: Creating thin provisioned stream backup image
[2022-02-21 11:25:13] INFO virtnbdbackup - backupConfig [MainThread]: Saving VM config to: /X/backupset.99/vmconfig.virtnbdbackup.1.xml
[2022-02-21 11:25:13] INFO virtnbdbackup - main [MainThread]: Backup jobs finished, stopping backup task.
[2022-02-21 11:25:13] INFO virtnbdbackup - main [MainThread]: Differencial backup requested, removing created checkpoint
[2022-02-21 11:25:14] INFO virtnbdbackup - main [MainThread]: Finished

SECOND DIFF:
[2022-02-21 11:25:31] INFO common - printVersion [MainThread]: Version: 0.45 Arguments: /usr/bin/virtnbdbackup -d X -l diff -o /X/backupset.99
[2022-02-21 11:25:31] INFO virtnbdbackup - main [MainThread]: Backup level: [diff]
[2022-02-21 11:25:31] INFO virtnbdbackup - main [MainThread]: Domain has 1 disks attached which support changed block tracking.
[2022-02-21 11:25:31] INFO virtnbdbackup - main [MainThread]: Concurrent backup processes: [1]
[2022-02-21 11:25:31] INFO libvirthelper - redefineCheckpoints [MainThread]: Loading checkpoint list from [/X/backupset.99/checkpoints]
[2022-02-21 11:25:31] INFO virtnbdbackup - main [MainThread]: Checkpoint handling
[2022-02-21 11:25:31] INFO virtnbdbackup - main [MainThread]: Next checkpoint id: 1
[2022-02-21 11:25:31] INFO virtnbdbackup - main [MainThread]: Parent checkpoint name virtnbdbackup.0
[2022-02-21 11:25:31] INFO virtnbdbackup - main [MainThread]: Using checkpoint name: virtnbdbackup.1
[2022-02-21 11:25:31] INFO virtnbdbackup - main [MainThread]: Temporary scratch file target directory: /var/tmp
[2022-02-21 11:25:31] INFO virtnbdbackup - main [MainThread]: NDB Endpoint socket: /var/tmp/virtnbdbackup.47152
[2022-02-21 11:25:31] INFO virtnbdbackup - main [MainThread]: Starting backup job.
[2022-02-21 11:25:31] INFO libvirthelper - fsFreeze [MainThread]: Freeze filesystems.
[2022-02-21 11:25:32] INFO libvirthelper - fsThaw [MainThread]: Thawed filesystems.
[2022-02-21 11:25:32] INFO nbdhelper - version [vda]: libnbd version: 1.6.1
[2022-02-21 11:25:32] INFO nbdhelper - waitForServer [vda]: Waiting until NBD server on socket [/var/tmp/virtnbdbackup.47152] is up.
[2022-02-21 11:25:33] INFO nbdhelper - getBlockInfo [vda]: Using Maximum Block size supported by NBD server: [33554432]
[2022-02-21 11:25:33] INFO nbdhelper - waitForServer [vda]: Connection to NBD backend succeeded.
[2022-02-21 11:25:34] INFO virtnbdbackup - backupDisk [vda]: Got 14379 extents to backup.
[2022-02-21 11:25:34] INFO virtnbdbackup - backupDisk [vda]: 68719476736 bytes disk size
[2022-02-21 11:25:34] INFO virtnbdbackup - backupDisk [vda]: 5062225920 bytes of data extents to backup
[2022-02-21 11:25:34] INFO virtnbdbackup - backupDisk [vda]: Write data to target file: /X/backupset.99/vda.diff.virtnbdbackup.1.data.partial
[2022-02-21 11:25:34] INFO virtnbdbackup - backupDisk [vda]: Creating thin provisioned stream backup image
[2022-02-21 11:26:07] INFO virtnbdbackup - backupConfig [MainThread]: Saving VM config to: /X/backupset.99/vmconfig.virtnbdbackup.1.xml
[2022-02-21 11:26:07] INFO virtnbdbackup - main [MainThread]: Backup jobs finished, stopping backup task.
[2022-02-21 11:26:07] INFO virtnbdbackup - main [MainThread]: Differencial backup requested, removing created checkpoint
[2022-02-21 11:26:07] INFO virtnbdbackup - main [MainThread]: Finished

Kind regards,
Stefan

[abbbi]

• It just makes a full backup, no differential

argh, i missed the section where it starts the correct meta context, now the metacontext for the NDB Service
should be set to the right checkpoint and no full beeing created, re-check with latest commit.

i think some more changes are required. I think for differencial it is not even required to create a checkpoint,
the defined backup operation XML should just include the last checkpoint that is known to be existant and
then the delta should be saved.

Its now saving only the delta, but after restore i get a corrupted filesystem, so theres still something
wrong here.

• It removes the checkpoint, also from the cpt file. Now it overwrites the last diff backup, because it uses the same name (virtnbdbackup.1)

yes, that would be desired behavior.

[skleijkers]

Hi,

Thanks! It now makes a differential/incremental backup, but I have the same problem, after restoring the backupset the qcow2 file is corrupt...

Regarding the overwrite, I don't know if that's the desired behavior. At the moment I'm making a full backup at Sunday (once a week) and the rest of the week I'm making an incremental. This works fine, but, of course, when I need something from the last day (Saturday), I need to recreate the restore from the full and all incrementals, if one incremental fails, the chain is broken and I can't restore after the failed incremental. That's one of the reasons I would like to have differentials, so I have a differential per day against the full week backup.
Furthermore those are all root disks, so de difference in space between the incrementals and differentials isn't that much. Therefor I prefer the differentials.
Well I could, of course, rename the differential to the different days in order to safe the differential per day, but, at the moment, I have no way to recreate the differential from the given day with only that differential and the full.

Kind regards,
Stefan

[abbbi]

Thanks! It now makes a differential/incremental backup, but I have the same problem, after restoring the backupset the qcow2 file is corrupt...

yes, i dont know why that is happening... the differencial backup does not have to create an checkpoint, it just needs to tell the domain backup XML what the last checkpoint is for the "incremental" backup:

<domainbackup mode="pull">
  <server transport="unix" socket="/var/tmp/virtnbdbackup.1491322" />
  <disks>
    <disk name="vdb">
      <scratch file="/var/tmp/backup.7109O.vdb" />
    </disk>
  </disks>
  <incremental>virtnbdbackup.0</incremental>
</domainbackup>

so the incremental flag includes the checkpoint that is used as base:

incremental
An optional element giving the name of an existing checkpoint of the domain, which will be used to make this backup an incremental one. In the push model, only changes since the named checkpoint are written to the destination. In the pull model, the NBD server uses the NBD_OPT_SET_META_CONTEXT extension to advertise to the client which portions of the export contain changes since the named checkpoint. If omitted, a full backup is performed.

and that is happening, just like during the incremental backup, the differencial backup sets the right meta context, too:

backupDisk [vdb]: INC/DIFF backup: set context to qemu:dirty-bitmap:backup-vdb

additionally i can see that the data is incrementing for each differencial backup taken, so it should behave like designed:

./virtnbdbackup -d os-alma84_default -l diff -o /tmp/DIFFTEST -i vdb  2>&1 | grep "bytes of data"
[2022-02-21 14:05:17] INFO virtnbdbackup - backupDisk [vdb]: 262144 bytes of data extents to backup
 ./virtnbdbackup -d os-alma84_default -l diff -o /tmp/DIFFTEST -i vdb  2>&1 | grep "bytes of data"
[2022-02-21 14:05:25] INFO virtnbdbackup - backupDisk [vdb]: 5505024 bytes of data extents to backup
 ./virtnbdbackup -d os-alma84_default -l diff -o /tmp/DIFFTEST -i vdb  2>&1 | grep "bytes of data"
[2022-02-21 14:05:32] INFO virtnbdbackup - backupDisk [vdb]: 10747904 bytes of data extents to backup

so it does save the accumulated differences since the last full backup,...

[abbbi]

NM,the -q option allways queries all changed blocks, because its not passing a bitmap to the qemu command.

[skleijkers]

Hmmm with the -q option I don't get the incremental/differential but the full backup again...

Where can I find the XML that gets send to libvirt?

Log from the diff:
[2022-02-21 14:42:38] INFO common - printVersion [MainThread]: Version: 0.45 Arguments: /usr/bin/virtnbdbackup -d X -l diff -o /X/backupset.99 -q
[2022-02-21 14:42:38] INFO virtnbdbackup - main [MainThread]: Backup level: [diff]
[2022-02-21 14:42:38] INFO virtnbdbackup - main [MainThread]: Domain has 1 disks attached which support changed block tracking.
[2022-02-21 14:42:38] INFO virtnbdbackup - main [MainThread]: Concurrent backup processes: [1]
[2022-02-21 14:42:38] INFO libvirthelper - redefineCheckpoints [MainThread]: Loading checkpoint list from: [/X/backupset.99/checkpoints]
[2022-02-21 14:42:38] INFO virtnbdbackup - main [MainThread]: Checkpoint handling
[2022-02-21 14:42:38] INFO virtnbdbackup - main [MainThread]: Next checkpoint id: 1
[2022-02-21 14:42:38] INFO virtnbdbackup - main [MainThread]: Parent checkpoint name virtnbdbackup.0
[2022-02-21 14:42:38] INFO virtnbdbackup - main [MainThread]: Offline or diff backup, use last known checkpoint, save only delta.
[2022-02-21 14:42:38] INFO virtnbdbackup - main [MainThread]: Using checkpoint name: virtnbdbackup.0
[2022-02-21 14:42:38] INFO virtnbdbackup - main [MainThread]: Temporary scratch file target directory: /var/tmp
[2022-02-21 14:42:38] INFO virtnbdbackup - main [MainThread]: NDB Endpoint socket: /var/tmp/virtnbdbackup.49446
[2022-02-21 14:42:38] INFO virtnbdbackup - main [MainThread]: Starting backup job.
[2022-02-21 14:42:38] INFO libvirthelper - fsFreeze [MainThread]: Freeze filesystems.
[2022-02-21 14:42:38] INFO libvirthelper - fsThaw [MainThread]: Thawed filesystems.
[2022-02-21 14:42:38] INFO virtnbdbackup - backupDisk [vda]: INC/DIFF backup: set context to qemu:dirty-bitmap:backup-vda
[2022-02-21 14:42:38] INFO nbdhelper - version [vda]: libnbd version: 1.6.1
[2022-02-21 14:42:38] INFO nbdhelper - waitForServer [vda]: Waiting until NBD server on socket [/var/tmp/virtnbdbackup.49446] is up.
[2022-02-21 14:42:39] INFO nbdhelper - getBlockInfo [vda]: Using Maximum Block size supported by NBD server: [33554432]
[2022-02-21 14:42:39] INFO nbdhelper - waitForServer [vda]: Connection to NBD backend succeeded.
[2022-02-21 14:42:39] INFO virtnbdbackup - backupDisk [vda]: Using qemu tools to query extents
[2022-02-21 14:42:42] INFO virtnbdbackup - backupDisk [vda]: Got 14385 extents to backup.
[2022-02-21 14:42:42] INFO virtnbdbackup - backupDisk [vda]: 68719476736 bytes disk size
[2022-02-21 14:42:42] INFO virtnbdbackup - backupDisk [vda]: 5062791168 bytes of data extents to backup
[2022-02-21 14:42:42] INFO virtnbdbackup - backupDisk [vda]: Write data to target file: /X/backupset.99/vda.diff.virtnbdbackup.0.data.partial
[2022-02-21 14:42:42] INFO virtnbdbackup - backupDisk [vda]: Creating thin provisioned stream backup image
[2022-02-21 14:43:15] INFO virtnbdbackup - backupConfig [MainThread]: Saving VM config to: /X/backupset.99/vmconfig.virtnbdbackup.0.xml
[2022-02-21 14:43:15] INFO virtnbdbackup - main [MainThread]: Backup jobs finished, stopping backup task.
[2022-02-21 14:43:15] INFO virtnbdbackup - main [MainThread]: Finished

[abbbi]

Hmmm with the -q option I don't get the incremental/differential but the full backup again...

yes, the qemu-img command is not passed an map so its allways returning all extents, not just the changed ones from the bitmap.

Where can I find the XML that gets send to libvirt?

use option -v

[abbbi]

ok, found the issue. Zeroed blocks were not written to the stream format if backup type is diff, .. thats why the resulting images were broken.
Now i can also mount an image recovered from an differencial backup.

Summary:

1. diff backup now saves the difference to the last checkpoint, doesnt matter if that one is full or incremental
2. no checkpoint is created
3. target file for diff backup is still overwritten as the checkpoint name is the same. (what would make more sense, having a timestamp within the filename in case of diff backup? And: how to handle it during restore,..)

[skleijkers]

If I'm correct I could do a manual recreate by moving the data files to another directory, as far as I now can see virtnbdrestore tries to use all the data files in the dataset directory. And if the until option is used it checks if it has reached the until checkpoint and stops, so this depends on the right order of the data files.

[abbbi]

yes, the restore utility operates on the file creation time of the data files and uses this order for restore. If you move them, you need to keep the original ctime in order for everything to work.

[skleijkers]

Yes, this works as expected, well now it's possible to make diffs and incrementals! Even making incrementals with multiple diffs on those incrementals. Only thing that has to be done by hand is the restore by leaving the right full, incrementals and diff in the directory and moving the other incrementals and diffs. Well I don't think this is very complicated, because I don't think there is a use case for very complicated trees, so the moving of files will be rather straightforward, it would help if the diff files got a timestamp in the filename. Of course the users should be aware of this, else the recreation of the image can give unexpected results (if all the diffs and incrementals get applied).

[abbbi]

it would help if the diff files got a timestamp in the filename. Of course the users should be aware of this, else the recreation of the image can give unexpected results (if all the diffs and incrementals get applied).

i think this should just work out, if the differencials get an unique name, the order applied during restore is the correct one and resulting
image should work no matter if multiple diffs or incrementals are applied. With --until it should be possible to reach the right point in time. Try with latest commit, it adds the timestamp to differencial backups instead of the checkpoint name.

[skleijkers]

Thanks will give it a try!

[skleijkers]

Hi,

I'm using the master from yesterday (0.47) at the moment with diffs against one full backup, really happy with it, thanks for implementing this feature!

The only point left is the need to reshuffle the diffs a bit if you don't want to restore the last diff, but that's a minor inconvenience. Because it isn't a chain like with the incrementals, it isn't a problem to (re)move some diffs to get the one you want.

Kind regards,
Stefan

[skleijkers]

Well --until needs a checkpoint name and the checkpoints won't get saved, besides all the diffs have the same checkpoint name (in the log) if you take them from the same full or inc.
In my situation the full gets the checkpoint name virtnbdbackup.0 and all the diffs get virtnbdbackup.1, although the checkpoints aren't saved. So there's no way to select a diff. So therefor it is needed to reshuffle/(re)move the files.

Futhermore if you have more complicated backup schema's you have to reshuffle a bit more, let's say you have the following tree:
full (.0), then a diff1 (.1, but it won't get saved), then a inc (.1) and then two diffs, diff2 and diff3, both named .2, but won't get saved.
Now if you want to restore diff1 you have to move inc, diff2 and diff3 and do a restore without until.
Now if you want to restore diff2, you have to move diff1 and diff3 and do a restore without until.
Else you could get an unexpected result

[abbbi]

i had in the back of my mind that --until is file based too, but it seems it checks the meta data header of the stream file to stop during restore
thats something that can be changed.

[abbbi]

check latest commits. It now sets the checkpointName in the header file to the same timestamp as included in the diff backup file, so this can be specified via --until:

example:

./virtnbdrestore -i /tmp/META/ -a dump -o /tmp/RESTORE
[2022-02-23 10:52:10] INFO virtnbdrestore - dump [MainThread]: /tmp/META/vdb.diff.1645609787.data
{'checkpointName': 1645609787,
[..]

[skleijkers]

Yeah that would be an option, but then you have to increment the numbering, even for diffs, and you still have troubles with more complicated schema's. I see you just made a commit to give diffs a numbering...
So maybe an option with which the user can provide a custom/manual restore path would be an easier thing to implement. Given the example above:
restoring diff1 with the option --restorepath vda.full.data,vda.diff1.data
and
restoring diff2 with the option --restorepath vda.full.data,vda.inc.data,vda.diff2.data
Of course the user has to figure out the correct path else you could end up with incorrect data of the filesystem.

[abbbi]

check latest release, now supports specifiying the sequence manually:

virtnbdrestore -i /tmp/META/ -o /tmp/RESTORE -a restore --sequence /tmp/META/vdb.full.data,/tmp/META/vdb.inc.virtnbdbackup.1.data

[jnsnow]

btw, check out the BitmapSyncMode parameter of blockdev-backup command: https://qemu-project.gitlab.io/qemu/interop/qemu-qmp-ref.html#qapidoc-422

on-success creates the behavior you expect with incremental backups, i.e. copied regions are cleared from the bitmap so they won't be copied again.
never doesn't clear any of the bits, so they'll be copied again. You can use this to create differentials with the blockdev-backup command.