Backup of Remote Host: Waiting for NBD Server connection

[aroskamp]

Hello,

the features that are provided by virtnbdbackup sound great. Thank You for this peace of software.

Unfortuntely I am having some problems establisching a backup of a remote VM. The VM (opensuse-15-6) is running on a Ubuntu Server 22.04.4 (vhst7) and virtnbdbackup is called on a OpenSUSE 15.5 host (fnasvha). Both PCs are in the same Subnet and there are no firewalls active. Even AppArmor is disabled.

fnasvha:~ # virtnbdbackup --version
2.11
fnasvha:~ # virtnbdbackup -U qemu+ssh://gtiadm@vhst7/system --ssh-user gtiadm -d opensuse-15-6 -l full -o /nas/BackupMirror7/vmbackvhst7/ -p
[2024-07-31 13:27:35] INFO lib common - printVersion [main]:  Version: 2.11 Arguments: /usr/bin/virtnbdbackup -U qemu+ssh://gtiadm@vhst7/system --ssh-user gtiadm -d opensuse-15-6 -l full -o /nas/BackupMirror7/vmbackvhst7/ -p
[2024-07-31 13:27:35] INFO root virtnbdbackup - main [main]:  Backup level: [full]
[2024-07-31 13:27:36] INFO virt client - _connect [main]:  Connected to remote host: [vhst7.ibb.com]
[2024-07-31 13:27:36] INFO root virtnbdbackup - main [main]:  Libvirt library version: [8000000]
[2024-07-31 13:27:36] INFO root virtnbdbackup - main [main]:  NBD library version: [1.18.1]
[2024-07-31 13:27:36] INFO root disktype - Optical [main]:  Skipping attached [cdrom] device: [sda].
[2024-07-31 13:27:36] INFO root virtnbdbackup - main [main]:  Backup will save [1] attached disks.
[2024-07-31 13:27:36] INFO root virtnbdbackup - main [main]:  Concurrent backup processes: [1]
[2024-07-31 13:27:36] INFO root checkpoint - create [main]:  Loading checkpoints from: [/nas/BackupMirror7/vmbackvhst7//opensuse-15-6.cpt]
[2024-07-31 13:27:36] INFO root checkpoint - redefine [main]:  Loading checkpoint list from: [/nas/BackupMirror7/vmbackvhst7//checkpoints]
[2024-07-31 13:27:36] INFO root checkpoint - create [main]:  Checkpoint handling.
[2024-07-31 13:27:36] INFO root checkpoint - create [main]:  Removing all existent checkpoints before full backup.
[2024-07-31 13:27:36] INFO root checkpoint - create [main]:  Using checkpoint name: [virtnbdbackup.0].
[2024-07-31 13:27:36] INFO ssh client - connect [main]:  Connecting remote system [vhst7.ibb.com] via ssh, username: [gtiadm]
[2024-07-31 13:27:36] INFO paramiko.transport transport - _log [Thread-1]:  Connected (version 2.0, client OpenSSH_8.9p1)
[2024-07-31 13:27:36] INFO paramiko.transport transport - _log [Thread-1]:  Authentication (publickey) successful!
[2024-07-31 13:27:36] INFO root virtnbdbackup - main [main]:  Remote NBD Endpoint host: [vhst7.ibb.com]
[2024-07-31 13:27:36] INFO root virtnbdbackup - main [main]:  Temporary scratch file target directory: [/var/tmp]
[2024-07-31 13:27:36] INFO root job - start [main]:  Starting backup job.
[2024-07-31 13:27:36] INFO fs fs - freeze [main]:  Freezed [1] filesystems.
[2024-07-31 13:27:36] INFO fs fs - thaw [main]:  Thawed [1] filesystems.
[2024-07-31 13:27:36] INFO root virtnbdbackup - main [main]:  Started backup job with checkpoint, saving information.
[2024-07-31 13:27:36] INFO root checkpoint - backup [main]:  Saving checkpoint config to: [/nas/BackupMirror7/vmbackvhst7//checkpoints/virtnbdbackup.0.xml]
[2024-07-31 13:27:36] INFO nbd client - connect [vda]:  Waiting until NBD server at [nbd://vhst7.ibb.com:10809/vda] is up.
[2024-07-31 13:27:37] INFO nbd client - connect [vda]:  Waiting for NBD Server connection, Retry: 0 [Unable to connect nbd server: nbd_connect_uri: recv: Connection refused (ECONNREFUSED)]
[2024-07-31 13:27:38] INFO nbd client - connect [vda]:  Waiting for NBD Server connection, Retry: 1 [Unable to connect nbd server: nbd_connect_uri: recv: Connection refused (ECONNREFUSED)]
[2024-07-31 13:27:39] INFO nbd client - connect [vda]:  Waiting for NBD Server connection, Retry: 2 [Unable to connect nbd server: nbd_connect_uri: recv: Connection refused (ECONNREFUSED)]
[2024-07-31 13:27:40] INFO nbd client - connect [vda]:  Waiting for NBD Server connection, Retry: 3 [Unable to connect nbd server: nbd_connect_uri: recv: Connection refused (ECONNREFUSED)]
[2024-07-31 13:27:41] INFO nbd client - connect [vda]:  Waiting for NBD Server connection, Retry: 4 [Unable to connect nbd server: nbd_connect_uri: recv: Connection refused (ECONNREFUSED)]
[2024-07-31 13:27:42] INFO nbd client - connect [vda]:  Waiting for NBD Server connection, Retry: 5 [Unable to connect nbd server: nbd_connect_uri: recv: Connection refused (ECONNREFUSED)]
[2024-07-31 13:27:43] INFO nbd client - connect [vda]:  Waiting for NBD Server connection, Retry: 6 [Unable to connect nbd server: nbd_connect_uri: recv: Connection refused (ECONNREFUSED)]
[2024-07-31 13:27:44] INFO nbd client - connect [vda]:  Waiting for NBD Server connection, Retry: 7 [Unable to connect nbd server: nbd_connect_uri: recv: Connection refused (ECONNREFUSED)]
[2024-07-31 13:27:45] INFO nbd client - connect [vda]:  Waiting for NBD Server connection, Retry: 8 [Unable to connect nbd server: nbd_connect_uri: recv: Connection refused (ECONNREFUSED)]
[2024-07-31 13:27:46] INFO nbd client - connect [vda]:  Waiting for NBD Server connection, Retry: 9 [Unable to connect nbd server: nbd_connect_uri: recv: Connection refused (ECONNREFUSED)]
[2024-07-31 13:27:47] INFO nbd client - connect [vda]:  Waiting for NBD Server connection, Retry: 10 [Unable to connect nbd server: nbd_connect_uri: recv: Connection refused (ECONNREFUSED)]
[2024-07-31 13:27:48] INFO nbd client - connect [vda]:  Waiting for NBD Server connection, Retry: 11 [Unable to connect nbd server: nbd_connect_uri: recv: Connection refused (ECONNREFUSED)]
[2024-07-31 13:27:49] INFO nbd client - connect [vda]:  Waiting for NBD Server connection, Retry: 12 [Unable to connect nbd server: nbd_connect_uri: recv: Connection refused (ECONNREFUSED)]
[2024-07-31 13:27:50] INFO nbd client - connect [vda]:  Waiting for NBD Server connection, Retry: 13 [Unable to connect nbd server: nbd_connect_uri: recv: Connection refused (ECONNREFUSED)]
[2024-07-31 13:27:51] INFO nbd client - connect [vda]:  Waiting for NBD Server connection, Retry: 14 [Unable to connect nbd server: nbd_connect_uri: recv: Connection refused (ECONNREFUSED)]
[2024-07-31 13:27:52] INFO nbd client - connect [vda]:  Waiting for NBD Server connection, Retry: 15 [Unable to connect nbd server: nbd_connect_uri: recv: Connection refused (ECONNREFUSED)]
[2024-07-31 13:27:53] INFO nbd client - connect [vda]:  Waiting for NBD Server connection, Retry: 16 [Unable to connect nbd server: nbd_connect_uri: recv: Connection refused (ECONNREFUSED)]
[2024-07-31 13:27:54] INFO nbd client - connect [vda]:  Waiting for NBD Server connection, Retry: 17 [Unable to connect nbd server: nbd_connect_uri: recv: Connection refused (ECONNREFUSED)]
[2024-07-31 13:27:55] INFO nbd client - connect [vda]:  Waiting for NBD Server connection, Retry: 18 [Unable to connect nbd server: nbd_connect_uri: recv: Connection refused (ECONNREFUSED)]
[2024-07-31 13:27:56] INFO nbd client - connect [vda]:  Waiting for NBD Server connection, Retry: 19 [Unable to connect nbd server: nbd_connect_uri: recv: Connection refused (ECONNREFUSED)]
[2024-07-31 13:27:57] ERROR root virtnbdbackup - main [main]:  Disk backup failed: [NBD endpoint: [TCP(exportName='vda', metaContext='', hostname='vhst7.ibb.com', tls=False, port=10809, backupSocket='')]: connection failed: [Timeout during connection to NBD server backend.]]
[2024-07-31 13:27:57] INFO root virtnbdbackup - main [main]:  Backup jobs finished, stopping backup task.
[2024-07-31 13:27:57] INFO root metadata - backupConfig [main]:  Saving VM config to: [/nas/BackupMirror7/vmbackvhst7//vmconfig.virtnbdbackup.0.xml]
[2024-07-31 13:27:57] ERROR ssh client - run [main]:  Executing remote command failed, return code: [1] stderr: [qemu-img: Could not open '/export/libvirt/images/pool/opensuse-15-6-vda.qcow2': Could not open '/export/libvirt/images/pool/opensuse-15-6-vda.qcow2': Permission denied], stdout: []
[2024-07-31 13:27:57] WARNING root metadata - backupDiskInfo [main]:  Failed to read qcow image info: [Error during remote command: [qemu-img info /export/libvirt/images/pool/opensuse-15-6-vda.qcow2 --output json --force-share]: [qemu-img: Could not open '/export/libvirt/images/pool/opensuse-15-6-vda.qcow2': Could not open '/export/libvirt/images/pool/opensuse-15-6-vda.qcow2': Permission denied] []]
[2024-07-31 13:27:57] ERROR root virtnbdbackup - main [main]:  Error during backup

During this attempt, the directory /var/tmp/backup.TRLDI.vda is created on the VM-Server (owned by user libvirt-qemu and group kvm, permission 600). The systemlog shows the folling lines:

Jul 31 13:27:57 vhst7.ibb.com libvirtd[7902]: Path '/var/tmp/backup.TRLDI.vda' is not accessible: No such file or directory
Jul 31 13:27:57 vhst7.ibb.com libvirtd[7902]: Unable to tear down cgroup access on /var/tmp/backup.TRLDI.vda
Jul 31 13:27:57 vhst7.ibb.com libvirtd[7902]: internal error: child reported (status=125): unable to stat: /var/tmp/backup.TRLDI.vda: No such file or directory
Jul 31 13:27:57 vhst7.ibb.com libvirtd[7902]: unable to stat: /var/tmp/backup.TRLDI.vda: No such file or directory
Jul 31 13:27:57 vhst7.ibb.com libvirtd[7902]: internal error: child reported (status=125): unable to stat: /var/tmp/backup.TRLDI.vda: No such file or directory
Jul 31 13:27:57 vhst7.ibb.com libvirtd[7902]: unable to stat: /var/tmp/backup.TRLDI.vda: No such file or directory
Jul 31 13:27:57 vhst7.ibb.com libvirtd[7902]: Unable to restore security label on /var/tmp/backup.TRLDI.vda

The content of the VM-Servers libvirtd.conf (nothing changed):

gtiadm@vhst7:~$ grep -vE "^#.*$" /etc/libvirt/libvirtd.conf | grep -vE "^$"
auth_unix_ro = "none"
auth_unix_rw = "none"
gtiadm@vhst7:~$ 

What can I do to make it all work?

Thank You very much.

[abbbi]

please add log with verbose option, it shows detailed informaiton on which interface the nbd server is setup.

its unable to connect to the nbd endpoint:

nbd://vhst7.ibb.com:10809/vda

possible causes:

1. firewall blocking access
2. hostname resolution on system where virtnbdbackup is executed resolves vhst7.ibb.com to a wrong ip
3. the nbd service on the remote host is bind to a wrong ip/interface due to resolve issues (maybe remote system resolves its own hostname to 127.0.0.1 and binds nbd endpoint to the loopback device) , you can try --nbd-ip option to bind nbd service to a specific ip that is reachable from the host youre executing virtnbdbackup from.

  -I NBD_IP, --nbd-ip NBD_IP
                        IP used to bind remote NBD service on (default: hostname returned by libvirtd)

the second issue is that the user your are executing the backup on the remote host with (using the ssh connection) is unable to open the qcow file to backup the image information, it needs at least read permissions on these:

backupDiskInfo [main]: Failed to read qcow image info: [Error during remote command: [qemu-img info /export/libvirt/images/pool/opensuse-15-6-vda.qcow2 --output json --force-share]: [qemu-img: Could not open '/export/libvirt/images/pool/opensuse-15-6-vda.qcow2': Could not open '/export/libvirt/images/pool/opensuse-15-6-vda.qcow2': Permission denied]

[aroskamp]

Thank You very much.

Both points helped me. It is true, that the remote server resolved the hostname to 127.0.1.1 (Ubuntu Server). Setting the NBD-IP helped to solve this problem. Together with read permissions on the datafile, it worked.

[aroskamp]

The solution worked