CA-392674: nbd_client_manager retry connect on nbd device busy

[liulinC]

to connect to nbd devices, nbd_client_manager will

1. protect the operation with /var/run/nonpersistent/nbd_client_manager file lock
2. check whether nbd is being used by nbd-client -check
3. load nbd kernel module by modprobe nbd
4. call nbd-client to connect to nbd device

However, step 3 will trigger systemd-udevd run asyncly, which would open and lock the same nbd devices, run udev rules, etc. This introduce races with step 4, e.g. both process want to open and lock the nbd device.

Note: the file lock in step 1 does NOT resovle the issue here, as it only coordinate multiple nbd_client_manager processes.

To fix the issue,

• we patch nbd-client to report the device busy from kernel to nbd_client_manager
• nbd_client_manager should check nbd-client exit code, and retry on device busy

[MarkSymsCtx]

However, step 3 will trigger systemd-udevd run asyncly, which would open and lock the same nbd devices, run udev rules, etc. This introduce races with step 4, e.g. both process want to open and lock the nbd device.

This would be better fixed by adding a rule to udev to ensure that system-udevd leaves the devices alone and thus doesn't result in them being found busy.

[liulinC]

However, step 3 will trigger systemd-udevd run asyncly, which would open and lock the same nbd devices, run udev rules, etc. This introduce races with step 4, e.g. both process want to open and lock the nbd device.

This would be better fixed by adding a rule to udev to ensure that system-udevd leaves the devices alone and thus doesn't result in them being found busy.

XS8 stream udev handle the nbd device as well. (and has this race issue as well).
I have already raise some PR to clean some unnecessary udev rules.
I doubt should we totally ignore nbd devices in udev rules. (as XS8 steam do take care of it)

BTW, udev rules is only run and executed with the device been opened and lock been hold by systemd-udevd(workers),
https://github.com/systemd/systemd/blob/main/src/udev/udev-worker.c#L184-L203
This means the udev rules can only narrow down the race time window, but can not resolve the issue.

So I prefer to resolve the races issues other than just totally ignore nbd devices in the udev rules.

[liulinC]

job: 4126316~4126325 looks good.

[lindig]

Can we get reviews from @MarkSymsCtx and @robhoes ?