Support for ZFS on top of VDO inline deduplication and best practices etc?

[jittygitty]

Describe the feature would like to see added to OpenZFS

Can we run ZFS on top of VDO (virtual data optimizer) instead of direct on real block device?
Has anyone tried, tested it? (What zfs changes or settings needed if any?)

How will this feature improve OpenZFS?

If it works well, better than built-in zfs deduplication it might be beneficial to run ZFS on top of VDO in some cases.

Additional context

I haven't had a chance to try it yet, was hoping to hear if its supported and/or recommended or not.
I assume you can't use zfs encryption in this case, you'd have to encrypt underlying VDO instead if you want the deduplication.
But I'd want to know any other ZFS settings to watch out for in order to support running on top of VDO.

For those who haven't heard of RedHat's VDO yet, here's some links:
https://www.redhat.com/en/blog/understanding-concepts-behind-virtual-data-optimizer-vdo-rhel-75-beta

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/pdf/deduplicating_and_compressing_storage/red_hat_enterprise_linux-8-deduplicating_and_compressing_storage-en-us.pdf
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/storage_administration_guide/vdo-quick-start

https://github.com/dm-vdo/vdo
https://github.com/dm-vdo/kvdo

https://is.muni.cz/th/rq7e2/petrovic_diploma_thesis.pdf
https://www.marksei.com/vdo-linux-deduplication/

[zfsbot]

easy answer: not supported, not recommended

[derzahla]

@jittygitty I was looking into this as well. I dont think OpenZFS would ever support it, but I see no reason why creating a zpool on a dm-vdo device wouldnt work. Problem is, Red Hat seems only focused on VDO for RHEL7&8. It isn't a part of Fedora and they are not putting much effort into porting VDO to any more recent kernels. I tried and failed to get it to compile on Fedora 35 and Arch.

[IvanVolosyuk]

What does it offer than ZFS doesn't offer by default? It looks like deduplication in VDO will have even more overhead than ZFS built-in one, with uncertain effect on redundancy.

[jittygitty]

@IvanVolosyuk I don't know the details that's why I was asking if anyone had experience with it. In fact that's why I didn't just try it myself, because I was a bit skeptical and I thought if I asked I might hear from someone: tried it and it sucked worse than btrfs or zfs deduplication and maybe it kills cpu or eats ram just as much etc. Unfortunately seems @derzahla says it doesn't compile very easily so you have to install RHEL 7 or 8 to try it out. And I guess Red Hat bought CentOS so they could kill it.

In fact I remember reading bad things about OpenDedupe in the past that it was dead slow and kills cpu (since on fuse?) so I never bothered to try that one either. Anyway thought would be interesting to know how VDO works with zfs on top of it etc.

[shodanshok]

@jittygitty I tested this configuration in the past with good results, but I never deployed it in production. To get good performance, I used sync=disabled on the underlying zvol. However, I want to stress again that I did not use it for production workloads due to the complexity of the setup and its potential failure modes.

What does it offer than ZFS doesn't offer by default? It looks like deduplication in VDO will have even more overhead than ZFS built-in one, with uncertain effect on redundancy.

VDO provides high-performance, high-efficiency compression and deduplication for 4K data blocks. To get somewhat similar from ZFS, one has to use a very small 4K recordsize (exploding metadata usage) with ashift=9 devices (requiring 512B native hardware to avoid heavy r/m/w overhead) and a very fast special device for storing DDT.

[jittygitty]

@shodanshok Many thanks for your great input on this! I been busy with other things but now that thanks to @pjd block cloning "reflink" support may be here soon ( #13392 ) along with the "offline deduplication" that it should make possible, there may not be as much reason to try vdo, unless one wants inline dedupe again. In near future will be interesting to compare zfs live inline dedupe to hopefully upcoming offline/out-of-band zfs deduplication in terms of efficiency etc.