ZTS: replacement/resilver_restart_001 fails on FreeBSD 14+

[snajpa]

System information

Type	Version/Name
Distribution Name	FreeBSD
Distribution Version	14+
Kernel Version
Architecture
OpenZFS Version

Describe the problem you're observing

   11:22:35.19 NOTE: Testing with_early_restart deferred resilvers
  11:22:35.20 vfs.zfs.resilver_defer_percent: 0 -> 100
  11:22:35.20 SUCCESS: set_tunable32 RESILVER_DEFER_PERCENT 100
  11:22:35.21 cleared 64 events
  11:22:35.21 SUCCESS: zpool events -c
  11:22:35.21 vfs.zfs.resilver_min_time_ms: 3000 -> 50
  11:22:35.21 SUCCESS: set_tunable32 RESILVER_MIN_TIME_MS 50
  11:22:35.85 SUCCESS: zpool replace testpool1 /var/tmp/file-2 /var/tmp/spare-1
  11:22:35.86 vfs.zfs.scan_suspend_progress: 0 -> 1
  11:22:35.87 SUCCESS: set_tunable32 SCAN_SUSPEND_PROGRESS 1
  11:22:35.89 NOTE: expected 1 resilver start(s), found 1
  11:22:36.08 SUCCESS: zpool offline testpool1 /var/tmp/file-3
  11:22:37.17 SUCCESS: zpool sync testpool1
  11:22:38.25 SUCCESS: zpool online testpool1 /var/tmp/file-3
  11:22:39.30 SUCCESS: zpool sync testpool1
  11:22:39.31 NOTE: expected 2 resilver start(s) after offline/online, found 1
  11:22:39.31 expected 2 resilver start(s) after offline/online, found 1
  11:22:39.31 NOTE: Performing test-fail callback (/usr/local/share/zfs/zfs-tests/callbacks/zfs_dmesg.ksh)

Filing this with the expectation that I'll fix this, but need a few days to get to it. If anyone can get me zfs_dbgmsg at the time when this happens, that could save me some time as I hope I could see what's wrong from that without the need to play around with this on FreeBSD.

Describe how to reproduce the problem

Run ZTS replacement/resilver_restart_001 on FreeBSD 14 or 14.1

[haowuge]

I am unable to run the script. How should I proceed?
Thank you

git clone https://github.com/openzfs/zfs
./autogen.sh 
MAKE=gmake ./configure
sudo gmake install

root@LoongBuild:~ # /usr/local/share/zfs/zfs-tests/tests/functional/replacement/resilver_restart_001.ksh
/usr/local/share/zfs/zfs-tests/tests/functional/replacement/resilver_restart_001.ksh[22]: .: /include/libtest.shlib: not found
root@LoongBuild:~ # freebsd-version -ku
14.2-RC1
14.2-RC1
root@LoongBuild:~ # 

[snajpa]

@haowuge there's a helper script that wraps the testsuite script runner - ./scripts/zfs-tests.sh, takes -h for help

you can run this one test like this:

cd zfs
./scripts/zfs-tests.sh -t tests/functional/replacement/resilver_restart_001.ksh

(just please be sure to only run it on a dev machine with no data you can't afford to lose)

[haowuge]

@haowuge there's a helper script that wraps the testsuite script runner - ./scripts/zfs-tests.sh, takes -h for help

you can run this one test like this:

cd zfs
./scripts/zfs-tests.sh -t tests/functional/replacement/resilver_restart_001.ksh

(just please be sure to only run it on a dev machine with no data you can't afford to lose)

I saw: “SKIP replacement/resilver_restart_001.ksh (expected PASS)”
Did it fail to run?
I might need to wait until I get home tonight to test it; I'm afraid that remotely restarting the NAS host will result in a failure.
The currently running ZFS in the system is from 14.2-RC1, and the test script is from OpenZFS master.

[haowuge]

Is this the normal execution completed?

❯ sudo ./zfs-tests.sh  -t tests/functional/replacement/resilver_restart_001.ksh
zfs-tests.sh: This script must not be run as root.
❯ ./zfs-tests.sh  -t tests/functional/replacement/resilver_restart_001.ksh
Test: /usr/local/share/zfs/zfs-tests/tests/functional/replacement/setup (run as root) [00:00] [FAIL]
Test: /usr/local/share/zfs/zfs-tests/tests/functional/replacement/resilver_restart_001.ksh (run as root) [00:00] [SKIP]
Test: /usr/local/share/zfs/zfs-tests/tests/functional/replacement/cleanup (run as root) [00:00] [PASS]

Results Summary
PASS       1
FAIL       1
SKIP       1

Running Time:   00:00:00
Percent passed: 33.3%
Log directory:  /var/tmp/test_results/20241201T190752

Tests with results other than PASS that are expected:

Tests with result of PASS that are unexpected:

Tests with results other than PASS that are unexpected:
    SKIP replacement/resilver_restart_001.ksh (expected PASS)
    FAIL replacement/setup (expected PASS)

[snajpa]

@haowuge there's a [FAIL] at setup step, so it skipped the test; you can find the logs of the last test run in /var/tmp/test_results/current by default

[haowuge]

@haowuge there's a [FAIL] at setup step, so it skipped the test; you can find the logs of the last test run in /var/tmp/test_results/current by default

I feel like it didn't successfully complete the task😄

https://haowuge.github.io/pull_requests_images/2024-12-01/test_results-20241201T101933.tar.gz

[mcmilk]

@haowuge - you have loaded the zfs.ko module. I think you need to blacklist this one, and insert openzfs.ko.
So you are testing the shipped default zfs module - which seems not to work.

[mcmilk]

This is still an issue and happens in all versions of FreeBSD (13.x 14x 15x).
Here are the Links:

• FreeBSD 13.4
• FreeBSD 14.2
• FreeBSD 15.0

I looks like some data isn't correctly flushed to disk... but already read and compared.
I think that I will check this a bit deeper also.

[snajpa]

@mcmilk thanks, I wonder what the FreeBSD-specific code does differently, I don't think it's my change to be blamed, probably will be some quirk I'm not expecting, which doesn't exist on Linux, but honestly... Linux is my primary area of focus and AFAICT there are enough FreeBSD fulltime contributors, very much unlike the current situation with OpenZFS + Linux, so I figured that if this is important someone else would fix it sooner than me (seems it's not as it's still not fixed)

[snajpa]

I think I can promise to not contribute such changes which focus on "user delight" in the spirit of original ZFS, as the changes OpenZFS is getting in recent years go very much against that. And with other people in the project refusing to put in more work, only caring for changes, which they need for their subsets of customers, I dunno, my enthusiasm has cooled off after a few discussions here. The xattr mess with Gluster is a fine example if you want some :D I'm not okay with breaking things for people and offering no solutions. The other discussion is the effect of VFS-level IO limits, which go against the premise of what ARC was supposed to do... I mean, meh.

[mcmilk]

I am also a bit more Linux only ... but when I see some failure I can maybe fix, then I will try this ;-)