Use New BIO_RW_FAILFAST_* API

[behlendorf]

As of zfs-0.4.5 we no longer unconditionally use the BIO_RW_FAILFAST flag. In newer kernels BIO_RW_FAILFAST was replaced with IO_RW_FAILFAST_{DEV|_TRANSPORT|_DRIVER}. The API change is a step in the right direction but the vdev disk code needs to be updated to take advantage of the new API.

For now if the legacy BIO_RW_FAILFAST flag is detected at configure time we use it. If it is missing it means we are running against a kernel with the newer API. With the new API we should be able to ensure some fairly smart behavior in the face of IO errors, but until then we are going to have a crazy number of unless retries at the lower layers.

Last week I observed a real disk failure while zfs-0.4.7 was under a kpios write load in RHEL5.4. While the software did handle the failure the BIO_RW_FAILFAST support does not appear to have worked correctly. The single drive failure hung the committing transaction group for at least 360 seconds while the low level mptsas driver retried the IO. Additionally, from what I can tell it does not appear that the IO failure was properly reported back to ZFS either. What we need to have happen is for the IO to fail immediately with minimal retries at the driver and scsi layer.

[rincebrain]

I have just observed this on an Ubuntu 10.04 system running a git checkout (commit ID: 7dfe24e).

My situation is much nastier, actually. I was testing making zvols into an MD RAID-6, using an ancient disk array connected via the aic7xxx driver in-kernel. I was doing some writes, and it was working nicely, then I pulled a disk to see what would happen.

Well, it's now about 10-15 minutes later, and i've seen no useful activity in ZFS since then. The pool the disk was in reported a lot of write errors but hasn't marked the disk/pool as offline, and consequently MD is blocking forever. Even re-inserting the disk did not prompt any useful behavior.

[behlendorf]

Fixed as much as possible without patching the SCSI the layer or individual drivers. As of 0.5.2 using linux 2.6.32 and the mpt2sas driver drive failures are now observed in roughly 2-3 minutes. This is acceptable for now, and it's worth pointing out that this same basic problem exists on OpenSolaris systems.