Exos X18 I/O error when waking up from Standby_z (HBA 9500-8i)

[domotiquefacile]

Hello,

I've got 2 Seagate Exos X18 16GB in a RAID1 (BTRFS) configuration connected to a HBA 9500-8i, and I regularly get I/O errors when they are waking up from Standby_z state.
If I deactivate Standby_z, I have absolutely no issue with idle_a, idle_b and idle_c states, so the issue doesn't seem to be related to cable or PSU.
I think I managed to reproduce the issue manually by forcing the Standby_z state using openSeaChest_PowerControl -d /dev/sda --transitionPower standby_z but the issue seems to appear much less often if I transition to the state manually compared to letting the hard drive goes to Standby_z after some idle time.

Output from dmesg:

[mer. nov. 19 07:39:01 2025] BTRFS info (device dm-1): device stats zeroed by btrfs (878417)
[mer. nov. 19 18:17:08 2025] sd 0:0:1:0: attempting task abort!scmd(0x000000000fcc91dd), outstanding for 1044 ms & timeout 1000 ms
[mer. nov. 19 18:17:08 2025] sd 0:0:1:0: [sda] tag#6734 CDB: ATA command pass through(16) 85 08 0e 00 d5 00 01 00 e0 00 4f 00 c2 00 b0 00
[mer. nov. 19 18:17:08 2025] scsi target0:0:1: handle(0x000d), sas_address(0x300062b208f03540), phy(0)
[mer. nov. 19 18:17:08 2025] scsi target0:0:1: enclosure logical id(0x300062b208093540), slot(0) 
[mer. nov. 19 18:17:08 2025] scsi target0:0:1: enclosure level(0x0000), connector name( C0.0)
[mer. nov. 19 18:17:12 2025] sd 0:0:1:0: task abort: SUCCESS scmd(0x000000000fcc91dd)
[mer. nov. 19 18:17:12 2025] sd 0:0:1:0: [sda] tag#6751 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=10s
[mer. nov. 19 18:17:12 2025] sd 0:0:1:0: [sda] tag#6751 Sense Key : Not Ready [current] 
[mer. nov. 19 18:17:12 2025] sd 0:0:1:0: [sda] tag#6751 Add. Sense: Logical unit not ready, cause not reportable
[mer. nov. 19 18:17:12 2025] sd 0:0:1:0: [sda] tag#6751 CDB: Read(16) 88 00 00 00 00 00 7e 2d 3a 30 00 00 00 10 00 00
[mer. nov. 19 18:17:12 2025] I/O error, dev sda, sector 2116893232 op 0x0:(READ) flags 0x1000 phys_seg 2 prio class 2
[mer. nov. 19 18:17:12 2025] BTRFS error (device dm-1): bdev /dev/dm-1 errs: wr 0, rd 1, flush 0, corrupt 0, gen 0
[mer. nov. 19 18:17:20 2025] sd 0:0:2:0: attempting task abort!scmd(0x00000000784af5b4), outstanding for 2304 ms & timeout 1000 ms
[mer. nov. 19 18:17:20 2025] sd 0:0:2:0: [sdb] tag#6755 CDB: ATA command pass through(16) 85 08 0e 00 d5 00 01 00 e0 00 4f 00 c2 00 b0 00
[mer. nov. 19 18:17:20 2025] scsi target0:0:2: handle(0x000e), sas_address(0x300062b208f03544), phy(4)
[mer. nov. 19 18:17:20 2025] scsi target0:0:2: enclosure logical id(0x300062b208093540), slot(4) 
[mer. nov. 19 18:17:20 2025] scsi target0:0:2: enclosure level(0x0000), connector name( C0.1)
[mer. nov. 19 18:17:23 2025] sd 0:0:2:0: task abort: SUCCESS scmd(0x00000000784af5b4)
[mer. nov. 19 18:17:24 2025] sd 0:0:2:0: Power-on or device reset occurred
[mer. nov. 19 18:17:24 2025] sd 0:0:1:0: Power-on or device reset occurred
[mer. nov. 19 18:17:24 2025] BTRFS info (device dm-1): read error corrected: ino 0 off 5708692340736 (dev /dev/dm-1 sector 2116860448)
[mer. nov. 19 18:17:24 2025] BTRFS info (device dm-1): read error corrected: ino 0 off 5708692344832 (dev /dev/dm-1 sector 2116860456)
[mer. nov. 19 18:17:24 2025] BTRFS info (device dm-1): read error corrected: ino 0 off 5708692348928 (dev /dev/dm-1 sector 2116860464)
[mer. nov. 19 18:17:24 2025] BTRFS info (device dm-1): read error corrected: ino 0 off 5708692353024 (dev /dev/dm-1 sector 2116860472)

Mainboard: Supermicro X12SCZ-F

  - Firmware Version = 01.05.23
  - Firmware Build Time = 07/14/2025
  - Redfish Version = 1.21.0
  - BIOS Firmware Version = BIOS Date: 07/11/2025 Ver 3.4

HBA: Broadcom HBA 9500-8i

  - Firmware Package Build = 36.00.00.00
  - Firmware Version = 36.00.00.00
  - Bios Version = 09.71.00.00_36.00.00.00
  - NVDATA Version = 36.00.00.11
  - PSOC FW Version = 0x006E
  - PSOC Part Number = 06021
  - Driver Name = mpt3sas
  - Driver Version = 43.100.00.00

HDDs: 2x Seagate Exos X18 16GB (ST16000NM000J-2TW103)

  - FwRev = SN04
  - EPC settings =
Name       Current Timer Default Timer Saved Timer   Recovery Time C S
Idle A     *1            *1            *1            1             Y Y
Idle B     *1200         *1200         *1200         4             Y Y
Idle C     *12000         6000         *12000        20            Y Y
Standby Z  *18000         9000         *18000        110           Y Y

OS: Debian 12.12 (OpenMediaVault) running in a Proxmox VM (with HBA passthrough)

output of openSeaChest_Info -d /dev/sda --echoCommandLine -i -v 4 | tee verboseInfo.txt
verboseInfo.txt

Is this a known issue with Standby_z ? It seems most people are not using this state, but for me I am almost never using those drives (only once every 2 weeks on average), so it makes sense to put them to sleep.
Perhaps this is an interaction problem with the HBA ?
Any help would be greatly appreciated. The only thing I haven't tried so far is to update the hard drive's firmware. But I am a bit afraid to lose the data on the hard drives.

Note: sorry in advance if this is not the correct project to report this issue

[alexmyczko]

@domotiquefacile that standby_z sounds like similiar/same like hdparm -Y /dev/sda (sdb), can you check if it makes a difference? what about on Debian 13? What when using later openseachest versions?

[domotiquefacile]

Do you mean hdparm -S ?
I've never used hdparm -Y as I only want a spindown, not a shutdown:

-Y     Force an IDE drive to immediately enter the lowest power
              consumption sleep mode, causing it to shut down completely.
              A hard or soft reset is required before the drive can be
              accessed again (the Linux IDE driver will automatically
              handle issuing a reset if/when needed).  The current power
              mode status can be checked using the -C option.

I'll test this week-end on Debian 13 and with the latest openseachest version.

[vonericsen]

@domotiquefacile,

I'm sorry I did not see this issue earlier!

When looking at that snippet, I see this:
sd 0:0:2:0: attempting task abort!scmd(0x00000000784af5b4), outstanding for 2304 ms & timeout 1000 ms
So the command to read from the drive was sent expecting a response in one second, but it waited 2.3 seconds before it gave up and cancelled it.
That is not enough time for the drive to spin up from standby_z and service the command.

FYI, the hdparm -Y option is the ATA sleep command which is different.
Standby_z is equivalent to the traditional ATA standby state per the ACS specifications.
Looking at the hdparm manpage, the lowercase y (-y) would be the equivalent.

The difference between standby and sleep states is whether a bus reset is required or not. In standby, a reset is not required to bring the drive back up. Transitions to idle, reads, writes, etc can all bring the drive back up.
In sleep mode, the only way back to ready is a bus reset. It's meant for a case like a full system sleep (hibernate?) state.

I don't know if BTRFS has a timeout you can specify, but you would need a longer timeout for coming out of standby. Something like 15 seconds should be more than enough with a little padding to allow the spin-up before it gives up.

Another thing you can try is seeing if behavior changes when using openSeaChest to go into standby by adding the --forceSCSI option. This forces it to use the SCSI command set instead of ATA passthrough to go into that power state. By doing this, it allows the HBA to perform the translation instead of the openSeaChest software which might make it work a little better since the HBA will now be more likely to be aware of the drive state (if the firmware tracks it or handles other things when entering standby).
This is more inline with what the operating system would do anyways.
For compatibility reasons, openSeaChest prefers to use the ATA passthrough commands most of the time.
With a 1 second timeout for reads, this may still fail nearly the same, but it may report a different issue as well.

[domotiquefacile]

Thanks, you gave a lot of good hints to try to avoid this issue !
OpenMediaVault is not yet on Debian 13, but I am testing the drives on Debian 13 (Linux 6.12.48+deb13-amd64) and with openseachest 24.08.1-1.
So far so good I would say, I still have those warning in logs but no more errors detected:

[lun.  1 déc. 17:38:58 2025] sd 1:0:2:0: attempting task abort!scmd(0x000000007898433b), outstanding for 10260 ms & timeout 30000 ms
[lun.  1 déc. 17:38:58 2025] sd 1:0:2:0: [sdc] tag#2059 CDB: Start/Stop Unit 1b 00 00 00 01 00
[lun.  1 déc. 17:38:58 2025] scsi target1:0:2: handle(0x000e), sas_address(0x300062b208f03544), phy(4)
[lun.  1 déc. 17:38:58 2025] scsi target1:0:2: enclosure logical id(0x300062b208093540), slot(4) 
[lun.  1 déc. 17:38:58 2025] scsi target1:0:2: enclosure level(0x0000), connector name( C0.1)
[lun.  1 déc. 17:38:59 2025] sd 1:0:2:0: task abort: SUCCESS scmd(0x000000007898433b)
[lun.  1 déc. 17:38:59 2025] sd 1:0:2:0: attempting device reset! scmd(0x000000007898433b)
[lun.  1 déc. 17:38:59 2025] sd 1:0:2:0: [sdc] tag#2059 CDB: Read(16) 88 00 00 00 00 00 fe da d9 00 00 00 00 20 00 00
[lun.  1 déc. 17:38:59 2025] scsi target1:0:2: handle(0x000e), sas_address(0x300062b208f03544), phy(4)
[lun.  1 déc. 17:38:59 2025] scsi target1:0:2: enclosure logical id(0x300062b208093540), slot(4) 
[lun.  1 déc. 17:38:59 2025] scsi target1:0:2: enclosure level(0x0000), connector name( C0.1)
[lun.  1 déc. 17:38:59 2025] sd 1:0:2:0: device reset: FAILED scmd(0x000000007898433b)
[lun.  1 déc. 17:38:59 2025] scsi target1:0:2: attempting target reset! scmd(0x000000007898433b)
[lun.  1 déc. 17:38:59 2025] sd 1:0:2:0: [sdc] tag#2059 CDB: Read(16) 88 00 00 00 00 00 fe da d9 00 00 00 00 20 00 00
[lun.  1 déc. 17:38:59 2025] scsi target1:0:2: handle(0x000e), sas_address(0x300062b208f03544), phy(4)
[lun.  1 déc. 17:38:59 2025] scsi target1:0:2: enclosure logical id(0x300062b208093540), slot(4) 
[lun.  1 déc. 17:38:59 2025] scsi target1:0:2: enclosure level(0x0000), connector name( C0.1)
[lun.  1 déc. 17:38:59 2025] scsi target1:0:2: target reset: SUCCESS scmd(0x000000007898433b)
[lun.  1 déc. 17:39:00 2025] sd 1:0:2:0: Power-on or device reset occurred
[lun.  1 déc. 17:39:31 2025] sd 1:0:1:0: attempting task abort!scmd(0x000000003b4fe1a9), outstanding for 10240 ms & timeout 30000 ms
[lun.  1 déc. 17:39:31 2025] sd 1:0:1:0: [sdb] tag#2105 CDB: Start/Stop Unit 1b 00 00 00 01 00
[lun.  1 déc. 17:39:31 2025] scsi target1:0:1: handle(0x000d), sas_address(0x300062b208f03540), phy(0)
[lun.  1 déc. 17:39:31 2025] scsi target1:0:1: enclosure logical id(0x300062b208093540), slot(0) 
[lun.  1 déc. 17:39:31 2025] scsi target1:0:1: enclosure level(0x0000), connector name( C0.0)
[lun.  1 déc. 17:39:31 2025] sd 1:0:1:0: task abort: SUCCESS scmd(0x000000003b4fe1a9)
[lun.  1 déc. 17:39:31 2025] sd 1:0:1:0: attempting device reset! scmd(0x000000003b4fe1a9)
[lun.  1 déc. 17:39:31 2025] sd 1:0:1:0: [sdb] tag#2105 CDB: Write(16) 8a 00 00 00 00 00 0f 6b 3c 20 00 00 00 20 00 00
[lun.  1 déc. 17:39:31 2025] scsi target1:0:1: handle(0x000d), sas_address(0x300062b208f03540), phy(0)
[lun.  1 déc. 17:39:31 2025] scsi target1:0:1: enclosure logical id(0x300062b208093540), slot(0) 
[lun.  1 déc. 17:39:31 2025] scsi target1:0:1: enclosure level(0x0000), connector name( C0.0)
[lun.  1 déc. 17:39:31 2025] sd 1:0:1:0: device reset: FAILED scmd(0x000000003b4fe1a9)
[lun.  1 déc. 17:39:31 2025] scsi target1:0:1: attempting target reset! scmd(0x000000003b4fe1a9)
[lun.  1 déc. 17:39:31 2025] sd 1:0:1:0: [sdb] tag#2105 CDB: Write(16) 8a 00 00 00 00 00 0f 6b 3c 20 00 00 00 20 00 00
[lun.  1 déc. 17:39:31 2025] scsi target1:0:1: handle(0x000d), sas_address(0x300062b208f03540), phy(0)
[lun.  1 déc. 17:39:31 2025] scsi target1:0:1: enclosure logical id(0x300062b208093540), slot(0) 
[lun.  1 déc. 17:39:31 2025] scsi target1:0:1: enclosure level(0x0000), connector name( C0.0)
[lun.  1 déc. 17:39:31 2025] scsi target1:0:1: target reset: SUCCESS scmd(0x000000003b4fe1a9)
[lun.  1 déc. 17:39:32 2025] sd 1:0:1:0: Power-on or device reset occurred

It seems that the timeout is 30 seconds now ? I don't know where the timeout is configured.
I will test further for one week to see how it goes.

Thanks for your help !

[domotiquefacile]

I haven't been able to locate where we can change the timeout value, I thought it was in /sys/block/device-name/device/timeout or /sys/block/device-name/device/eh_timeout but I am trying to confirm.
In OpenMediaVault, I upgraded the linux kernel to the backport version 6.12.57+deb12-amd64.
Now it seems the timeout has been modified to 30s.
After one week, I haven't noticed any issue anymore.
But just yesterday, I had one timeout again:

[ven. déc.  5 13:40:46 2025] sd 0:0:2:0: attempting task abort!scmd(0x000000002f5a74f5), outstanding for 10104 ms & timeout 10000 ms
[ven. déc.  5 13:40:46 2025] sd 0:0:2:0: [sdb] tag#4939 CDB: ATA command pass through(16) 85 08 0e 00 d5 00 01 00 e0 00 4f 00 c2 00 b0 00
[ven. déc.  5 13:40:46 2025] scsi target0:0:2: handle(0x000e), sas_address(0x300062b208f03544), phy(4)
[ven. déc.  5 13:40:46 2025] scsi target0:0:2: enclosure logical id(0x300062b208093540), slot(4) 
[ven. déc.  5 13:40:46 2025] scsi target0:0:2: enclosure level(0x0000), connector name( C0.1)
[ven. déc.  5 13:40:46 2025] sd 0:0:2:0: task abort: SUCCESS scmd(0x000000002f5a74f5)
[ven. déc.  5 13:40:47 2025] sd 0:0:2:0: Power-on or device reset occurred
[sam. déc.  6 13:42:07 2025] sd 0:0:2:0: attempting task abort!scmd(0x00000000c97311bd), outstanding for 10180 ms & timeout 30000 ms
[sam. déc.  6 13:42:07 2025] sd 0:0:2:0: [sdb] tag#4987 CDB: Start/Stop Unit 1b 00 00 00 01 00
[sam. déc.  6 13:42:07 2025] scsi target0:0:2: handle(0x000e), sas_address(0x300062b208f03544), phy(4)
[sam. déc.  6 13:42:07 2025] scsi target0:0:2: enclosure logical id(0x300062b208093540), slot(4) 
[sam. déc.  6 13:42:07 2025] scsi target0:0:2: enclosure level(0x0000), connector name( C0.1)
[sam. déc.  6 13:42:08 2025] sd 0:0:2:0: task abort: SUCCESS scmd(0x00000000c97311bd)
[sam. déc.  6 13:42:08 2025] sd 0:0:2:0: attempting device reset! scmd(0x00000000c97311bd)
[sam. déc.  6 13:42:08 2025] sd 0:0:2:0: [sdb] tag#4987 CDB: Read(16) 88 00 00 00 00 00 fe da 16 60 00 00 00 20 00 00
[sam. déc.  6 13:42:08 2025] scsi target0:0:2: handle(0x000e), sas_address(0x300062b208f03544), phy(4)
[sam. déc.  6 13:42:08 2025] scsi target0:0:2: enclosure logical id(0x300062b208093540), slot(4) 
[sam. déc.  6 13:42:08 2025] scsi target0:0:2: enclosure level(0x0000), connector name( C0.1)
[sam. déc.  6 13:42:08 2025] sd 0:0:2:0: device reset: FAILED scmd(0x00000000c97311bd)
[sam. déc.  6 13:42:08 2025] scsi target0:0:2: attempting target reset! scmd(0x00000000c97311bd)
[sam. déc.  6 13:42:08 2025] sd 0:0:2:0: [sdb] tag#4987 CDB: Read(16) 88 00 00 00 00 00 fe da 16 60 00 00 00 20 00 00
[sam. déc.  6 13:42:08 2025] scsi target0:0:2: handle(0x000e), sas_address(0x300062b208f03544), phy(4)
[sam. déc.  6 13:42:08 2025] scsi target0:0:2: enclosure logical id(0x300062b208093540), slot(4) 
[sam. déc.  6 13:42:08 2025] scsi target0:0:2: enclosure level(0x0000), connector name( C0.1)
[sam. déc.  6 13:42:08 2025] scsi target0:0:2: target reset: SUCCESS scmd(0x00000000c97311bd)
[sam. déc.  6 13:42:09 2025] sd 0:0:2:0: Power-on or device reset occurred

I don't understand why sometimes the timeout is 30s and sometimes 10s.
But I will investigate and report here !

EDIT: I think I found how to change the timeout value to 30s using smartctl -l scterc,300,300 /dev/sda

[vonericsen]

The timeout may be baked in some weird place in that software or the file system....I'm not sure. I know there is the kernel file for timeout, but I don't know how that is used by software or by file systems. The files you mention sound correct to change, but if neither are set to the ten seconds, then it is going to be somewhere else less obvious.

The SCT ERC (Error Recovery Control) is a drive level feature that does not affect the OS's timeout which is what really needs to be changed instead.
What this does is it tells the drive a time limit to perform error recovery for reads or writes.
I wouldn't expect this to change anything other than when it reports an aborted command versus continuing to wait to spin up and eventually need a reset since the firmware is still waiting for the disk to get up to speed to read properly.

I've run into some customers setting really aggressive timeouts in the past thinking that will improve their IO, but they usually end up changing it later because of similar issues.
The "new" thing that has been added to the kernel and newer drives is CDL or command duration limits.
This allows the OS/filesystem to indicate the priority of a command, completion time required, and how to report the error to the system for each of the 7 buckets allowed for reads and writes.
This is a better way of managing high priority IO requests, but not every drive (or software/file system) supports it yet...it's very new.

If you run the option to show the EPC settings with --forceSCSI do those match without --forceSCSI?

[domotiquefacile]

/sys/block/device-name/device/timeout was already at 30 and I set /sys/block/device-name/device/eh_timeout to 30 as well (it was previously at 10).
But I still got the 10s timeout.

Since I changed the SCT ERC to 30s, there is no error reported at all in the logs (and no errors reported in the BTRFS array).
It is not that I got lines with a timeout value of 30s, I don't have error lines in the logs at all.

What a mystery... I searched and searched but no idea where those 10s timeout are coming from. But so far it seems that using the SCT ERC trick, I don't have any timeout anymore.

Without --forceSCSI I got the following results:

openSeaChest_PowerControl -d /dev/sda --showEPCSettings
==========================================================================================
 openSeaChest_PowerControl - openSeaChest drive utilities - NVMe Enabled
 Copyright (c) 2014-2024 Seagate Technology LLC and/or its Affiliates, All Rights Reserved
 openSeaChest_PowerControl Version: 3.4.0-6_2_0 X86_64
 Build Date: Jun  9 2024
 Today: Tue Dec  9 21:17:30 2025	User: root
==========================================================================================

/dev/sg2 - ST16000NM000J-2TW103 - ZR5DGQC2 - SN04 - ATA


===EPC Settings===
	* = timer is enabled
	C column = Changeable
	S column = Savable
	All times are in 100 milliseconds

Name       Current Timer Default Timer Saved Timer   Recovery Time C S
Idle A     *1            *1            *1            1             Y Y
Idle B     *1200         *1200         *1200         4             Y Y
Idle C     *12000         6000         *12000        20            Y Y
Standby Z  *18000         9000         *18000        110           Y Y

With --forceSCSI I got the following results:

openSeaChest_PowerControl -d /dev/sda --showEPCSettings --forceSCSI
==========================================================================================
 openSeaChest_PowerControl - openSeaChest drive utilities - NVMe Enabled
 Copyright (c) 2014-2024 Seagate Technology LLC and/or its Affiliates, All Rights Reserved
 openSeaChest_PowerControl Version: 3.4.0-6_2_0 X86_64
 Build Date: Jun  9 2024
 Today: Tue Dec  9 21:17:46 2025	User: root
==========================================================================================
	Forcing SCSI Drive

/dev/sg2 - ST16000NM000J-2TW103 - ZR5DGQC2 - SN04 - SCSI

===EPC Settings===
	* = timer is enabled
	C column = Changeable
	S column = Savable
	All times are in 100 milliseconds

Name       Current Timer Default Timer Saved Timer   Recovery Time C S
Idle A     *16777216     *16777216     *16777216     1             Y N
Idle B     *2953052160   *2953052160   *2953052160   1             Y N
Idle C     *3761111040   *1880555520   *3761111040   1             Y N
Standby Y   0             0             0            0             N N

Not sure what I can deduce from that ?

Thanks for your help !

[vonericsen]

Since I changed the SCT ERC to 30s, there is no error reported at all in the logs (and no errors reported in the BTRFS array).
It is not that I got lines with a timeout value of 30s, I don't have error lines in the logs at all.

What a mystery... I searched and searched but no idea where those 10s timeout are coming from. But so far it seems that using the SCT ERC trick, I don't have any timeout anymore.

Weird that you couldn't find the 10 second timeout anywhere....maybe it was a software update (or kernel, etc).

By setting the ERC timeout, this basically tells the drive return status in 30 seconds or less and do not take any longer than that which it is now doing. Since this matches the software's timeout it is probably allowing it to get past this and move on without logging any errors or triggering resets in the kernel.

There are downsides to limiting error recovery though. Instead of letting the drive retry to read the data and perform corrections, it now gives up and just reports an error. This can potentially make other software/services think it has lost data or force them to attempt correction if software can correct the data here.

It's a pros vs cons type of issue here. Since this is when it is spinning the drive back up, 30 seconds should be plenty of time based on the recovery time reported. Most reads/writes complete in milliseconds or less anyways so it is unlikely to be a problem.
If ERC was set much lower, I can see that potentially causing other issues, but this is probably fine. If you notice anything reporting other errors, then it may be good to revisit this and see if something else is going on.

Not sure what I can deduce from that ?

Well this looks like a bug...but I don't know where. Possibly in openSeaChest, possibly in HBA firmware.
Can you share the output of openSeaChest_PowerControl -d /dev/sdb --showEPCSettings --forceSCSI -v4 | tee verboseEPCforceSCSI.txt?
This will report all the raw responses so I can walk through the code and see what went wrong

[domotiquefacile]

Not sure if I understand: when accessing a disk, if it is not available for any reason, the ERC timeout is started to give the chance to the disk to answer (and if necessary for the disk firmware to perform error recovery). And when the ERC timeout is reached, then we start the /sys/block/device-name/device/timeout (or another timeout located somewhere else with a value of 10s) ?
Or both these timeout are started simultaneously, starting from the access request ?
Also I am not sure when an issue is logged ? It seems to be only logged when the ERC timeout is reached ?

I am not sure if I should set the ERC timeout to a value lower than 30 s (which is the current value of /sys/block/device-name/device/timeout). Having a higher ERC timeout, means that the hard drive will take longer to give up correcting the issue, thus reducing performance in case of faulty drive. In case of healthy hard drive, it is an acceptable compromise it seems.

I have attached the file you requested.

Again, thanks a lot for your help !

verboseEPCforceSCSI.txt

[vonericsen]

Not sure if I understand: when accessing a disk, if it is not available for any reason, the ERC timeout is started to give the chance to the disk to answer (and if necessary for the disk firmware to perform error recovery). And when the ERC timeout is reached, then we start the /sys/block/device-name/device/timeout (or another timeout located somewhere else with a value of 10s) ?
Or both these timeout are started simultaneously, starting from the access request ?
Also I am not sure when an issue is logged ? It seems to be only logged when the ERC timeout is reached ?

Sorry if my last response confused you, but this is a confusing subject for a lot of people since it spans multiple layers.

The short answer here is that both of these timers start at the same time (more or less, small gap between submission from OS to drive, time through drivers and the wire).

Now, a very long answer....

For storage products (SATA, SAS, NVMe, etc) by default/history there is no time limit to error recovery or how long a command should take to process.
The idea is more or less, you will get back a response (pass/abort/uncorrectable/etc) in whatever time it takes for the device to figure out what the appropriate response is, which may include error correction. Error correction can take a lot of time as an HDD must wait for the disk to rotate each time back to the same position to reread. While it may not seem like a lot of time, each revolutions is X milliseconds long to read, read ECC data, process the data, compare to the last result, change things between results, etc, so retries can happen many many times and take a very long time to complete depending on how many the device performs. SAS drives can tell you the retry count in sense data and I have seen some very large numbers in the thousands before, but that was a test case meant to exhaust the ECC algorithm. Real world can be very different. If the first pass is so bad that there just isn't anything to work with, a response can come back immediately as uncorrectable data. If there is some readable data, then that is where the ECC starts to kick in and retries will begin.

At some point in time (and I cannot tell you when as it was likely before I was born haha) a timeout was introduced for issuing IO to storage devices. This was done so that the OS wouldn't sit and wait on a device that is hung or just taking too long to respond.
The way this has historically been managed is that if that device does not respond within this OS defined timeout, the OS will send a reset to the device to reset it and stop the command.
This is what is being logged more or less. The OS timeout ran out (which it gives up to about 1.5 seconds beyond what you set from what I've seen) so it reset the device and because this is considered an issue, it logs it.

As you would expect, a lot of systems, especially servers/data centers, don't like this idea and it doesn't often work in modern systems the same.
Today there is so much redundancy either through drives in the same system (RAID/ZFS/etc) or across multiple servers that they want a response within X number of seconds which can come from one of many places.
So, if 8 different drives have the same data, the request may go out to all of them. If any one drive responds within that time, then they want the others to stop what they are doing and move on to a different request.

ERC is one aspect of this, and it is similar to the SCSI error recovery control that SAS/SCSI has had since about 1986 per the CCS (Common Command Set) that I have a copy of (Firmware may have implemented this later, I do not know for sure).
SATA introduced this with SCT in about 2004 (early proposal doc I have is dated 2003). There was also an attempt at a Time Limited Commands (TLC) feature around this time as well that is similar.
SATA Adopted these much later as enterprises wanted to purchase lower cost storage (SATA) instead of more expensive hardware (SAS) to build out these systems.

So now what is happening is setting the ERC on the drive is telling it to limit how long it takes to service a read or write command before providing a response. By making this in-sync with the OS it will now return a result (pass/abort/uncorrectable/etc) within that time so the OS will not need to perform the resets and other recovery activity that was doing, and therefore no need to log anything. That is because the drive is now responding within the same time limit that the OS is expecting.

If you set ERC too low, you can run into other issues. If it's set to something like 500 milliseconds, that is allowed per the spec, however that is so fast only a healthy drive is going to respond within that time and one that has the heads loaded and is ready to read or write at any given moment and an empty queue. If this drive has to do any error correction, it will quickly exceed this aggressive time and report an error. This could cause the OS to think data corruption has occurred and rewrite things, fail to open things, etc. It may make a user think the drive has gone bad, but that may not be the case at all....it could just be loading the heads or correcting data that is still correctable. If it is this aggressive, it is more likely to create data loss since the drive is no longer recovering the correctable data and moving corrections to a different system (like the file system).

This would be way too aggressive of an action...I highly recommend leaving it where it is at now at 30 seconds. That is plenty of time and should work well with the OS's timeouts it is setting.

There have been other changes such as NCQ PRIO or a way to say when a command has a higher priority so the drive's queue can become more like a priority queue rather than a more basic queue that just lets the drive figure out how to service it as quickly as possible.

The latest change to SAS and SATA and recovery times is CDL or Command Duration Limits. This feature takes a lot of what enterprises/data centers have learned over time as well as past things like TLC and ERC to create this new method.
In existing features (ERC/TLC) these affect every command.
With CDL, now there are 7 different classes/buckets you can configure for reads and writes separately.
Each bucket is essentially a different "priority" defined by the host (OS/datacenter/file system/etc).
Each command is issued specifying which one of these "buckets" it should follow for timing as well as how to complete the command.
If time runs out, what should the drive do? Abort? Pass? Fail? etc. There are a few different actions that can be assigned separately for each one of these buckets.

CDL is still rolling out today and being developed, but new products do have some support for it and latest Linux kernels are supporting it as well. Even FIO has some support for testing this now if you configure it.
We are rolling in a JSON based configuration file to set CDL in openSeaChest with our next release, but it only affects the drive side. Other OS side changes may be needed (as well as other supporting HBAs/etc).

I have attached the file you requested.

Thanks for sharing this! I will review it and figure out where the issue is since the reported values all look bogus in this mode.

I know this was very long and still likely confusing (sometimes it still confuses me), but I hope this helps explain it a little more...if not, let me know and I will do my best to answer more questions 😄

[vonericsen]

Alright, in the SCSI log you shared, the raw data the HBA is translating has the wrong endianness.
Looks like they straight copied SATA data (little endian) to the SAS fields (big endian) so it's outputting strange values because the translation is wrong.

I haven't figured out why it output standby Y instead of Z, but I'm still debugging that to figure it out.

[vonericsen]

@domotiquefacile,

Can you share more info about your HBA so I can try to report the endianness issue to Broadcom?
lspci -vvv will give you some info about it.
openSeaChest_PowerControl -d <handle> --llInfo will give driver version info.

What I will need is controller firmware version, driver revision, etc.
If you have Broadcom's storcli tool, that will report everything they could possibly need to look into this issue themselves. I don't recall the command to run to show controller info though.

[domotiquefacile]

Ok I reverted eh_timeout to 10 s to experiment.
And... I got the issue again (even with ERC error to 30 s).
I tried with different value of ERC error (10, 15, 25 and 30) but I can immediately reproduce the issue (by forcing standby_z and waking up).

So it does seem that eh_timeout has a key role.
I set eh_timeout to 30 s again and ERC error to 10 s, and I cannot reproduce the issue manually. I will wait again for one week to see how it goes. Perhaps I did something wrong before, we'll see.

Thanks a lot for your explanation, I understand now.
If the OS timeout value happens before the ERC error timeout, we should have an issue logged then.
Now I understand what I observe.

Here is the result of lspci -vvv:
lspci.txt

And here is the result of openSeaChest_PowerControl -d /dev/sda --llInfo

==========================================================================================
 openSeaChest_PowerControl - openSeaChest drive utilities - NVMe Enabled
 Copyright (c) 2014-2024 Seagate Technology LLC and/or its Affiliates, All Rights Reserved
 openSeaChest_PowerControl Version: 3.4.0-6_2_0 X86_64
 Build Date: Jun  9 2024
 Today: Thu Dec 11 22:10:54 2025	User: root
==========================================================================================

/dev/sg2 - ST16000NM000J-2TW103 - ZR5DGQC2 - SN04 - ATA

---Low Level tDevice information---
	---Drive Info---
		media type: HDD
		drive type: ATA
		interface type: SCSI/SAS/FC/etc
		zoned type: not zoned
		---adapter info---
			PCI/PCIe:
			VendorID: 1000h
			ProductID: 00E6h
			Revision: 0000h
		---driver info---
			driver name: mpt3sas
			driver version string: 48.100.00.00

				major ver: 48
				minor ver: 100
				revision: 0
		---ata flags---
		SCSI Version: 6
		---Passthrough Hacks---
			Passthrough type: SAT/system/none
				---SCSI Hacks---
				---NVMe Hacks---
				---ATA Hacks---
	---OS Info---
		handle name: /dev/sg2
		friendly name: sg2
		minimum memory alignment: 8
		---Linux Unique info---
			FD is valid
			Second Handle name: /dev/sda
			Second Handle friendly name: sda
			SG Driver Version:
				Major: 3
				Minor: 5
				Revision: 36
		OS read-write recommended: false
		last recorded error: 0
		File system Info:
			No active file systems detected

[domotiquefacile]

Since last time I am using SCT ERC to 10s and eh_timeout to 15s and I haven't got any more warning or errors in the log.