Exos X18 18TB FastFormat to 4kn failed, drive bricked?

[ManChicken1911]

I am about to build a home NAS and got 8 x ST18000NM000J 18TB drives. Figured I'd do some pre-prep and using Seachest_Lite from the Seagate website, running in Linux, I successfully got 7 of them changed over to 4kn. But one of them failed, giving an error after the ~1 min the others normally took. The drive reset at what was usually the end of the process, but then ultimately spun down.

I can still sort-of talk to the drive, but for instance when I ask it --showSupportedFormats, 4k sectors are no longer on the list. Powering everything off, when I turn just the drive on (I'm powering it externally but am doing this hooked up SATA direct to a Zimaboard) it spins up and does it's initializing noises, then stops and remains spinning indefinitely. But when I power up the computer, as soon as the UEFI tries to probe the drive, it resets, half spins down, spins back up, makes some more init noises, then spins down and hangs the system until it times out. Linux boots but spits out tons of errors trying to talk to the drive, but again I can still kind of access it with the tools even though it's not spinning -- it even inexplicably let me update the firmware, which it claims succeeded and shows the right version when I ask it, but otherwise is in this semi-bricked state.

So, the question is -- is this thing seriously now useless with no way to recover it, just from trying to switch it to 4kn?! The drive firmware seems to be running, it just seems crazy that there's no way (that I can see) to convince this thing back into working, but found this Github and am hoping someone might have some ideas or insight (I'm more curious as to the technical reason why this is such a delicate procedure, whether it's salvageable or not.)

[vonericsen]

Hi @ManChicken1911,

Have you tried running the --setSectorSize option again?
That should attempt to restart the format and correct the issue, but if you run into any issues trying to run that, can you share the verbose output with me?
example: openSeaChest_Format -d <handle> --setSectorSize 4096 -v 4 | tee set4knVerbose.txt 

[ManChicken1911]

Trying the format again doesn't work - "Setting sector size not supported on this device."

Attached is the log. Thanks for responding & looking!
set4knVerbose.txt

[vonericsen]

Thank you for sharing the log.
It looks like the drive is not responding to one of the commands that is issued to check if the feature is supported before issuing the command.
Let me add a workaround for this to force it past this point to see if it helps get this corrected for you.
I'll share a binary on this discussion for you to try once I have one built from the CI.

[vonericsen]

Can you please try this build with the following options?
openSeaChest_Format -d <handle> --setSectorSize 4096 --force
This --force flag will bypass the "is this supported" check in the code to try and force issuing the command. If that fails still fails the force flag will drop into the automatic recovery code to try and recover from the issue as well.

openSeaChest-feature-more_utilities-linux-x86_64-portable.zip

If you still have issues after this, can you share the -v 4 log with the force flag again?

[ManChicken1911]

No such luck. I just get back:

Sense Information:
Fixed format, current; Sense key: Illegal Request
Additional sense: Unaligned write command

Guess it's probably time to just send these back. I live in CO, too bad they don't take walk-ins in Longmont! 😆

However the $64 question is, once I get replacements, what is the best way to try and get them successfully flipped to 4kn? I'm already using the Seagate bootable USB stripped down Linux environment; should I try in Windows instead? I thought for sure that'd be far more dangerous but I dunno at this point...

Once again I appreciate all your effort on this.

[vonericsen]

I'm sorry that did not work. I've been thinking really hard about this issue and what else can possibly be done to help prevent it.
On the branch feature/hardening I've been working on a lot of security related improvements, one of which is file permissions.
One more thing I can try is different permissions when opening the drive handle or changing the permissions before this command is issued.
I don't want you to experiment and test this, so I need some time to try this on some drives myself to see if it works or not. There are a few other less destructive commands that are affected by similar issues that this could be tested with too (ATA Security erase is the big one).

I live in CO, too bad they don't take walk-ins in Longmont! 😆

😆

However the $64 question

For me, this issue is more like a $1M question. I have been trying to figure out the best way to ensure this command goes uninterrupted both from the software and from general advice overall.
I've heard some people use a live Ubuntu USB and run these tools from that live environment.
The bootable USB is what I would generally recommend since it uses Tinycore Linux which is running off a ram disk, but if that is still causing trouble with this, then I'm not sure of a better thing to try.

Let me work on the permissions flags and do some testing to see if it is helping on my end, then I can build something to share on here.

[ManChicken1911]

Just a follow-up on my whole saga 😄

I got my replacement drives back (finally) from Seagate, and this time I built a little Windows PE boot disk with Open Seatools in it, and successfully used that to set them to 4Kn. Hurray!

[vonericsen]

That is great news! I'm glad you were able to get that to work.

I'm still doing the best I can to investigate what else we can to do try and prevent this from happening in software for any other users.

Do you have any instructions you followed to create the Windows PE disk that you can share?

[ManChicken1911]

I basically just followed this guide: https://www.elevenforum.com/t/winpe-create-a-custom-windows-install-usb.4804/

I skipped steps 2.3 through 2.7 as the goal was not to make a custom Windows installer, just a PE environment to run the SeaChest tools from. Instead I copied all the SeaChest EXEs into a Program Files\SeaChest directory in the mounted virtual disk. Then, picked back up on step 2.8 to save and dismount the virtual disk and create the .iso file from it. Lastly I skipped parts 4 and 5 entirely and just used Rufus to write the .iso onto a USB stick.

Booting off the USB, you'll automatically get dumped into an Administrator command prompt, and can find all the SeaChest tools X:\Program Files\SeaChest

[fl0wfr]

Having the same exact issue with a brand new WD WUH721414AL5200.

[vonericsen]

Hi @fl0wfr,

Which build of the tools did you use when you ran this?
I have a release nearly ready that addresses the interruptions from other software which should mitigate many of the issues seen while changing sector size.

[fl0wfr]

Hi @vonericsen,
I used version 24.00.1-1. I finally managed to switch to 4Kn by using "--formatUnit 4096", but it's strange, I didn't have the issue with another WD 14TB SAS drive.

[vonericsen]

Got it!
The problem that happens during fast format as I have seen it using bus analyzers is that something interrupts the format with a reset while it is running the format. Depending on when the format is interrupted, it may cause the drive to enter a bad state or format corrupt state. On SAS you can usually correct this by running a full format (--formatUnit current in openSeaChest_Format), but on SATA there is no equivalent command which is why Seagate has a vendor unique command that we try to run to recover if something does go wrong.
So go ahead and try that full format to recover....sometimes retrying a fast format will be enough, but there is a lot of "it depends" situations here. The downside to a full formatunit is it typically takes ~2hours per terabyte to complete.

The reset trigger has been difficult to track down and find a solution to, but the release (which I think I will publish tomorrow) introduces a fix I've found by opening the handle with exclusive access.
I was finally able to find a system that repeated the interruptions regularly enough to find this as a workable solution....it always seemed random before then.
We found that smartd and udisks2 services are running constantly in the background and either by a timer or some other method get triggered and when they try accessing a busy disk, the system sends a reset.

I don't think this solves all the problems as some systems may have additional management hardware that this will not block, but it should greatly reduce these interruptions.