Test various M.2-key NVMe SSD drives

[geerlingguy]

I have on hand and would like to test the following with the Pi via an M.2 adapter, to come around full circle on the video I posted in Summer 2020: https://youtu.be/3yTyhR0Adao:

• Samsung 970 EVO Plus NVMe M.2
• WD Black SN750 NVMe SSD
• XPG SX6000 Lite SSD
• Kingston SA400 SSD

I already tested the 970 EVO Plus and the XPG SX6000 for the Raspberry Pi Compute Module 4 Review, but I'd like to go more in depth, and compare all the cards in terms of whether they work direct through a PCIe adapter, and how fast they are using the disk-benchmark.sh script.

[geerlingguy]

Besides general performance, I'd also like to discuss booting off an NVMe drive—specifically, since direct booting from one is not supported, how do you get the Pi to initialize off a microSD card then finish the boot off an NVMe?

[Magnum-Pl]

I tested a 980 Pro and wasn't able to get it to work. I tried a 970 EVO first and was able to see that using lspci and lsblk. The Pi won't even boot with the 980 attached. I can plug it in after the Pi boots, but I can't see it with lspci or lsblk.

I used a Cablecc PCI-E 3.0 x4 M.2 NGFF M-Key NVME adapter

[geerlingguy]

@Magnum-Pl thanks for sharing. Weird that the 980 would behave any different. Did you confirm it works in another computer or USB adapter? Also, what capacity is it? I've only been testing with 512GB or less, if that makes any difference.

[Magnum-Pl]

They're both 500GB. The 980 works fine with my Mac and PC. I've been using this Wanfocyu USB-C NVMe adapter. They're both formatted NTFS. I tried FAT32 on the 980, but that didn't work either. The adapter doesn't work when plugged into the USB 2 ports on the IO board, but I have a Syba SD-PEX20199 on the way.

[garlick]

I can verify that the Samsung 970 EVO (without the "plus") also works great.

I'm using this Simtech adapter.

[ThomasKaiser]

how fast they are using the disk-benchmark.sh script.

Just a quick note: Based on a lot of testing years ago with RK3399 boards and NVMe SSDs I came to the conclusion that different settings are needed to test fast SSDs compared to rather slow SD cards.

hdparm is a problem in itself since it uses a hardcoded block size of 128K which was fine when this was defined (last century) but doesn't represent today's access patterns any more. Simply compare hdparm numbers with the results from iozone -e -I -a -s 100M -r 4k -r 128k -r 16384k -i 0 -I 1.

And when using iozone to test for sequential transfer rates I found that 100M with iozone is not enough with SSDs which can be easily observed comparing a 100M run with 1000M.

[geerlingguy]

@ThomasKaiser - Just noting here (also noted in another issue) that I updated the disk-benchmark.sh script quite a bit, to use much larger sequential size for the testing, and a different set of tests all based on fio and iozone (discarding dd/hdparm).

See: https://github.com/geerlingguy/raspberry-pi-dramble/blob/master/setup/benchmarks/disk-benchmark.sh

Also, I've been doing a bunch of testing with a WD_Black SN750 500 GB drive in the MirkoPC, and one interesting thing to note is it consumes a good bit of power at full tilt. I wish the MirkoPC accepted more than 5v/3A power input...

[t5b6de]

I wanna test a nvme samsung 950 pro 250gb parallel with a vl805 4 port usb card connected to a 2 port pcie switch.

if someone is interested into that results. i may reply to that posts here. Maybe about end of the week i should have some first results

[Johnnie390]

Trying to test a Samsung 980 Pro* as NVME boot device on a CM4 Lite. At boot - "cannot open nvme", EEPROM is up to date (19.05.21).
A Synology SNV3400-400G NVME SSD boots as expected using the abovementioned EEPROM.

• This NVME SSD works elsewhere (RPI, x86 Linux) flawlessly.

[t5b6de]

You to not have any other pcie switch or else inbetween?

to my tests:

With the pcie switch plugged in, i cannot boot from the nvme nor the usb stick connected to the card.
If i plug in the card directly into that slot, the rpi firmware can boot of it without any problems.

My pcie to m.2 adapter does not fit into that x1 slot on the pi board, its a x4 pcb. So i cannot test it directly.
Booting is from sd card, which modified cmdline.txt which points to /dev/nvme0n1p1 where / is located.

Performance with a real anvme (even over a pcie switch) is awesome. never thought that a raspberry pi can run that smoothly.

about 420MiB/s are the limit through the PCIe switch. And all does work out of the box, i do not have to change anything regarding to the kernel itself or device tree.

Kind regards.