Test SATA adapter (I/O Crest 4 port Marvell 9215)

[geerlingguy]

6by9 on the Pi Forums mentioned:

For those wanting to know about PCI-e compatibility, I have one here with a Pericom PI7C9X 1 to 3 way PCI-e bridge, and Marvell 9215 4 port SATA card connected to that. (My VL805 USB3 card is still to be delivered). With a couple of extra kernel modules enabled (mainly CONFIG_ATA, and CONFIG_SATA_AHCI) it's the basis of my next NAS.

<24W with a pair of 8TB SATA drives spinning and a 240GB SSD. <10W with the spinning rust in standby.

I bought this I/O Crest 4 Port SATA III PCIe card and would like to see if I can get a 4-drive RAID array going:

Relevant Links:

• Gist: Increase the BAR memory address space for PCIe devices on CM4
• Pi Forum Topic: BAR space for PCIe allocation on CM4?
• Nicer cables: Silverstone Tek Ultra Thin Lateral 90 Degree SATA Cables with Custom Low-Profile Connectors (CP11) (thanks to this comment)

[geerlingguy]

I'm going to try out the IO Crest 4-port SATA adapter.

[geerlingguy]

It has arrived!

[geerlingguy]

And... I just realized I have no SATA power supply cable, just the data cable. So I'll have to wait for one of those to come in before I can actually test one of my SATA drives.

[geerlingguy]

First light is good:

$ lspci

01:00.0 SATA controller: Marvell Technology Group Ltd. Device 9215 (rev 11) (prog-if 01 [AHCI 1.0])
	Subsystem: Marvell Technology Group Ltd. Device 9215
	Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 0
	Region 0: I/O ports at 0000
	Region 1: I/O ports at 0000
	Region 2: I/O ports at 0000
	Region 3: I/O ports at 0000
	Region 4: I/O ports at 0000
	Region 5: Memory at 600040000 (32-bit, non-prefetchable) [size=2K]
	Expansion ROM at 600000000 [size=256K]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit-
		Address: 00000000  Data: 0000
	Capabilities: [70] Express (v2) Legacy Endpoint, MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0, Latency L0s <1us, L1 <8us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <512ns, L1 <64us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [e0] SATA HBA v0.0 BAR4 Offset=00000004
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-

[geerlingguy]

Though dmesg shows that it's hitting BAR default address space limits again:

[    0.925795] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges:
[    0.925818] brcm-pcie fd500000.pcie:   No bus range found for /scb/pcie@7d500000, using [bus 00-ff]
[    0.925884] brcm-pcie fd500000.pcie:      MEM 0x0600000000..0x0603ffffff -> 0x00f8000000
[    0.925948] brcm-pcie fd500000.pcie:   IB MEM 0x0000000000..0x00ffffffff -> 0x0100000000
[    0.953526] brcm-pcie fd500000.pcie: link up, 5 GT/s x1 (SSC)
[    0.953827] brcm-pcie fd500000.pcie: PCI host bridge to bus 0000:00
[    0.953844] pci_bus 0000:00: root bus resource [bus 00-ff]
[    0.953866] pci_bus 0000:00: root bus resource [mem 0x600000000-0x603ffffff] (bus address [0xf8000000-0xfbffffff])
[    0.953933] pci 0000:00:00.0: [14e4:2711] type 01 class 0x060400
[    0.954172] pci 0000:00:00.0: PME# supported from D0 D3hot
[    0.957560] PCI: bus0: Fast back to back transfers disabled
[    0.957582] pci 0000:00:00.0: bridge configuration invalid ([bus ff-ff]), reconfiguring
[    0.957802] pci 0000:01:00.0: [1b4b:9215] type 00 class 0x010601
[    0.957874] pci 0000:01:00.0: reg 0x10: [io  0x8000-0x8007]
[    0.957911] pci 0000:01:00.0: reg 0x14: [io  0x8040-0x8043]
[    0.957947] pci 0000:01:00.0: reg 0x18: [io  0x8100-0x8107]
[    0.957984] pci 0000:01:00.0: reg 0x1c: [io  0x8140-0x8143]
[    0.958021] pci 0000:01:00.0: reg 0x20: [io  0x800000-0x80001f]
[    0.958058] pci 0000:01:00.0: reg 0x24: [mem 0x00900000-0x009007ff]
[    0.958095] pci 0000:01:00.0: reg 0x30: [mem 0x00000000-0x0003ffff pref]
[    0.958262] pci 0000:01:00.0: PME# supported from D3hot
[    0.961586] PCI: bus1: Fast back to back transfers disabled
[    0.961605] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
[    0.961674] pci 0000:00:00.0: BAR 8: assigned [mem 0x600000000-0x6000fffff]
[    0.961698] pci 0000:01:00.0: BAR 6: assigned [mem 0x600000000-0x60003ffff pref]
[    0.961722] pci 0000:01:00.0: BAR 5: assigned [mem 0x600040000-0x6000407ff]
[    0.961744] pci 0000:01:00.0: BAR 4: no space for [io  size 0x0020]
[    0.961759] pci 0000:01:00.0: BAR 4: failed to assign [io  size 0x0020]
[    0.961774] pci 0000:01:00.0: BAR 0: no space for [io  size 0x0008]
[    0.961788] pci 0000:01:00.0: BAR 0: failed to assign [io  size 0x0008]
[    0.961803] pci 0000:01:00.0: BAR 2: no space for [io  size 0x0008]
[    0.961817] pci 0000:01:00.0: BAR 2: failed to assign [io  size 0x0008]
[    0.961831] pci 0000:01:00.0: BAR 1: no space for [io  size 0x0004]
[    0.961845] pci 0000:01:00.0: BAR 1: failed to assign [io  size 0x0004]
[    0.961860] pci 0000:01:00.0: BAR 3: no space for [io  size 0x0004]
[    0.961873] pci 0000:01:00.0: BAR 3: failed to assign [io  size 0x0004]
[    0.961891] pci 0000:00:00.0: PCI bridge to [bus 01]
[    0.961914] pci 0000:00:00.0:   bridge window [mem 0x600000000-0x6000fffff]
[    0.962217] pcieport 0000:00:00.0: enabling device (0140 -> 0142)
[    0.962439] pcieport 0000:00:00.0: PME: Signaling with IRQ 55
[    0.962813] pcieport 0000:00:00.0: AER: enabled with IRQ 55

[geerlingguy]

I just increased the BAR allocation following the directions in this Gist, but when I rebooted (without the card in), I got:

[    0.926161] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges:
[    0.926184] brcm-pcie fd500000.pcie:   No bus range found for /scb/pcie@7d500000, using [bus 00-ff]
[    0.926247] brcm-pcie fd500000.pcie:      MEM 0x0600000000..0x063fffffff -> 0x00c0000000
[    0.926312] brcm-pcie fd500000.pcie:   IB MEM 0x0000000000..0x00ffffffff -> 0x0100000000
[    1.521386] brcm-pcie fd500000.pcie: link down

Powering off completely, then booting again, it works. So note to self: if you get a link down, try a hard power reset instead of reboot.

[geerlingguy]

Ah... looking closer, those 'failed to assign' errors are for IO BARs, which are unsupported on the Pi.

So... I posted in the BAR space thread on Pi Forums asking 6by9 if that user has had the same logs and if they can be safely ignored. Still waiting on a way to power my drive so I can do an end-to-end test :)

[kitlith]

something else that may be interesting is if you can get a sas adapter/raid card working. I know I was looking into SBCs w/ pcie awhile back for the purpose of building a low power/low heat host for some sas drives I have. (ended up just throwing it in a computer and not running 24/7)

[geerlingguy]

That would be an interesting thing to test, though it'll have to wait a bit as I'm trying to get through some other cards and might also test 2.5 Gbps or 5 Gbps networking if I am able to!

[geerlingguy]

Without the kernel modules enabled, lsblk shows no device:

$ lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
mmcblk0     179:0    0 29.8G  0 disk 
├─mmcblk0p1 179:1    0  256M  0 part /boot
└─mmcblk0p2 179:2    0 29.6G  0 part /

Going to try adding those modules and see what happens!

[geerlingguy]

# Install dependencies
sudo apt install -y git bc bison flex libssl-dev make libncurses5-dev

# Clone source
git clone --depth=1 https://github.com/raspberrypi/linux

# Apply default configuration
cd linux
export KERNEL=kernel7l # use kernel8 for 64-bit, or kernel7l for 32-bit
make bcm2711_defconfig

# Customize the .config further with menuconfig
make menuconfig
# Enable the following:
# Device Drivers:
#   -> Serial ATA and Parallel ATA drivers (libata)
#     -> AHCI SATA support
#     -> Marvell SATA support
#
# Alternatively add the following in .config manually:
# CONFIG_ATA=m
# CONFIG_ATA_VERBOSE_ERROR=y
# CONFIG_SATA_PMP=y
# CONFIG_SATA_AHCI=m
# CONFIG_SATA_MOBILE_LPM_POLICY=0
# CONFIG_ATA_SFF=y
# CONFIG_ATA_BMDMA=y
# CONFIG_SATA_MV=m

nano .config
# (edit CONFIG_LOCALVERSION and add a suffix that helps you identify your build)

# Build the kernel and copy everything into place
make -j4 zImage modules dtbs # 'Image' on 64-bit
sudo make modules_install
sudo cp arch/arm/boot/dts/*.dtb /boot/
sudo cp arch/arm/boot/dts/overlays/*.dtb* /boot/overlays/
sudo cp arch/arm/boot/dts/overlays/README /boot/overlays/
sudo cp arch/arm/boot/zImage /boot/$KERNEL.img

[geerlingguy]

Yahoo, it worked!

$ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda           8:0    1 223.6G  0 disk 
├─sda1        8:1    1   256M  0 part /media/pi/boot
└─sda2        8:2    1 223.3G  0 part /media/pi/rootfs
mmcblk0     179:0    0  29.8G  0 disk 
├─mmcblk0p1 179:1    0   256M  0 part /boot
└─mmcblk0p2 179:2    0  29.6G  0 part /

[geerlingguy]

Repartitioning the drive:

sudo fdisk /dev/sda
d 1    # delete partition 1
d 2    # delete partition 2
n    # create new partition
p    # primary (default)
1    # partition 1 (default)
2048    # First sector (default)
468862127    # Last sector (default)
w    # write new partition table

Got the following:

The partition table has been altered.
Failed to remove partition 1 from system: Device or resource busy
Failed to remove partition 2 from system: Device or resource busy
Failed to add partition 1 to system: Device or resource busy

The kernel still uses the old partitions. The new table will be used at the next reboot. 
Syncing disks.

Rebooted the Pi, then:

$ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda           8:0    1 223.6G  0 disk 
└─sda1        8:1    1 223.6G  0 part 
mmcblk0     179:0    0  29.8G  0 disk 
├─mmcblk0p1 179:1    0   256M  0 part /boot
└─mmcblk0p2 179:2    0  29.6G  0 part /

To format the device, use mkfs:

$ sudo mkfs.ext4 /dev/sda1
mke2fs 1.44.5 (15-Dec-2018)
Discarding device blocks: done                            
Creating filesystem with 58607510 4k blocks and 14655488 inodes
Filesystem UUID: dd4fa95d-edbf-4696-a9e1-ddf1f17da580
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
	4096000, 7962624, 11239424, 20480000, 23887872

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (262144 blocks): done
Writing superblocks and filesystem accounting information: done 

Then mount it somewhere:

$ sudo mkdir /mnt/sata-sda
$ sudo mount /dev/sda1 /mnt/sata-sda
$ mount
...
/dev/sda1 on /mnt/sata-sda type ext4 (rw,relatime)

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
...
/dev/sda1       220G   61M  208G   1% /mnt/sata-sda