Description
Description:
Since the UEK7u2, SR-IOV is broken for Mellanox ConnectX4, if one or both ports are configured for ethernet mode.
The same works fine if the ports are configure for IB mode.
Additionally, everything is fine on UEK7u1 regardless of the configuration.
Diagnostic info:
Port 0 is in IB mode
Port 1 is in ETH mode
Output of lspci :
lspci |grep Mellanox
0b:00.0 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]
0b:00.1 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]
0b:00.2 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
0b:00.3 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
0b:00.4 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
0b:00.5 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
0b:00.6 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
0b:00.7 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
Output of from dmesg:
dmesg |grep mlx
[ 1.349024] mlx5_core 0000:0b:00.0: firmware version: 12.28.2006
[ 1.349050] mlx5_core 0000:0b:00.0: 63.008 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x8 link)
[ 4.071873] mlx5_core 0000:0b:00.0: Port module event: module 0, Cable plugged
[ 4.265254] mlx5_core 0000:0b:00.1: firmware version: 12.28.2006
[ 4.265304] mlx5_core 0000:0b:00.1: 63.008 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x8 link)
[ 4.683438] mlx5_core 0000:0b:00.1: E-Switch: Total vports 10, per vport: max uc(1024) max mc(16384)
[ 4.687324] mlx5_core 0000:0b:00.1: Port module event: module 1, Cable plugged
[ 4.932149] mlx5_core 0000:0b:00.1: Supported tc offload range - chains: 4294967294, prios: 4294967295
[ 4.941115] mlx5_core 0000:0b:00.1: MLX5E: StrdRq(0) RqSz(1024) StrdSz(256) RxCqeCmprss(0 basic)
[ 4.943224] mlx5_core 0000:0b:00.1 enp11s0f1np1: renamed from eth0
[ 5.655608] mlx5_core 0000:0b:00.1: Successfully registered panic handler for port 1
[ 5.773746] mlx5_core 0000:0b:00.1: mlx5_cmd_out_err:803:(pid 814): QUERY_HCA_CAP(0x100) op_mod(0x40) failed, status bad parameter(0x3), syndrome (0x5add95), err(-22)
[ 5.787044] mlx5_core 0000:0b:00.1: mlx5_device_enable_sriov:82:(pid 814): failed to enable eswitch SRIOV (-22)
[ 5.787047] mlx5_core 0000:0b:00.1: mlx5_sriov_enable:168:(pid 814): mlx5_device_enable_sriov failed : -22
[ 5.787222] mlx5_core 0000:0b:00.1 mlxen0: renamed from enp11s0f1np1
[ 6.213460] mlx5_core 0000:0b:00.0 mlxib0: renamed from ib0
[ 7.054459] mlx5_core 0000:0b:00.1 mlxen0: Link up
[ 7.057046] 8021q: adding VLAN 0 to HW filter on device mlxen0
[ 8.055388] IPv6: ADDRCONF(NETDEV_CHANGE): mlxen0: link becomes ready
[ 8.284101] IPv6: ADDRCONF(NETDEV_CHANGE): mlxib0: link becomes ready
[ 8.557689] IPv6: ADDRCONF(NETDEV_CHANGE): mlxib0: link becomes ready
[ 8.580025] br1: port 1(mlxen0) entered blocking state
[ 8.580028] br1: port 1(mlxen0) entered disabled state
[ 8.580067] device mlxen0 entered promiscuous mode
[ 8.580957] br1: port 1(mlxen0) entered blocking state
[ 8.580960] br1: port 1(mlxen0) entered listening state
[ 8.595823] mlx5_core 0000:0b:00.1: mlx5e_fs_set_rx_mode_work:843:(pid 156): S-tagged traffic will be dropped while C-tag vlan stripping is enabled
[ 10.635172] br1: port 1(mlxen0) entered learning state
[ 25.931048] br1: port 1(mlxen0) entered forwarding state
Manually trying to add VFs to the ETH port results with the following error:
echo 0 > /sys/class/net/mlxen0/device/sriov_numvfs
echo 7 > /sys/class/net/mlxen0/device/sriov_numvfs
-bash: echo: write error: Invalid argument
The same works just fine for the IB port:
echo 0 > /sys/class/net/mlxib0/device/sriov_numvfs
echo 7 > /sys/class/net/mlxib0/device/sriov_numvfs