Skip to content

Add SR-IOV doc for shift-on-stack #28418

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed

Add SR-IOV doc for shift-on-stack #28418

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1b271d2
Add SR-IOV doc for shift-on-stack
zshi-redhat Jan 7, 2021
b44901c
Apply edits
jboxman Jan 8, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 34 additions & 6 deletions modules/nw-sriov-networknodepolicy-object.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,10 @@ spec:
deviceID: "<device_id>" <10>
pfNames: ["<pf_name>", ...] <11>
rootDevices: ["<pci_bus_id>", "..."] <12>
deviceType: <device_type> <13>
isRdma: false <14>
linkType: <link_type> <15>
netFilter: "<filter_string>" <13>
Copy link
Contributor Author

@zshi-redhat zshi-redhat Jan 8, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like the indent is not perfect yet, we need to keep it the same as other selectors like rootDevices.

Copy link

@atyronesmith atyronesmith Jan 8, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indent still doesn't look correct?

deviceType: <device_type> <14>
isRdma: false <15>
linkType: <link_type> <16>
----
<1> The name for the CR object.

Expand Down Expand Up @@ -60,22 +61,24 @@ If you specify both `pfNames` and `rootDevices` at the same time, ensure that th

<12> An array of one or more PCI bus addresses for the PF of the device. Provide the address in the following format: `0000:02:00.1`.

<13> Optional: The driver type for the virtual functions. The only allowed values are `netdevice` and `vfio-pci`. The default value is `netdevice`.
<13> Optional: The platform specific filter string. The only supported platform is {rh-openstack-first}. Acceptable values are of the following format: `openstack/NetworkID:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx`. Replace `xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx` with the value from the `/var/config/openstack/latest/network_data.json` metadata file.
Copy link
Contributor Author

@zshi-redhat zshi-redhat Jan 8, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice change!

Copy link

@atyronesmith atyronesmith Jan 8, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In addition, the administrator should also know the OpenStack Network UUID apriori as part of network planning. The OpenStack networks are created to glue together the different VNF/CNF components in the system and have been architected to carry the necessary information.


<14> Optional: The driver type for the virtual functions. The only allowed values are `netdevice` and `vfio-pci`. The default value is `netdevice`.
+
[NOTE]
====
For a Mellanox card to work in Data Plane Development Kit (DPDK) mode on bare metal nodes, use the `netdevice` driver type and set `isRdma` to `true`.
====

<14> Optional: Whether to enable remote direct memory access (RDMA) mode. The default value is `false`.
<15> Optional: Whether to enable remote direct memory access (RDMA) mode. The default value is `false`.
+
[NOTE]
====
If the `isRDMA` parameter is set to `true`, you can continue to use the RDMA enabled VF as a normal network device.
A device can be used in either mode.
====

<15> Optional: The link type for the VFs. You can specify one of the following values: `eth` or `ib`. `eth` is ethernet and `ib` is InfiniBand. The default value is `eth` if it is not explicitly set. When `linkType` is set to `ib`, `isRdma` will be set to `true` by SR-IOV Network Operator webhook automatically. When `linkType` is set to `ib`, `deviceType` should not be set to `vfio-pci`.
<16> Optional: The link type for the VFs. You can specify one of the following values: `eth` or `ib`. `eth` is ethernet and `ib` is InfiniBand. The default value is `eth` if it is not explicitly set. When `linkType` is set to `ib`, `isRdma` will be set to `true` by SR-IOV Network Operator webhook automatically. When `linkType` is set to `ib`, `deviceType` should not be set to `vfio-pci`.

[id="sr-iov-network-node-configuration-examples_{context}"]
== SR-IOV network node configuration examples
Expand Down Expand Up @@ -103,3 +106,28 @@ spec:
linkType: ib
isRdma: true
----

The following example describes the configuration for an SR-IOV device in a {rh-openstack} virtual machine:
Copy link
Contributor Author

@zshi-redhat zshi-redhat Jan 8, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed the platform is using {rh-openstack-first} in <13>, but this one is {rh-openstack}, is there any difference?


.Example configuration for an SR-IOV device in a virtual machine
[source,yaml]
----
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
name: policy-sriov-net-openstack-1
namespace: openshift-sriov-network-operator
spec:
resourceName: sriovnic1
nodeSelector:
feature.node.kubernetes.io/network-sriov.capable: "true"
numVfs: 1 <1>
nicSelector:
vendor: "15b3"
Copy link

@atyronesmith atyronesmith Jan 8, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 'vendor' and 'deviceID' fields are mostly superfluous as there real match is to the UUID.

Copy link
Contributor Author

@zshi-redhat zshi-redhat Jan 11, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@atyronesmith I don't have environment to test, if we remove vendor/device IDs from policy, will it expose virtio or other non-sriov devices as k8s resource when the given network ID matches ?

Copy link

@atyronesmith atyronesmith Jan 11, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My environment is down now as well.... However, if I remember correctly (and I look at https://github.com/openshift/sriov-network-operator/blob/f6aba07546d2c85b319bf3bfeb9deb45a2f007f5/api/v1/helper.go#L429)

A non-zero netFilter will only match that specific netfiler.

Copy link

@mikemckiernan mikemckiernan Feb 1, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@atyronesmith , Jason asked me to do light wordsmithing on this PR and then get it merged. Regarding these two comments about the nicSelector:

Should netFilter be aligned with rootDevices? (I'm guessing yes if the same receiver function can handle both in the helper.go file, but I'm using my imagination.)

The 'vendor' and 'deviceID' fields are mostly superfluous as there real match is to the UUID.

Do you want the doc to suggest using netFilter because it is the most specific? Depending on your answer to this question and if you indicate that netFilter is subordinate to nicSelector, then netFilter deserves some mention in the callout for #8, nicSelector.

Copy link

@atyronesmith atyronesmith Feb 1, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @mikemckiernan, netFilter should be used as it is the most specific. RootDevices does not need to be used as netFilter is specific. NetFilter is subordinate to nicSelector and is at the same level as rootDevices, vendor, devicdeID, etc...

deviceID: "101b"
netFilter: "openstack/NetworkID:ea24bd04-8674-4f69-b0ee-fa0b3bd20509" <2>
----

<1> The `numVfs` field is always set to `1` when configuring the node network policy in a virtual machine environment.

<2> On {rh-openstack} the `netFilter` field must refer to a network ID. Valid values for `netFilter` are available from a `SriovNetworkNodeState` object.