Skip to content

Conversation

@PlaidCat
Copy link
Collaborator

Update process (This kernel CentOS base for 5.14.0-570)

  • Kernel History Rebuild Process for all src.rpms hosted by RESF
  • Create sig-cloud-9/5.14.0-570.X.1.el9_6 branch
  • Check if any maintained code is included in the new el release.
  • Cherry-pick all code from previous branch into new branch (skipping unneeded code)
    • Fix conflicts as they arise
  • Build and Test

Removed Commits

None

Forward Port Process

Note: this is a new experimental process that forces the FIPS protection scan AND have reduced excessive the output. There will soon be a PR to kernel-src-tree-tools

[jmaple@devbox code]$ cat RR.resf_kernel-5.14.0-570.55.1.el9_6.v2.log
[rolling release update] Rolling Product:  rlc-9
[rolling release update] Checking out branch:  rlc-9/5.14.0-570.52.1.el9_6
[rolling release update] Gathering all the RESF kernel Tags
[rolling release update] Found 23 RESF kernel tags
[rolling release update] Checking out branch:  rocky9_6
[rolling release update] Gathering all the RESF kernel Tags
[rolling release update] Found 24 RESF kernel tags
[rolling release update] Latest RESF tag sha:  b'c18abab0e029'
"c18abab0e0297b9f20d92e476a57f82b62b79ed3 Rebuild rocky9_6 with kernel-5.14.0-570.52.1.el9_6"
[rolling release update] Checking for FIPS protected changes between the common tag and HEAD
[rolling release update] Checking for FIPS protected changes
[rolling release update] Getting SHAS c18abab0e029..HEAD
[rolling release update] Number of commits to check:  15
[rolling release update] Checking modifications of shas
[rolling release update] Checked 1 of 15 commits
[rolling release update] Checked 2 of 15 commits
[rolling release update] Checked 3 of 15 commits
[rolling release update] Checked 4 of 15 commits
[rolling release update] Checked 5 of 15 commits
[rolling release update] Checked 6 of 15 commits
[rolling release update] Checked 7 of 15 commits
[rolling release update] Checked 8 of 15 commits
[rolling release update] Checked 9 of 15 commits
[rolling release update] Checked 10 of 15 commits
[rolling release update] Checked 11 of 15 commits
[rolling release update] Checked 12 of 15 commits
[rolling release update] Checked 13 of 15 commits
[rolling release update] Checked 14 of 15 commits
[rolling release update] Checked 15 of 15 commits
[rolling release update] 0 of 15 commits have FIPS protected changes
[rolling release update] Checking out old rolling branch:  rlc-9/5.14.0-570.52.1.el9_6
[rolling release update] Finding the CIQ Kernel and Associated Upstream commits between the last resf tag and HEAD
[rolling release update] Last RESF tag sha:  b'c18abab0e029'
[rolling release update] Total commits in old branch: 8
[rolling release update] Checking out new base branch:  rocky9_6
[rolling release update] Finding the kernel version for the new rolling release
[rolling release update] New Branch to create: rlc-9/5.14.0-570.55.1.el9_6
[rolling release update] Creating new branch: rlc-9/5.14.0-570.55.1.el9_6
[rolling release update] Creating new branch for PR:  jmaple_rlc-9/5.14.0-570.55.1.el9_6
[rolling release update] Creating Map of all new commits from last rolling release fork
[rolling release update] Total commits in new branch: 14
[rolling release update] Checking if any of the commits from the old rolling release are already present in the new base branch
[rolling release update] Found 0 duplicate commits to remove
[rolling release update] Applying 8 remaining commits to the new branch
  [1/8] 5e5d9d09daaf selftests/mm temporary fix of hmm infinite loop
  [2/8] 604bffbeede6 tools: hv: Enable debug logs for hv_kvp_daemon
  [3/8] 1ef4a26a3b90 scsi: storvsc: Increase the timeouts to storvsc_timeout
  [4/8] 43055cbbc89d Drivers: hv: Allow vmbus_sendpacket_mpb_desc() to create multiple ranges
  [5/8] 5e0479c96305 hv_netvsc: Use vmbus_sendpacket_mpb_desc() to send VMBus messages
  [6/8] 6b4a254162de hv_netvsc: Preserve contiguous PFN grouping in the page buffer array
  [7/8] 87948443885b hv_netvsc: Remove rmsg_pgcnt
  [8/8] 424dfec1a52b Drivers: hv: vmbus: Remove vmbus_sendpacket_pagebuffer()
[rolling release update] Successfully applied all 8 commits

BUILD

[jmaple@devbox code]$ egrep -B 5 -A 5 "\[TIMER\]|^Starting Build" $(ls -t kbuild* | head -n1)
  CLEAN   scripts/mod
  CLEAN   scripts/selinux/genheaders
  CLEAN   scripts/selinux/mdp
  CLEAN   scripts
  CLEAN   include/config include/generated arch/x86/include/generated .config .config.old .version Module.symvers certs/signing_key.pem certs/signing_key.x509 certs/x509.genkey
[TIMER]{MRPROPER}: 9s
x86_64 architecture detected, copying config
'configs/kernel-x86_64-rhel.config' -> '.config'
Setting Local Version for build
CONFIG_LOCALVERSION="-jmaple_rlc-9_5.14.0-570.55.1.el9_6-3f4cd04f0c2c"
Making olddefconfig
--
  HOSTCC  scripts/kconfig/util.o
  HOSTLD  scripts/kconfig/conf
#
# configuration written to .config
#
Starting Build
  SYSHDR  arch/x86/include/generated/uapi/asm/unistd_32.h
  SYSHDR  arch/x86/include/generated/uapi/asm/unistd_64.h
  SYSTBL  arch/x86/include/generated/asm/syscalls_32.h
  SYSHDR  arch/x86/include/generated/uapi/asm/unistd_x32.h
  SYSHDR  arch/x86/include/generated/asm/unistd_32_ia32.h
--
  LD [M]  sound/xen/snd_xen_front.ko
  BTF [M] sound/virtio/virtio_snd.ko
  BTF [M] sound/usb/usx2y/snd-usb-usx2y.ko
  BTF [M] sound/x86/snd-hdmi-lpe-audio.ko
  BTF [M] sound/xen/snd_xen_front.ko
[TIMER]{BUILD}: 1711s
Making Modules
  INSTALL /lib/modules/5.14.0-jmaple_rlc-9_5.14.0-570.55.1.el9_6-3f4cd04f0c2c+/kernel/arch/x86/crypto/blake2s-x86_64.ko
  INSTALL /lib/modules/5.14.0-jmaple_rlc-9_5.14.0-570.55.1.el9_6-3f4cd04f0c2c+/kernel/arch/x86/crypto/blowfish-x86_64.ko
  INSTALL /lib/modules/5.14.0-jmaple_rlc-9_5.14.0-570.55.1.el9_6-3f4cd04f0c2c+/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko
  INSTALL /lib/modules/5.14.0-jmaple_rlc-9_5.14.0-570.55.1.el9_6-3f4cd04f0c2c+/kernel/arch/x86/crypto/camellia-aesni-avx2.ko
--
  STRIP   /lib/modules/5.14.0-jmaple_rlc-9_5.14.0-570.55.1.el9_6-3f4cd04f0c2c+/kernel/sound/xen/snd_xen_front.ko
  SIGN    /lib/modules/5.14.0-jmaple_rlc-9_5.14.0-570.55.1.el9_6-3f4cd04f0c2c+/kernel/sound/virtio/virtio_snd.ko
  SIGN    /lib/modules/5.14.0-jmaple_rlc-9_5.14.0-570.55.1.el9_6-3f4cd04f0c2c+/kernel/sound/x86/snd-hdmi-lpe-audio.ko
  SIGN    /lib/modules/5.14.0-jmaple_rlc-9_5.14.0-570.55.1.el9_6-3f4cd04f0c2c+/kernel/sound/xen/snd_xen_front.ko
  DEPMOD  /lib/modules/5.14.0-jmaple_rlc-9_5.14.0-570.55.1.el9_6-3f4cd04f0c2c+
[TIMER]{MODULES}: 9s
Making Install
sh ./arch/x86/boot/install.sh 5.14.0-jmaple_rlc-9_5.14.0-570.55.1.el9_6-3f4cd04f0c2c+ \
        arch/x86/boot/bzImage System.map "/boot"
[TIMER]{INSTALL}: 24s
Checking kABI
kABI check passed
Setting Default Kernel to /boot/vmlinuz-5.14.0-jmaple_rlc-9_5.14.0-570.55.1.el9_6-3f4cd04f0c2c+ and Index to 0
Hopefully Grub2.0 took everything ... rebooting after time metrices
[TIMER]{MRPROPER}: 9s
[TIMER]{BUILD}: 1711s
[TIMER]{MODULES}: 9s
[TIMER]{INSTALL}: 24s
[TIMER]{TOTAL} 1759s
Rebooting in 10 seconds

KSelfTest

[jmaple@devbox code]$ ~/workspace/auto_kernel_history_rebuild/Rocky10/rocky10/code/get_kselftest_diff.sh
kselftest.5.14.0-jmaple_sig-cloud-9_5.14.0-570.30.1.el9_6-180c5309d27e+.log
317
kselftest.5.14.0-jmaple_sig-cloud-9_5.14.0-570.33.2.el9_6-b4790268d6a9+.log
317
kselftest.5.14.0-jmaple_sig-cloud-9_5.14.0-570.39.1.el9_6-f6a810230c4c+.log
318
kselftest.5.14.0-jmaple_rlc-9_5.14.0-570.55.1.el9_6-3f4cd04f0c2c+.log
318
Before: kselftest.5.14.0-jmaple_sig-cloud-9_5.14.0-570.39.1.el9_6-f6a810230c4c+.log
After: kselftest.5.14.0-jmaple_rlc-9_5.14.0-570.55.1.el9_6-3f4cd04f0c2c+.log
Diff:
No differences found.

PlaidCat and others added 8 commits October 24, 2025 17:41
jira SECO-170

In Rocky9 if you run ./run_vmtests.sh -t hmm it will fail and cause an
infinite loop on ASSERTs in FIXTURE_TEARDOWN()
This temporary fix is based on the discussion here
https://patchwork.kernel.org/project/linux-kselftest/patch/26017fe3-5ad7-6946-57db-e5ec48063ceb@suse.cz/#25046055

We will investigate further kselftest updates that will resolve the root
causes of this.

Signed-off-by: Jonathan Maple <jmaple@ciq.com>
Signed-off-by: Shreeya Patel <spatel@ciq.com>
Signed-off-by: Jonathan Maple <jmaple@ciq.com>
jira LE-3207
feature tools_hv
commit-author Shradha Gupta <shradhagupta@linux.microsoft.com>
commit a9c0b33

Allow the KVP daemon to log the KVP updates triggered in the VM
with a new debug flag(-d).
When the daemon is started with this flag, it logs updates and debug
information in syslog with loglevel LOG_DEBUG. This information comes
in handy for debugging issues where the key-value pairs for certain
pools show mismatch/incorrect values.
The distro-vendors can further consume these changes and modify the
respective service files to redirect the logs to specific files as
needed.

	Signed-off-by: Shradha Gupta <shradhagupta@linux.microsoft.com>
	Reviewed-by: Naman Jain <namjain@linux.microsoft.com>
	Reviewed-by: Dexuan Cui <decui@microsoft.com>
Link: https://lore.kernel.org/r/1744715978-8185-1-git-send-email-shradhagupta@linux.microsoft.com
	Signed-off-by: Wei Liu <wei.liu@kernel.org>
Message-ID: <1744715978-8185-1-git-send-email-shradhagupta@linux.microsoft.com>
(cherry picked from commit a9c0b33)
	Signed-off-by: Jonathan Maple <jmaple@ciq.com>
Signed-off-by: Jonathan Maple <jmaple@ciq.com>
Signed-off-by: Shreeya Patel <spatel@ciq.com>
Signed-off-by: Jonathan Maple <jmaple@ciq.com>
jira LE-3545
commit-author Dexuan Cui <decui@microsoft.com>
commit b2f9665

Currently storvsc_timeout is only used in storvsc_sdev_configure(), and
5s and 10s are used elsewhere. It turns out that rarely the 5s is not
enough on Azure, so let's use storvsc_timeout everywhere.

In case a timeout happens and storvsc_channel_init() returns an error,
close the VMBus channel so that any host-to-guest messages in the
channel's ringbuffer, which might come late, can be safely ignored.

Add a "const" to storvsc_timeout.

	Cc: stable@kernel.org
	Signed-off-by: Dexuan Cui <decui@microsoft.com>
Link: https://lore.kernel.org/r/1749243459-10419-1-git-send-email-decui@microsoft.com
	Reviewed-by: Long Li <longli@microsoft.com>
	Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
(cherry picked from commit b2f9665)
	Signed-off-by: Sultan Alsawaf <sultan@ciq.com>
Signed-off-by: Jonathan Maple <jmaple@ciq.com>
Signed-off-by: Shreeya Patel <spatel@ciq.com>
Signed-off-by: Jonathan Maple <jmaple@ciq.com>
jira LE-3554
commit-author Michael Kelley <mhklinux@outlook.com>
commit 380b75d

vmbus_sendpacket_mpb_desc() is currently used only by the storvsc driver
and is hardcoded to create a single GPA range. To allow it to also be
used by the netvsc driver to create multiple GPA ranges, no longer
hardcode as having a single GPA range. Allow the calling driver to
specify the rangecount in the supplied descriptor.

Update the storvsc driver to reflect this new approach.

	Cc: <stable@vger.kernel.org> # 6.1.x
	Signed-off-by: Michael Kelley <mhklinux@outlook.com>
Link: https://patch.msgid.link/20250513000604.1396-2-mhklinux@outlook.com
	Signed-off-by: Jakub Kicinski <kuba@kernel.org>
(cherry picked from commit 380b75d)
	Signed-off-by: Shreeya Patel <spatel@ciq.com>
Signed-off-by: Jonathan Maple <jmaple@ciq.com>
Signed-off-by: Shreeya Patel <spatel@ciq.com>
Signed-off-by: Jonathan Maple <jmaple@ciq.com>
jira LE-3554
commit-author Michael Kelley <mhklinux@outlook.com>
commit 4f98616

netvsc currently uses vmbus_sendpacket_pagebuffer() to send VMBus
messages. This function creates a series of GPA ranges, each of which
contains a single PFN. However, if the rndis header in the VMBus
message crosses a page boundary, the netvsc protocol with the host
requires that both PFNs for the rndis header must be in a single "GPA
range" data structure, which isn't possible with
vmbus_sendpacket_pagebuffer(). As the first step in fixing this, add a
new function netvsc_build_mpb_array() to build a VMBus message with
multiple GPA ranges, each of which may contain multiple PFNs. Use
vmbus_sendpacket_mpb_desc() to send this VMBus message to the host.

There's no functional change since higher levels of netvsc don't
maintain or propagate knowledge of contiguous PFNs. Based on its
input, netvsc_build_mpb_array() still produces a separate GPA range
for each PFN and the behavior is the same as with
vmbus_sendpacket_pagebuffer(). But the groundwork is laid for a
subsequent patch to provide the necessary grouping.

	Cc: <stable@vger.kernel.org> # 6.1.x
	Signed-off-by: Michael Kelley <mhklinux@outlook.com>
Link: https://patch.msgid.link/20250513000604.1396-3-mhklinux@outlook.com
	Signed-off-by: Jakub Kicinski <kuba@kernel.org>
(cherry picked from commit 4f98616)
	Signed-off-by: Shreeya Patel <spatel@ciq.com>
Signed-off-by: Jonathan Maple <jmaple@ciq.com>
Signed-off-by: Shreeya Patel <spatel@ciq.com>
Signed-off-by: Jonathan Maple <jmaple@ciq.com>
jira LE-3554
commit-author Michael Kelley <mhklinux@outlook.com>
commit 41a6328

Starting with commit dca5161 ("hv_netvsc: Check status in
SEND_RNDIS_PKT completion message") in the 6.3 kernel, the Linux
driver for Hyper-V synthetic networking (netvsc) occasionally reports
"nvsp_rndis_pkt_complete error status: 2".[1] This error indicates
that Hyper-V has rejected a network packet transmit request from the
guest, and the outgoing network packet is dropped. Higher level
network protocols presumably recover and resend the packet so there is
no functional error, but performance is slightly impacted. Commit
dca5161 is not the cause of the error -- it only added reporting
of an error that was already happening without any notice. The error
has presumably been present since the netvsc driver was originally
introduced into Linux.

The root cause of the problem is that the netvsc driver in Linux may
send an incorrectly formatted VMBus message to Hyper-V when
transmitting the network packet. The incorrect formatting occurs when
the rndis header of the VMBus message crosses a page boundary due to
how the Linux skb head memory is aligned. In such a case, two PFNs are
required to describe the location of the rndis header, even though
they are contiguous in guest physical address (GPA) space. Hyper-V
requires that two rndis header PFNs be in a single "GPA range" data
struture, but current netvsc code puts each PFN in its own GPA range,
which Hyper-V rejects as an error.

The incorrect formatting occurs only for larger packets that netvsc
must transmit via a VMBus "GPA Direct" message. There's no problem
when netvsc transmits a smaller packet by copying it into a pre-
allocated send buffer slot because the pre-allocated slots don't have
page crossing issues.

After commit 14ad6ed ("net: allow small head cache usage with
large MAX_SKB_FRAGS values") in the 6.14-rc4 kernel, the error occurs
much more frequently in VMs with 16 or more vCPUs. It may occur every
few seconds, or even more frequently, in an ssh session that outputs a
lot of text. Commit 14ad6ed subtly changes how skb head memory is
allocated, making it much more likely that the rndis header will cross
a page boundary when the vCPU count is 16 or more. The changes in
commit 14ad6ed are perfectly valid -- they just had the side
effect of making the netvsc bug more prominent.

Current code in init_page_array() creates a separate page buffer array
entry for each PFN required to identify the data to be transmitted.
Contiguous PFNs get separate entries in the page buffer array, and any
information about contiguity is lost.

Fix the core issue by having init_page_array() construct the page
buffer array to represent contiguous ranges rather than individual
pages. When these ranges are subsequently passed to
netvsc_build_mpb_array(), it can build GPA ranges that contain
multiple PFNs, as required to avoid the error "nvsp_rndis_pkt_complete
error status: 2". If instead the network packet is sent by copying
into a pre-allocated send buffer slot, the copy proceeds using the
contiguous ranges rather than individual pages, but the result of the
copying is the same. Also fix rndis_filter_send_request() to construct
a contiguous range, since it has its own page buffer array.

This change has a side benefit in CoCo VMs in that netvsc_dma_map()
calls dma_map_single() on each contiguous range instead of on each
page. This results in fewer calls to dma_map_single() but on larger
chunks of memory, which should reduce contention on the swiotlb.

Since the page buffer array now contains one entry for each contiguous
range instead of for each individual page, the number of entries in
the array can be reduced, saving 208 bytes of stack space in
netvsc_xmit() when MAX_SKG_FRAGS has the default value of 17.

[1] https://bugzilla.kernel.org/show_bug.cgi?id=217503

Closes: https://bugzilla.kernel.org/show_bug.cgi?id=217503
	Cc: <stable@vger.kernel.org> # 6.1.x
	Signed-off-by: Michael Kelley <mhklinux@outlook.com>
Link: https://patch.msgid.link/20250513000604.1396-4-mhklinux@outlook.com
	Signed-off-by: Jakub Kicinski <kuba@kernel.org>
(cherry picked from commit 41a6328)
	Signed-off-by: Shreeya Patel <spatel@ciq.com>
Signed-off-by: Jonathan Maple <jmaple@ciq.com>
Signed-off-by: Shreeya Patel <spatel@ciq.com>
Signed-off-by: Jonathan Maple <jmaple@ciq.com>
jira LE-3554
commit-author Michael Kelley <mhklinux@outlook.com>
commit 5bbc644

init_page_array() now always creates a single page buffer array entry
for the rndis message, even if the rndis message crosses a page
boundary. As such, the number of page buffer array entries used for
the rndis message must no longer be tracked -- it is always just 1.
Remove the rmsg_pgcnt field and use "1" where the value is needed.

	Cc: <stable@vger.kernel.org> # 6.1.x
	Signed-off-by: Michael Kelley <mhklinux@outlook.com>
Link: https://patch.msgid.link/20250513000604.1396-5-mhklinux@outlook.com
	Signed-off-by: Jakub Kicinski <kuba@kernel.org>
(cherry picked from commit 5bbc644)
	Signed-off-by: Shreeya Patel <spatel@ciq.com>
Signed-off-by: Jonathan Maple <jmaple@ciq.com>
Signed-off-by: Shreeya Patel <spatel@ciq.com>
Signed-off-by: Jonathan Maple <jmaple@ciq.com>
jira LE-3554
commit-author Michael Kelley <mhklinux@outlook.com>
commit 45a442f

With the netvsc driver changed to use vmbus_sendpacket_mpb_desc()
instead of vmbus_sendpacket_pagebuffer(), the latter has no remaining
callers. Remove it.

	Cc: <stable@vger.kernel.org> # 6.1.x
	Signed-off-by: Michael Kelley <mhklinux@outlook.com>
Link: https://patch.msgid.link/20250513000604.1396-6-mhklinux@outlook.com
	Signed-off-by: Jakub Kicinski <kuba@kernel.org>
(cherry picked from commit 45a442f)
	Signed-off-by: Shreeya Patel <spatel@ciq.com>
Signed-off-by: Jonathan Maple <jmaple@ciq.com>
Signed-off-by: Shreeya Patel <spatel@ciq.com>
Signed-off-by: Jonathan Maple <jmaple@ciq.com>
@PlaidCat PlaidCat requested a review from a team October 28, 2025 18:42
@PlaidCat PlaidCat self-assigned this Oct 28, 2025
Copy link
Collaborator

@bmastbergen bmastbergen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🥌

Copy link

@jdieter jdieter left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@PlaidCat PlaidCat merged commit 3f4cd04 into rlc-9/5.14.0-570.55.1.el9_6 Oct 28, 2025
4 checks passed
@PlaidCat PlaidCat deleted the jmaple_rlc-9/5.14.0-570.55.1.el9_6 branch October 28, 2025 19:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

6 participants