Add automated Kernel CI workflow #675

shreeya-patel98 · 2025-11-07T11:05:49Z

Add automated kernel CI workflow with kselftest and PR creation
Implements a 5-stage GitHub Actions pipeline for automated kernel testing and PR creation.
Uses kernel-container-build automated-testing-v1 branch for build/test tooling.

Stage 1: Build (15-30 min)

Checkout kernel source + kernel-container-build repo (automated-testing-v1)
Build kernel in CIQ builder container with kABI checking
Convert built container to QCOW2 VM image
Upload: kernel-build.log, QCOW2 image

Stage 2: Boot Verification (2-5 min)

Download QCOW2 image
Boot kernel in QEMU (KVM or TCG) and validate login prompt appears
Upload: boot logs

Stage 3: Kernel Selftests (40-60 min)

Download QCOW2 image
Execute comprehensive kselftests in QEMU with dual serial consoles
Upload: kselftest TAP logs, dmesg output

Stage 4: Compare Results (1-2 min)
Purpose: Detect test regressions by comparing against base branch

Steps:

Checkout with full history (fetch-depth: 0) for git merge-base ops
Download current kselftest logs
Smart base branch detection:
- For PRs: Uses PR's target branch
- For pushes: Sorts branches by commit date, checks 30 most recent,
  finds closest common ancestor via git merge-base
- For force-pushes: Doesn't do the detection again, instead uses the
  same base branch as detected during the first PR creation and also
  compares the result against that base branch only
- Outputs: base_branch (reused by PR stage)
Download baseline logs from base branch (searches last 5 successful runs)
Compare results:
- Counts passing/failing tests (before/after)
- Fails if >±3 tests changed
- Outputs: comparison_status, comparison_message

Stage 5: Create Pull Request (1-2 min)
Purpose: Auto-create/update PR after all tests pass

Prerequisites: Only runs if build + boot + kselftest passed, no regressions detected

Steps:

Check all stages passed and comparison_status != failed
Checkout (shallow: fetch-depth: 50) for commit messages
Download all artifacts (build/boot/test logs)
Extract statistics (pass/fail counts, build times)
Get commit info:
- Single commit: Use commit message
- Multiple commits: Create summary
Create/Update PR:
- Reuses base_branch from compare-results (no duplication!)
- Generate PR body with test results via create-pr-body.sh
- Creates new PR or updates existing one in case of force pushes

Note :-
To skip this github action workflow, add [ci skip] or [skip ci]
into the head commit when pushing the patches.

Implements a 5-stage GitHub Actions pipeline for automated kernel testing and PR creation. Uses kernel-container-build automated-testing-v1 branch for build/test tooling. Stage 1: Build (15-30 min) - Checkout kernel source + kernel-container-build repo (automated-testing-v1) - Build kernel in CIQ builder container with kABI checking - Convert built container to QCOW2 VM image - Upload: kernel-build.log, QCOW2 image Stage 2: Boot Verification (2-5 min) - Download QCOW2 image - Boot kernel in QEMU (KVM or TCG) and validate login prompt appears - Upload: boot logs Stage 3: Kernel Selftests (40-60 min) - Download QCOW2 image - Execute comprehensive kselftests in QEMU with dual serial consoles - Upload: kselftest TAP logs, dmesg output Stage 4: Compare Results (1-2 min) Purpose: Detect test regressions by comparing against base branch Steps: 1. Checkout with full history (fetch-depth: 0) for git merge-base ops 2. Download current kselftest logs 3. Smart base branch detection: - For PRs: Uses PR's target branch - For pushes: Sorts branches by commit date, checks 30 most recent, finds closest common ancestor via git merge-base - For force-pushes: Doesn't do the detection again, instead uses the same base branch as detected during the first PR creation and also compares the result against that base branch only - Outputs: base_branch (reused by PR stage) 4. Download baseline logs from base branch (searches last 5 successful runs) 5. Compare results: - Counts passing/failing tests (before/after) - Fails if >±3 tests changed - Outputs: comparison_status, comparison_message Stage 5: Create Pull Request (1-2 min) Purpose: Auto-create/update PR after all tests pass Prerequisites: Only runs if build + boot + kselftest passed, no regressions detected Steps: 1. Check all stages passed and comparison_status != failed 2. Checkout (shallow: fetch-depth: 50) for commit messages 3. Download all artifacts (build/boot/test logs) 4. Extract statistics (pass/fail counts, build times) 5. Get commit info: - Single commit: Use commit message - Multiple commits: Create summary 6. Create/Update PR: - Reuses base_branch from compare-results (no duplication!) - Generate PR body with test results via create-pr-body.sh - Creates new PR or updates existing one in case of force pushes Note :- To skip this github action workflow, add [ci skip] or [skip ci] into the head commit when pushing the patches. Signed-off-by: Shreeya Patel <spatel@ciq.com>

Script to generate detailed PR descriptions with kselftest results. Signed-off-by: Shreeya Patel <spatel@ciq.com>

.github/workflows/kernel-build-and-test-x86_64.yml

roxanan1996 · 2025-11-12T13:46:13Z

.github/scripts/create-pr-body.sh

+RUN_ID="$5"
+COMPARISON_SECTION="$6"
+REPO="$7"
+


minor, but just to be sure, I would check if nb of arguments is as expected.

either number or args OR qualify each.

roxanan1996 · 2025-11-12T15:22:09Z

.github/workflows/kernel-build-and-test-x86_64.yml

+        fi
+
+        if [ -z "$BASE_BRANCH" ]; then
+          echo "::warning::Could not determine base branch for comparison - no common ancestor found"


I would put error here, not warning.

And exit 1 or sth else, not 0.

# Only check the 30 most recently updated branches for performance
I think this is the reason its set as a warning rather than as an error.

As discussed in private, this does not work because we also have personal branches pushed.
As seen here #683.

We can extract the base branch based on the branch, or we can ignore the branches with '{}', or we can keep a list of the branches that are bases. This is anyway the idea to move forward for automation. Keeping this list in a yaml file would be the norm, so I don't see it as an issue here that we "hardcode" this.

roxanan1996 · 2025-11-12T15:27:08Z

.github/workflows/kernel-build-and-test-x86_64.yml

+      id: commit_msg
+      run: |
+        # Count commits since origin/main (or appropriate base branch)
+        BASE_BRANCH="main"


Why not use the same as Determine base branch for comparison?

PlaidCat · 2025-11-12T20:50:29Z

.github/scripts/create-pr-body.sh

+\`\`\`
+EOF
+
+cat /tmp/commit_message.txt


This is created outside of this script, can you add a check and fail before this section ... just in case.

Additionally this is hardcoded in two separate files we should probably make this an input parameter

PlaidCat · 2025-11-12T21:21:55Z

.github/scripts/create-pr-body.sh

+REPO="$7"
+
+# Convert seconds to minutes for better readability
+convert_time() {


Ideally there are not issues here but we should validate the BUILD_TIME and TOTAL_TIME here or at script input validation.

Mostly concerned about accidental drift.

PlaidCat · 2025-11-12T23:09:02Z

.github/workflows/kernel-build-and-test-x86_64.yml

+        name: kselftest-logs-x86_64
+        path: |
+          output/kselftests-*.log
+          output/dmesg-*.log


if we didn't need the dmesg logs could we keep these for like 60 days?

@PlaidCat We have a bigger problem actually which I wanted to discuss. All the logs will only be there for 7 days in github. Unfortunately that's the policy of github if we use their runners.

PlaidCat · 2025-11-12T23:47:19Z

.github/workflows/kernel-build-and-test-x86_64.yml

+          sudo apt-get install -y gh
+        fi
+
+    - name: Determine base branch for comparison


This whole step seems overly complicated

name: Automated kernel build and test (x86_64) on: push: branches: - '*_ciqlts9_2'

We're only allowing this to run on specific patterns, if that is built into the branch name it should be pretty easy to create a map and validate that the HEAD of the base branch is contained in this branch?

Then maybe use an error.

For a completely generic case this might work but also requires relatively recent updates.

If i'm missing something here please correct me.

PlaidCat · 2025-11-12T23:49:00Z

.github/workflows/kernel-build-and-test-x86_64.yml

+
+    - name: Download baseline kselftest logs from base branch
+      if: steps.base_branch.outputs.base_branch != ''
+      uses: dawidd6/action-download-artifact@v3


this appear to be a user action, we should do a code audit of this.

@bmastbergen has mentioned this before too, maybe as CIQ we should be authoring our own sort of actions like this so that we can contribute to additional community members and we know a company is behind an action code we're running

shreeya-patel98 requested a review from a team November 7, 2025 11:05

shreeya-patel98 self-assigned this Nov 7, 2025

shreeya-patel98 requested review from PlaidCat, bmastbergen and kerneltoast as code owners November 7, 2025 11:05

shreeya-patel98 added 2 commits November 11, 2025 14:22

Add PR body generation script

965ef63

Script to generate detailed PR descriptions with kselftest results. Signed-off-by: Shreeya Patel <spatel@ciq.com>

shreeya-patel98 force-pushed the shreeya_kernelci_main branch from 53a82b2 to 965ef63 Compare November 11, 2025 14:23

roxanan1996 reviewed Nov 11, 2025

View reviewed changes

.github/workflows/kernel-build-and-test-x86_64.yml Show resolved Hide resolved

roxanan1996 reviewed Nov 12, 2025

View reviewed changes

PlaidCat reviewed Nov 12, 2025

View reviewed changes

Add automated Kernel CI workflow #675

Are you sure you want to change the base?

Add automated Kernel CI workflow #675

Uh oh!

Conversation

shreeya-patel98 commented Nov 7, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PlaidCat Nov 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PlaidCat Nov 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

4 participants

PlaidCat Nov 12, 2025 •

edited

Loading

PlaidCat Nov 12, 2025 •

edited

Loading