[CI Failure Doctor] CI Failure Investigation - SBOM Upload Timing Change (Run #21102091204)

[github-actions]

🏥 CI Failure Investigation - Run #21102091204

Summary

⚠️ INVESTIGATION LIMITED - Cannot access workflow run logs due to API authentication restrictions. Analysis based on code changes from PR #10501.

Change: PR #10501 ("Move SBOM generation after release creation in release workflow") restructured the release workflow to generate and upload SBOM files AFTER creating the GitHub release, rather than including them in the initial release creation.

Failure Details

• Run: 21102091204
• Commit: 3fc1435
• Trigger: push to tag or manual workflow_dispatch
• PR: Move SBOM generation after release creation in release workflow #10501
• Author: Copilot (AI agent)

Code Changes Analysis

What Changed

Before (.github/workflows/release.md - original):

1. Download Go modules
2. Generate SBOM (SPDX + CycloneDX formats)
3. Audit SBOM files for secrets
4. Upload SBOM artifacts
5. Create GitHub release with binaries + SBOM files

After (.github/workflows/release.md - PR #10501):

1. Create GitHub release with binaries only
2. Download Go modules
3. Generate SBOM (SPDX + CycloneDX formats)
4. Audit SBOM files for secrets
5. Upload SBOM artifacts
6. NEW STEP: Upload SBOM files to existing release using gh release upload

Potential Issues Identified

1. 🔴 Release Tag Variable Dependency

The new "Upload SBOM files to release" step depends on:

RELEASE_TAG: ${{ needs.config.outputs.release_tag }}

Risk: If the config job output is not properly set or the variable is not interpolated correctly, the upload will fail with "release not found" errors.

2. 🔴 Race Condition with Draft Releases

The workflow supports draft releases (DRAFT_MODE = true/false). The new step attempts to upload to a release that may be in draft state:

gh release upload "$RELEASE_TAG" \
  sbom.spdx.json \
  sbom.cdx.json

Risk: gh release upload may behave differently for draft vs. published releases, potentially failing on draft releases.

3. 🔴 File Availability After Release Creation

SBOM files are generated AFTER the release is created. The workflow assumes files will still be present in the working directory when the upload step runs.

Risk: If the GitHub Actions runner cleans up files or if there's a working directory change, the SBOM files might not be found.

4. 🟡 No Error Handling for Upload Failures

The new upload step lacks error handling:

echo "Uploading SBOM files to release: $RELEASE_TAG"
gh release upload "$RELEASE_TAG" \
  sbom.spdx.json \
  sbom.cdx.json
echo "✓ SBOM files uploaded to release"

Risk: If upload fails (network issue, permissions, file not found), the workflow continues without clear failure indication.

Likely Root Causes

Without access to logs, the most probable causes are:

Primary Suspects

1. Release not found: $RELEASE_TAG variable not properly interpolated or release doesn't exist yet
2. SBOM files not found: sbom.spdx.json and sbom.cdx.json missing from working directory
3. Draft release issue: gh release upload command failing on draft releases
4. Permissions: GH_TOKEN lacking permissions to upload to releases

Secondary Suspects

5. Job dependency timing: The step runs too quickly after release creation, before GitHub's API stabilizes
6. Working directory mismatch: Files generated in different directory than where upload command runs
7. anchore/sbom-action output location: The SBOM action may not be writing files to expected location

Recommended Actions

Immediate Investigation (Manual)

1. Access the workflow run logs:

   gh run view 21102091204 --log

2. Check which step failed:

   gh run view 21102091204 --log | grep -i "error\|failed"

3. Examine SBOM upload step specifically:

   gh run view 21102091204 --log | grep -A 20 "Upload SBOM files to release"

Code Fixes (Based on likely issues)

Fix 1: Add Debug Output and Error Handling

- name: Upload SBOM files to release
  env:
    GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    RELEASE_TAG: ${{ needs.config.outputs.release_tag }}
  run: |
    set -e  # Exit on any error
    
    # Debug: Show environment
    echo "Working directory: $(pwd)"
    echo "Release tag: $RELEASE_TAG"
    echo "Files present:"
    ls -la sbom.*.json || echo "ERROR: SBOM files not found"
    
    # Verify release exists
    if ! gh release view "$RELEASE_TAG" > /dev/null 2>&1; then
      echo "ERROR: Release $RELEASE_TAG not found"
      exit 1
    fi
    
    # Upload with error checking
    echo "Uploading SBOM files to release: $RELEASE_TAG"
    if gh release upload "$RELEASE_TAG" sbom.spdx.json sbom.cdx.json; then
      echo "✓ SBOM files uploaded successfully"
    else
      echo "ERROR: Failed to upload SBOM files"
      exit 1
    fi

Fix 2: Add Explicit File Path Verification

- name: Verify SBOM files exist
  run: |
    for file in sbom.spdx.json sbom.cdx.json; do
      if [ ! -f "$file" ]; then
        echo "ERROR: Missing SBOM file: $file"
        echo "Current directory: $(pwd)"
        echo "Files present: $(ls -la)"
        exit 1
      fi
      echo "✓ Found: $file ($(stat -f%z "$file" 2>/dev/null || stat -c%s "$file") bytes)"
    done

- name: Upload SBOM files to release
  # ... existing step

Fix 3: Handle Draft Releases Explicitly

- name: Upload SBOM files to release
  env:
    GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    RELEASE_TAG: ${{ needs.config.outputs.release_tag }}
    DRAFT_MODE: ${{ needs.config.outputs.draft_mode }}
  run: |
    echo "Uploading SBOM files to release: $RELEASE_TAG (draft: $DRAFT_MODE)"
    
    # Upload works for both draft and published releases
    gh release upload "$RELEASE_TAG" \
      sbom.spdx.json \
      sbom.cdx.json \
      --clobber  # Overwrite if files already exist
    
    echo "✓ SBOM files uploaded to release"

Prevention Strategies

1. Add integration test for SBOM upload:

   • Create a test workflow that creates a test release
   • Generate dummy SBOM files
   • Upload to test release
   • Verify files are attached

2. Add workflow validation:

   • Use gh aw compile to ensure workflow compiles correctly
   • Add pre-merge CI check that validates release workflow structure

3. Improve logging:

   • Add verbose logging to all release workflow steps
   • Include file listing, variable values, and API responses
   • Use set -x for shell debugging when troubleshooting

Historical Context

• Issue [plan] Add SBOM generation to CI/CD pipeline and releases #5539: Original task to add SBOM generation to releases (completed)
• Issue [step-names] Align step names in release.lock.yml, super-linter.lock.yml, and mergefest.lock.yml #10404: Recent step naming alignment in release.lock.yml (completed)
• No similar failures found: This appears to be a new failure pattern introduced by the SBOM upload restructuring

AI Team Self-Improvement

Add to developer instructions (AGENTS.md or developer.instructions):

### Release Workflow Changes

When modifying the release workflow (.github/workflows/release.md):

1. **Always add debug output** for file operations and API calls:
   - List files before upload operations
   - Print environment variables being used
   - Echo API responses

2. **Verify job outputs are available**:
   - When using `needs.X.outputs.Y`, verify the output is set in job X
   - Add debug step to print all job outputs at the start of dependent jobs

3. **Test with both draft and published releases**:
   - Release workflow supports both modes via `draft_mode` input
   - Ensure all steps work correctly in both scenarios

4. **Add explicit error handling**:
   - Use `set -e` in bash scripts to exit on errors
   - Check file existence before operations
   - Verify releases exist before uploading assets

5. **Don't remove SBOM files from initial release creation without thorough testing**:
   - Changes to release asset upload timing require integration testing
   - Consider keeping dual upload (initial + post-creation) for safety

6. **Test locally before PR**:
   ```bash
   # Simulate release process locally
   make build
   ./scripts/build-release.sh v0.0.1-test
   # Verify SBOM generation
   go mod download
   # Check files exist
   ls -la sbom.*.json

## Next Steps

1. **Human investigation required**: Access workflow logs at https://github.com/githubnext/gh-aw/actions/runs/21102091204
2. **Identify exact failure point**: Determine which step failed and error message
3. **Apply appropriate fix**: Based on error message, implement one of the recommended fixes
4. **Add regression test**: Ensure this failure type is caught in CI before merge
5. **Document the fix**: Update this issue with root cause and solution once identified

---

**Investigation Status**: ⚠️ INCOMPLETE - Requires GitHub API access to view workflow logs

> AI generated by [CI Failure Doctor](https://github.com/githubnext/gh-aw/actions/runs/21102134811)

<!-- agentic-workflow: CI Failure Doctor, engine: copilot, run: https://github.com/githubnext/gh-aw/actions/runs/21102134811 -->




> AI generated by [CI Failure Doctor](https://github.com/githubnext/gh-aw/actions/runs/21102134811)
>
> To add this workflow in your repository, run `gh aw add githubnext/agentics/workflows/ci-doctor.md@ea350161ad5dcc9624cf510f134c6a9e39a6f94d`. See [usage guide](https://githubnext.github.io/gh-aw/guides/packaging-imports/).

<!-- agentic-workflow: CI Failure Doctor, engine: copilot, run: https://github.com/githubnext/gh-aw/actions/runs/21102134811 -->