feat: Implement CICDPipelineVisibilityAssessor

[jeremyeder]

feat: Implement CICDPipelineVisibilityAssessor

Attribute Definition

Attribute ID: cicd_pipeline_visibility (Attribute #24 - Tier 3)

Definition: Clear, well-documented CI/CD configuration files committed to repository.

Why It Matters: AI can understand build/test/deploy processes by reading CI configs. When builds fail, AI can suggest targeted fixes. Visible pipelines enable collaboration and debugging.

Impact on Agent Behavior:

• CI improvement proposals
• Pipeline failure debugging
• Workflow optimization suggestions
• Better understanding of deployment process

Measurable Criteria:

• CI config file in repository:
  • GitHub Actions: .github/workflows/
  • GitLab CI: .gitlab-ci.yml
  • CircleCI: .circleci/config.yml
  • Travis CI: .travis.yml
• Clear job/step names (not "step1", "step2")
• Comments explaining complex logic
• Fast feedback: Tests complete <10 minutes
• Fail fast: Stop on first failure to save compute
• Parallelization: Run independent jobs concurrently
• Caching: Dependencies, build artifacts
• Artifacts: Test results, coverage reports, logs

Implementation Requirements

File Location: src/agentready/assessors/testing.py

Class Name: CICDPipelineVisibilityAssessor

Tier: 3 (Important)

Default Weight: 0.015 (1.5% of total score)

Assessment Logic

Scoring Approach: Detect CI config file and assess quality

Evidence to Check (score components):

1. CI config exists (50%)

   • Check for common CI platforms
   • GitHub Actions, GitLab CI, CircleCI, Travis CI, Jenkins

2. Config quality (30%)

   • Parse YAML/JSON config
   • Check for descriptive job names
   • Verify caching is configured
   • Look for parallelization

3. Best practices (20%)

   • Comments in config
   • Fail-fast strategy
   • Artifact uploading
   • Status badges in README

Scoring Logic:

if ci_config_exists:
    config_score = 50

    # Quality checks
    quality_score = 0
    if has_descriptive_names:
        quality_score += 10
    if has_caching:
        quality_score += 10
    if has_parallelization:
        quality_score += 10

    # Best practices
    practices_score = 0
    if has_comments:
        practices_score += 10
    if uploads_artifacts:
        practices_score += 10

    total_score = config_score + quality_score + practices_score
else:
    total_score = 0

status = "pass" if total_score >= 75 else "fail"

Code Pattern to Follow

Reference: PreCommitHooksAssessor for config file detection

Pattern:

1. Check for CI config files in common locations
2. Parse YAML config (GitHub Actions, GitLab, CircleCI)
3. Extract job names, steps, caching config
4. Validate best practices
5. Calculate proportional score
6. Provide improvement suggestions

Example Finding Responses

Pass (Score: 90)

Finding(
    attribute=self.attribute,
    status="pass",
    score=90.0,
    measured_value="GitHub Actions with best practices",
    threshold="CI config present",
    evidence=[
        "CI config: .github/workflows/ci.yml",
        "3 jobs defined: lint, test, build",
        "Descriptive job/step names",
        "Caching configured for dependencies",
        "Parallel job execution",
        "Artifacts uploaded: test results, coverage",
    ],
    remediation=None,
    error_message=None,
)

Fail (Score: 50)

Finding(
    attribute=self.attribute,
    status="fail",
    score=50.0,
    measured_value="basic CI config",
    threshold="CI with best practices",
    evidence=[
        "CI config: .github/workflows/test.yml",
        "Generic job names: 'build', 'test'",
        "No caching configured",
        "Sequential job execution (no parallelization)",
        "No artifacts uploaded",
    ],
    remediation=self._create_remediation(),
    error_message=None,
)

Fail (Score: 0)

Finding(
    attribute=self.attribute,
    status="fail",
    score=0.0,
    measured_value="no CI config",
    threshold="CI config present",
    evidence=[
        "No CI/CD configuration found",
        "Checked: GitHub Actions, GitLab CI, CircleCI, Travis CI",
    ],
    remediation=self._create_remediation(),
    error_message=None,
)

Registration

Add to src/agentready/services/scanner.py in create_all_assessors():

from ..assessors.testing import (
    TestCoverageAssessor,
    PreCommitHooksAssessor,
    TestNamingConventionsAssessor,
    CICDPipelineVisibilityAssessor,  # Add this import
)

def create_all_assessors() -> List[BaseAssessor]:
    return [
        # ... existing assessors ...
        CICDPipelineVisibilityAssessor(),  # Add this line
    ]

Testing Guidance

Test File: tests/unit/test_assessors_testing.py

Test Cases to Add:

1. test_cicd_pass_github_actions: Repository with well-configured GitHub Actions
2. test_cicd_pass_gitlab_ci: Repository with GitLab CI config
3. test_cicd_fail_basic_config: CI exists but lacks best practices
4. test_cicd_fail_no_config: No CI configuration found
5. test_cicd_partial_score: Some best practices, not all

Note: AgentReady has GitHub Actions workflows, should score well (85+).

Dependencies

External Tools: None (YAML parsing only)

Python Standard Library:

• pathlib.Path for file detection
• yaml (PyYAML) for parsing CI configs

Remediation Steps

def _create_remediation(self) -> Remediation:
    return Remediation(
        summary="Add or improve CI/CD pipeline configuration",
        steps=[
            "Create CI config for your platform (GitHub Actions, GitLab CI, etc.)",
            "Define jobs: lint, test, build",
            "Use descriptive job and step names",
            "Configure dependency caching",
            "Enable parallel job execution",
            "Upload artifacts: test results, coverage reports",
            "Add status badge to README",
        ],
        tools=["github-actions", "gitlab-ci", "circleci"],
        commands=[
            "# Create GitHub Actions workflow",
            "mkdir -p .github/workflows",
            "touch .github/workflows/ci.yml",
            "",
            "# Validate workflow",
            "gh workflow view ci.yml",
        ],
        examples=[
            """# .github/workflows/ci.yml - Good example

name: CI Pipeline

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  lint:
    name: Lint Code
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
          cache: 'pip'  # Caching

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Run linters
        run: |
          black --check .
          isort --check .
          ruff check .

  test:
    name: Run Tests
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
          cache: 'pip'

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Run tests with coverage
        run: pytest --cov --cov-report=xml

      - name: Upload coverage reports
        uses: codecov/codecov-action@v3
        with:
          files: ./coverage.xml

  build:
    name: Build Package
    runs-on: ubuntu-latest
    needs: [lint, test]  # Runs after lint/test pass
    steps:
      - uses: actions/checkout@v4

      - name: Build package
        run: python -m build

      - name: Upload build artifacts
        uses: actions/upload-artifact@v3
        with:
          name: dist
          path: dist/
""",
            """# Bad CI config - Generic names, no caching

name: CI

on: [push]

jobs:
  build:  # Generic name
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt  # No caching
      - run: pytest  # No coverage, no artifacts
""",
        ],
        citations=[
            Citation(
                source="GitHub",
                title="GitHub Actions Documentation",
                url="https://docs.github.com/en/actions",
                relevance="Official GitHub Actions guide",
            ),
            Citation(
                source="CircleCI",
                title="CI/CD Best Practices",
                url="https://circleci.com/blog/ci-cd-best-practices/",
                relevance="Industry best practices for CI/CD",
            ),
        ],
    )

Implementation Notes

1. Config Detection: Check for files in priority order:
   • .github/workflows/*.yml
   • .gitlab-ci.yml
   • .circleci/config.yml
   • .travis.yml
   • Jenkinsfile
2. YAML Parsing: Parse config to extract jobs, steps, caching
3. Name Quality: Check if job/step names are descriptive (>3 words, not "build", "test")
4. Caching Detection: Look for cache:, actions/cache@, or platform-specific caching
5. Parallelization: Check if multiple jobs defined or matrix strategy used
6. Artifacts: Look for artifact upload steps
7. Edge Cases: Handle multiple workflow files (count all), invalid YAML (skip)

[github-actions]

🎉 This issue has been resolved in version 1.16.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀