Skip to content

PDD-CLI Bug: Generates Features Without Corresponding Test Coverage #589

@jiaminc-cmu

Description

@jiaminc-cmu

PDD-CLI Bug: Generates Features Without Corresponding Test Coverage

PDD-CLI implements new features but doesn't generate tests for them. Features ship without validation that they work correctly.

Why this matters: Bugs aren't caught until production, technical debt accumulates, refactoring becomes risky.

Concrete Example

For a YAML template rendering feature:

# PDD generated feature (NO TESTS):
# template_renderer.py
import yaml
from jinja2 import Template

def render_yaml_template(template_path: str, data: dict) -> str:
    """Render YAML template with Jinja2."""
    with open(template_path) as f:
        template_content = f.read()
    
    template = Template(template_content)
    rendered = template.render(**data)
    
    # Parse to validate YAML
    yaml.safe_load(rendered)
    
    return rendered

But no tests exist:

# tests/test_template_renderer.py
# FILE DOESN'T EXIST!

What went wrong: PDD implemented template rendering but didn't generate tests to verify it handles edge cases (missing files, invalid YAML, special characters, etc.).

Impact: Feature ships untested. First user discovers bugs in production (file not found, YAML parsing errors, etc.).

Why PDD Makes This Mistake

PDD-CLI currently:

  • Focuses on feature implementation
  • Treats testing as separate step
  • Doesn't follow TDD (Test-Driven Development)

But it should:

  1. Generate tests alongside feature code
  2. Cover happy path and edge cases
  3. Validate error handling

How to Prevent This in PDD-CLI

What PDD should do differently:

  1. Generate tests with features (TDD approach):

    # tests/test_template_renderer.py
    import pytest
    from template_renderer import render_yaml_template
    
    def test_render_yaml_template_basic():
        result = render_yaml_template('templates/test.yaml', {'name': 'Alice'})
        assert 'Alice' in result
        assert yaml.safe_load(result)  # Valid YAML
    
    def test_render_yaml_template_missing_file():
        with pytest.raises(FileNotFoundError):
            render_yaml_template('nonexistent.yaml', {})
    
    def test_render_yaml_template_invalid_yaml():
        with pytest.raises(yaml.YAMLError):
            render_yaml_template('templates/invalid.yaml', {})
  2. Calculate test coverage: Ensure generated tests cover main paths.

  3. Generate test data/fixtures: Create sample templates for testing.

Example improvement:

Current: "Add YAML template rendering"
       → Generate template_renderer.py
       → No tests
       → Bugs found in production

Improved: "Add YAML template rendering"
        → Generate template_renderer.py
        → Generate test_template_renderer.py
        → Generate test fixtures (sample templates)
        → Run tests to validate
        → Confident feature works correctly

Severity

P2 - Medium Priority

  • Frequency: High - affects all new features
  • Impact: Medium - technical debt, bugs in production
  • Detectability: Low - absence of tests is passive
  • Prevention cost: Medium - requires test generation logic

Category

incomplete-implementation

Related Issues


For Contributors: Discovered when YAML template rendering was added without tests, manual tests added later in commit 34a651d5. Tests should be generated alongside features.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions