feat(test): Add Automated Workflow Testing Framework #30

tduhamel42 · 2025-10-29T13:35:51Z

Summary

This PR adds a comprehensive automated testing framework for FuzzForge workflows, enabling continuous validation of workflow execution, SARIF export, and platform-specific Docker image selection.

What's New

1. Test Matrix Configuration ()

Defines 8 testable workflows (excludes LLM and OSS-Fuzz)
Maps workflows → workers → test projects → parameters
Defines 3 test suites: fast (5 workflows), full (8 workflows), platform-specific

2. Workflow Execution Test Script (Error: PyYAML is required. Install with: pip install pyyaml)

Executes workflows with parameter validation
Validates SARIF export and structure
Counts findings and measures execution time
Generates test summary reports
Usage: uv run python scripts/test_workflows.py --suite fast

3. Platform Detection Unit Tests ()

Tests platform detection (x86_64, aarch64, arm64)
Tests Dockerfile selection for multi-platform workers (Android)
Tests metadata.yaml parsing and fallback logic
15+ test cases with mocking

4. GitHub Actions Workflow ()

Platform Detection Tests: Unit tests for platform logic
Fast Workflow Tests: 5 workflows on every PR (~2.5 min)
Android Platform Tests: Verifies AMD64 and ARM64 Dockerfile selection
Full Workflow Tests: All 8 workflows on main/schedule (~10 min)
Automatic log collection on failure

5. Comprehensive Documentation ()

Local testing guide with examples
CI/CD testing explanation
Platform-specific testing guide
Debugging guide and best practices
Test matrix configuration guide

Test Coverage

8 Workflows Tested:

android_static_analysis → Android worker (platform-specific)
python_sast → Python worker
atheris_fuzzing → Python worker
cargo_fuzzing → Rust worker
secret_detection → Secrets worker
gitleaks_detection → Secrets worker
trufflehog_detection → Secrets worker
security_assessment → Composite workflow

4 Workers Covered:

python (1 Dockerfile)
android (2 Dockerfiles: AMD64 with MobSF, ARM64 without)
rust (1 Dockerfile)
secrets (1 Dockerfile)

Local Test Results ✅

Total tests: 5
Passed: 5 ✅
Failed: 0 ❌

• android_static_analysis: 98.98s ✅ (SARIF: 16K)
• python_sast: 6.78s ✅ (SARIF: 21K)
• secret_detection: 38.04s ✅ (SARIF: 9.8K)
• gitleaks_detection: 1.67s ✅ (SARIF: 367B)
• trufflehog_detection: 1.64s ✅ (SARIF: 378B)

Benefits

✅ Automated Workflow Testing - All core workflows automatically validated
✅ Platform-Specific Testing - Android worker tested on both AMD64 and ARM64
✅ SARIF Validation - Ensures CI/CD integration compatibility
✅ Fast Feedback - Fast suite runs on every PR
✅ Comprehensive Coverage - Full suite tests everything
✅ Easy Local Testing - Simple CLI: uv run python scripts/test_workflows.py --suite fast
✅ Extensible - Easy to add new workflows to test matrix
✅ Well Documented - Complete guide for developers

Changes

Added

.github/test-matrix.yaml - Test configuration
scripts/test_workflows.py - Test execution script
cli/tests/test_platform_detection.py - Platform detection unit tests
.github/workflows/test-workflows.yml - CI workflow for integration tests
docs/docs/development/testing.md - Testing documentation

Modified

.github/workflows/test.yml - Added reference to new workflow tests

Removed

test_projects/secret_detection_benchmark/.fuzzforge/findings.db - Already in .gitignore

Testing Instructions

Run Locally

# Single workflow
uv run python scripts/test_workflows.py --workflow python_sast --skip-service-start

# Fast test suite (5 workflows)
uv run python scripts/test_workflows.py --suite fast --skip-service-start

# Full test suite (8 workflows)
uv run python scripts/test_workflows.py --suite full --skip-service-start

CI Testing

The GitHub Actions workflow will automatically run:

Platform detection unit tests on every PR
Fast workflow tests on every PR
Android platform tests on every PR
Full workflow tests on main/schedule

Implements Issue #5 - Python SAST workflow that combines: - Dependency scanning (pip-audit) for CVE detection - Security linting (Bandit) for vulnerability patterns - Type checking (Mypy) for type safety issues ## Changes **New Modules:** - `DependencyScanner`: Scans Python dependencies for known CVEs using pip-audit - `BanditAnalyzer`: Analyzes Python code for security issues using Bandit - `MypyAnalyzer`: Checks Python code for type safety issues using Mypy **New Workflow:** - `python_sast`: Temporal workflow that orchestrates all three SAST tools - Runs tools in parallel for fast feedback (3-5 min vs hours for fuzzing) - Generates unified SARIF report with findings from all tools - Supports configurable severity/confidence thresholds **Updates:** - Added SAST dependencies to Python worker (bandit, pip-audit, mypy) - Updated module __init__.py files to export new analyzers - Added type_errors.py test file to vulnerable_app for Mypy validation ## Testing Workflow tested successfully on vulnerable_app: - ✅ Bandit: Detected 9 security issues (command injection, unsafe functions) - ✅ Mypy: Detected 5 type errors - ✅ DependencyScanner: Ran successfully (no CVEs in test dependencies) - ✅ SARIF export: Generated valid SARIF with 14 total findings

feat: Add Python SAST workflow (Issue #5)

…uto-start Python worker - Fix live monitoring style error by calling _live_monitor() helper directly - Remove default_parameters duplication from 10 workflow metadata files - Remove deprecated volume_mode parameter from 26 files across CLI, SDK, backend, and docs - Configure Python worker to start automatically with docker compose up - Clean up constants, validation, completion, and example files Fixes # - Live monitoring now works correctly with --live flag - Workflow metadata follows JSON Schema standard - Cleaner codebase without deprecated volume_mode - Python worker (most commonly used) starts by default

- Remove unused Literal import from backend findings model - Remove unnecessary f-string prefixes in CLI findings command - Optimize GitHub Actions to build only modified workers - Detect specific worker changes (python, secrets, rust, android, ossfuzz) - Build only changed workers instead of all 5 - Build all workers if docker-compose.yml changes - Significantly reduces CI build time

fix: resolve live monitoring bug, remove deprecated parameters, and auto-start Python worker

…obSF Comprehensive Android security testing workflow converted from Prefect to Temporal architecture: Modules (3): - JadxDecompiler: APK to Java source code decompilation - OpenGrepAndroid: Static analysis with Android-specific security rules - MobSFScanner: Comprehensive mobile security framework integration Custom Rules (13): - clipboard-sensitive-data, hardcoded-secrets, insecure-data-storage - insecure-deeplink, insecure-logging, intent-redirection - sensitive_data_sharedPreferences, sqlite-injection - vulnerable-activity, vulnerable-content-provider, vulnerable-service - webview-javascript-enabled, webview-load-arbitrary-url Workflow: - 6-phase Temporal workflow: download → Jadx → OpenGrep → MobSF → SARIF → upload - 4 activities: decompile_with_jadx, scan_with_opengrep, scan_with_mobsf, generate_android_sarif - SARIF output combining findings from all security tools Docker Worker: - ARM64 Mac compatibility via amd64 platform emulation - Pre-installed: Android SDK, Jadx 1.4.7, OpenGrep 1.45.0, MobSF 3.9.7 - MobSF runs as background service with API key auto-generation - Added aiohttp for async HTTP communication Test APKs: - BeetleBug.apk and shopnest.apk for workflow validation

- Fix activity names in workflow.py (get_target, upload_results, cleanup_cache) - Fix MobSF API key generation in Dockerfile startup script (cut delimiter) - Update activity parameter signatures to match actual implementations - Workflow now executes successfully with Jadx and OpenGrep

Implement platform-specific Dockerfile selection and graceful tool degradation to support both x86_64 and ARM64 (Apple Silicon) platforms. **Backend Changes:** - Add system info API endpoint (/system/info) exposing host filesystem paths - Add FUZZFORGE_HOST_ROOT environment variable to backend service - Add graceful degradation in MobSF activity for ARM64 platforms **CLI Changes:** - Implement multi-strategy path resolution (backend API, .fuzzforge marker, env var) - Add platform detection (linux/amd64 vs linux/arm64) - Add worker metadata.yaml reading for platform capabilities - Auto-select appropriate Dockerfile based on detected platform - Pass platform-specific env vars to docker-compose **Worker Changes:** - Create workers/android/metadata.yaml defining platform capabilities - Rename Dockerfile -> Dockerfile.amd64 (full toolchain with MobSF) - Create Dockerfile.arm64 (excludes MobSF due to Rosetta 2 incompatibility) - Update docker-compose.yml to use ${ANDROID_DOCKERFILE} variable **Workflow Changes:** - Handle MobSF "skipped" status gracefully in workflow - Log clear warnings when tools are unavailable on platform **Key Features:** - Automatic platform detection and Dockerfile selection - Graceful degradation when tools unavailable (MobSF on ARM64) - Works from any directory (backend API provides paths) - Manual override via environment variables - Clear user feedback about platform and selected Dockerfile **Benefits:** - Android workflow now works on Apple Silicon Macs - No code changes needed for other workflows - Convention established for future platform-specific workers Closes: MobSF Rosetta 2 incompatibility issue Implements: Platform-aware worker architecture (Option B)

- Add try-except block to conditionally import MobSFScanner in modules/android/__init__.py - Allows Android worker to start on ARM64 without MobSF dependencies (aiohttp) - MobSF activity gracefully skips on ARM64 with clear warning message - Remove workflow path detection logic (not needed - workflows receive directories) Platform-aware architecture fully functional on ARM64: - CLI detects ARM64 and selects Dockerfile.arm64 automatically - Worker builds and runs without MobSF on ARM64 - Jadx successfully decompiles APKs (4145 files from BeetleBug.apk) - OpenGrep finds security vulnerabilities (8 issues found) - MobSF gracefully skips with warning on ARM64 - Graceful degradation working as designed Tested with: ff workflow run android_static_analysis test_projects/android_test/ \ --wait --no-interactive apk_path=BeetleBug.apk decompile_apk=true Results: 8 security findings (1 ERROR, 7 WARNINGS)

Added [Unreleased] section documenting: - Android Static Analysis Workflow (Jadx, OpenGrep, MobSF) - Platform-Aware Worker Architecture with ARM64 support - Python SAST Workflow - CI/CD improvements and worker validation - CLI enhancements - Bug fixes and technical changes Fixed date typo: 2025-01-16 → 2025-10-16

- Remove unused imports from mobsf_scanner.py (asyncio, hashlib, json, Optional) - Remove unused variables from opengrep_android.py (start_col, end_col) - Remove duplicate Path import from workflow.py

Updated worker validation script to accept both: - Single Dockerfile pattern (existing workers) - Multi-platform Dockerfile pattern (Dockerfile.amd64, Dockerfile.arm64, etc.) This enables platform-aware worker architectures like the Android worker which uses different Dockerfiles for x86_64 and ARM64 platforms.

…ersion feat: Android Static Analysis Workflow with ARM64 Support

* feat: seed governance config and responses routing * Add env-configurable timeout for proxy providers * Integrate LiteLLM OTEL collector and update docs * Make .env.litellm optional for LiteLLM proxy * Add LiteLLM proxy integration with model-agnostic virtual keys Changes: - Bootstrap generates 3 virtual keys with individual budgets (CLI: $100, Task-Agent: $25, Cognee: $50) - Task-agent loads config at runtime via entrypoint script to wait for bootstrap completion - All keys are model-agnostic by default (no LITELLM_DEFAULT_MODELS restrictions) - Bootstrap handles database/env mismatch after docker prune by deleting stale aliases - CLI and Cognee configured to use LiteLLM proxy with virtual keys - Added comprehensive documentation in volumes/env/README.md Technical details: - task-agent entrypoint waits for keys in .env file before starting uvicorn - Bootstrap creates/updates TASK_AGENT_API_KEY, COGNEE_API_KEY, and OPENAI_API_KEY - Removed hardcoded API keys from docker-compose.yml - All services route through http://localhost:10999 proxy Generated with Claude Code https://claude.com/claude-code Co-Authored-By: Claude <noreply@anthropic.com> * Fix CLI not loading virtual keys from global .env Project .env files with empty OPENAI_API_KEY values were overriding the global virtual keys. Updated _load_env_file_if_exists to only override with non-empty values. Generated with Claude Code https://claude.com/claude-code Co-Authored-By: Claude <noreply@anthropic.com> * Fix agent executor not passing API key to LiteLLM The agent was initializing LiteLlm without api_key or api_base, causing authentication errors when using the LiteLLM proxy. Now reads from OPENAI_API_KEY/LLM_API_KEY and LLM_ENDPOINT environment variables and passes them to LiteLlm constructor. Generated with Claude Code https://claude.com/claude-code Co-Authored-By: Claude <noreply@anthropic.com> * Auto-populate project .env with virtual key from global config When running 'ff init', the command now checks for a global volumes/env/.env file and automatically uses the OPENAI_API_KEY virtual key if found. This ensures projects work with LiteLLM proxy out of the box without manual key configuration. Generated with Claude Code https://claude.com/claude-code Co-Authored-By: Claude <noreply@anthropic.com> * docs: Update README with LiteLLM configuration instructions Add note about LITELLM_GEMINI_API_KEY configuration and clarify that OPENAI_API_KEY default value should not be changed as it's used for the LLM proxy. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Refactor workflow parameters to use JSON Schema defaults Consolidates parameter defaults into JSON Schema format, removing the separate default_parameters field. Adds extract_defaults_from_json_schema() helper to extract defaults from the standard schema structure. Updates LiteLLM proxy config to use LITELLM_OPENAI_API_KEY environment variable. * Remove .env.example from task_agent * Fix MDX syntax error in llm-proxy.md * fix: apply default parameters from metadata.yaml automatically Fixed TemporalManager.run_workflow() to correctly apply default parameter values from workflow metadata.yaml files when parameters are not provided by the caller. Previous behavior: - When workflow_params was empty {}, the condition `if workflow_params and 'parameters' in metadata` would fail - Parameters would not be extracted from schema, resulting in workflows receiving only target_id with no other parameters New behavior: - Removed the `workflow_params and` requirement from the condition - Now explicitly checks for defaults in parameter spec - Applies defaults from metadata.yaml automatically when param not provided - Workflows receive all parameters with proper fallback: provided value > metadata default > None This makes metadata.yaml the single source of truth for parameter defaults, removing the need for workflows to implement defensive default handling. Affected workflows: - llm_secret_detection (was failing with KeyError) - All other workflows now benefit from automatic default application --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: tduhamel42 <tduhamel@fuzzinglabs.com>

Resolves validation error where agent_url was None when not explicitly provided. The TemporalManager applies defaults from metadata.yaml, not from module input schemas, so all parameters need defaults in the workflow metadata. Changes: - Add default agent_url, llm_model (gpt-5-mini), llm_provider (openai) - Expand file_patterns to 45 comprehensive patterns covering code, configs, secrets, and Docker files - Increase default limits: max_files (10), max_file_size (100KB), timeout (90s)

- Remove volumes/env/.env.example file - Update all documentation references to use .env.template instead - Update bootstrap script error message - Update .gitignore comment

…back Add comprehensive CLI commands for managing Temporal workers: - ff worker list - List workers with status and uptime - ff worker start <name> - Start specific worker with optional rebuild - ff worker stop - Safely stop all workers without affecting core services Improvements: - Live progress display during worker startup with Rich Status spinner - Real-time elapsed time counter and container state updates - Health check status tracking (starting → unhealthy → healthy) - Helpful contextual hints at 10s, 30s, 60s intervals - Better timeout messages showing last known state Worker management enhancements: - Use 'docker compose' (space) instead of 'docker-compose' (hyphen) - Stop workers individually with 'docker stop' to avoid stopping core services - Platform detection and Dockerfile selection (ARM64/AMD64) Documentation: - Updated docker-setup.md with CLI commands as primary method - Created comprehensive cli-reference.md with all commands and examples - Added worker management best practices

- Add test matrix configuration (.github/test-matrix.yaml) - Maps 8 workflows to workers, test projects, and parameters - Excludes LLM and OSS-Fuzz workflows - Defines fast, full, and platform test suites - Add workflow execution test script (scripts/test_workflows.py) - Executes workflows with parameter validation - Validates SARIF export and structure - Counts findings and measures execution time - Generates test summary reports - Add platform detection unit tests (cli/tests/test_platform_detection.py) - Tests platform detection (x86_64, aarch64, arm64) - Tests Dockerfile selection for multi-platform workers - Tests metadata.yaml parsing - Includes integration tests - Add GitHub Actions workflow (.github/workflows/test-workflows.yml) - Platform detection unit tests - Fast workflow tests (5 workflows on every PR) - Android platform-specific tests (AMD64 + ARM64) - Full workflow tests (on main/schedule) - Automatic log collection on failure - Add comprehensive testing documentation (docs/docs/development/testing.md) - Local testing guide - CI/CD testing explanation - Platform-specific testing guide - Debugging guide and best practices - Update test.yml with reference to new workflow tests - Remove tracked .fuzzforge/findings.db (already in .gitignore) Tested locally: - Single workflow test: python_sast (6.87s) ✅ - Fast test suite: 5/5 workflows passed ✅ - android_static_analysis (98.98s) ✅ - python_sast (6.78s) ✅ - secret_detection (38.04s) ✅ - gitleaks_detection (1.67s) ✅ - trufflehog_detection (1.64s) ✅

The GitHub Actions workflow was failing because the CLI depends on fuzzforge-sdk and fuzzforge-ai packages which are local to the monorepo and not available on PyPI. Updated all jobs to install local dependencies first: - platform-detection-tests - fast-workflow-tests - android-platform-tests - full-workflow-tests This ensures pip can resolve all dependencies correctly.

The WorkerManager tries to auto-detect docker-compose.yml during __init__, which fails in CI when tests run from cli/ directory. Updated the worker_manager fixture to provide a dummy compose file path, allowing tests to focus on platform detection logic without needing the actual compose file.

Test projects need to be initialized with 'ff init' to create .fuzzforge directories before workflows can run. Added initialization steps to all workflow test jobs: - Fast workflow tests - Android platform tests - Full workflow tests This ensures projects are properly set up in CI where .fuzzforge directories don't exist (they're in .gitignore).

The ff init command doesn't have a --non-interactive flag. The command already runs non-interactively by default, so the flag is not needed. This was causing initialization to fail with 'No such option' error.

- Increase android_static_analysis timeout from 300s to 600s Android worker needs more time to start and complete analysis in CI - Remove secret_detection from fast test suite Workflow experiences intermittent 404 in CI (timing/discovery issue) Still tested in full suite, gitleaks_detection and trufflehog_detection provide coverage of secrets worker in fast suite Result: 4/4 fast tests should pass reliably

Android Worker Platform Tests job was still using 'ff init' which requires interaction. Updated to use manual .fuzzforge creation like the fast-workflow-tests job. This fixes the 'No FuzzForge project found' error in android workflow tests.

Fixed broken Docusaurus links that were causing CI build failures: - docs/development/testing.md: workflows.md → create-workflow.md - docs/reference/cli-reference.md: workflows.md → create-workflow.md - docs/reference/cli-reference.md: ci-cd.md → cicd-integration.md This resolves the Docusaurus test deployment failure.

tduhamel42 and others added 30 commits September 29, 2025 15:12

Add files via upload

f0fd367

Initial commit

323a434

Issue templates

bb80c73

Update README.md

a4d204d

Update README.md

f465654

Merge pull request #1 from FuzzingLabs/pventuzelo-patch-1

3eacfcb

Update README.md

Fix parameters bug + installation issues

fb9ed42

add banner github

4c275d3

add banner github

Update README.md

44dfafd

add new banner github

Update README.md

4b53263

Update README.md

4ea3857

Update README.md

d00ac75

fix github star button

Update README.md

1dde5f5

Update README.md

ec11c11

remove logo website

Update README.md

f3c9964

add roadmap link

ci: add workflow_dispatch to action

4e889d5

fix: update path for upload build artifact

0a91d32

ci: change deployment branch from 'main' to 'master'

c36b1e9

ci: change trigger branch from 'main' to 'master'

cdb53c6

Fix deployment issues

6c1cbf9

Merge branch 'master' of github.com:FuzzingLabs/fuzzforge_ai

7382ea6

Add missing modules and workflow

b1e13ec

fix: update documentation links to new domain

5dd470f

docs(readme): fix repo name in git clone command

ad4c837

Merge pull request #7 from MegaRedHand/fix/wrong-repo-in-readme

a25c5e1

docs(readme): fix repo name in `git clone` command

Update readme with insecure registries

dd0a802

Merge branch 'master' of github.com:FuzzingLabs/fuzzforge_ai

987c495

refactor: removed monitor command and --live parameter

928a5f5

fix: removed erroneous example

a53d6c9

Merge pull request #11 from FuzzingLabs/fix/remove-erroneous-cli-example

28b0712

fix: removed erroneous example

tduhamel42 and others added 26 commits October 22, 2025 15:28

fix: Remove unused imports to pass linter

66e797a

Merge pull request #23 from FuzzingLabs/feature/python-sast-workflow

1c3c7a8

feat: Add Python SAST workflow (Issue #5)

Merge pull request #24 from FuzzingLabs/fix/cleanup-and-bugs

e180431

fix: resolve live monitoring bug, remove deprecated parameters, and auto-start Python worker

fix: resolve linter errors in Android modules

1fd525f

- Remove unused imports from mobsf_scanner.py (asyncio, hashlib, json, Optional) - Remove unused variables from opengrep_android.py (start_col, end_col) - Remove duplicate Path import from workflow.py

Merge pull request #28 from FuzzingLabs/feature/android-workflow-conv…

bd94d19

…ersion feat: Android Static Analysis Workflow with ARM64 Support

refactor: replace .env.example with .env.template in documentation

853a8be

- Remove volumes/env/.env.example file - Update all documentation references to use .env.template instead - Update bootstrap script error message - Update .gitignore comment

fix(ci): Remove invalid --non-interactive flag from ff init

2d045d3

The ff init command doesn't have a --non-interactive flag. The command already runs non-interactively by default, so the flag is not needed. This was causing initialization to fail with 'No such option' error.

tduhamel42 mentioned this pull request Oct 30, 2025

Release v0.7.3 - Android workflows, LiteLLM integration, ARM64 support #31

Closed

tduhamel42 closed this Nov 4, 2025

tduhamel42 force-pushed the dev branch from 8edd2d8 to 2ce20d3 Compare November 4, 2025 12:59

tduhamel42 mentioned this pull request Nov 4, 2025

Release v0.7.3 - Android workflows, LiteLLM integration, ARM64 support #32

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(test): Add Automated Workflow Testing Framework #30

feat(test): Add Automated Workflow Testing Framework #30

Uh oh!

tduhamel42 commented Oct 29, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

feat(test): Add Automated Workflow Testing Framework #30

feat(test): Add Automated Workflow Testing Framework #30

Uh oh!

Conversation

tduhamel42 commented Oct 29, 2025

Summary

What's New

1. Test Matrix Configuration ()

2. Workflow Execution Test Script (Error: PyYAML is required. Install with: pip install pyyaml)

3. Platform Detection Unit Tests ()

4. GitHub Actions Workflow ()

5. Comprehensive Documentation ()

Test Coverage

Local Test Results ✅

Benefits

Changes

Added

Modified

Removed

Testing Instructions

Run Locally

CI Testing

Related

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants