Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
8ff6641
feat: add RepositoryAnalysisAgent base structure
naaa760 Nov 17, 2025
519ab06
feat: implement RepositoryAnalysisAgent with LangGraph workflow
naaa760 Nov 17, 2025
b2e3205
feat: define Pydantic models for repository analysis
naaa760 Nov 17, 2025
8c6a580
feat: implement LangGraph nodes for repository analysis workflow
naaa760 Nov 17, 2025
fc607b3
feat: add LLM prompts for contributing guidelines analysis
naaa760 Nov 17, 2025
4a0ad3f
test: add comprehensive tests for RepositoryAnalysisAgent
naaa760 Nov 17, 2025
a65019a
docs: add README documentation for RepositoryAnalysisAgent
naaa760 Nov 17, 2025
3ae89e1
feat: add API endpoints for rule recommendations
naaa760 Nov 17, 2025
719cb76
update
naaa760 Nov 17, 2025
212bcfd
update
naaa760 Nov 17, 2025
1db7a60
update
naaa760 Nov 27, 2025
23b3710
add normalized diff metadata and LLM-friendly summaries for PR files.
naaa760 Nov 29, 2025
3dd71e1
surface the new diff context (details + top changed files) in the eng…
naaa760 Nov 29, 2025
544b3eb
regression test to keep the diff-summarizer behavior (truncation/empt…
naaa760 Nov 29, 2025
bdb4227
added the diff-aware validators (diff_pattern, related_tests, require…
naaa760 Nov 29, 2025
1c6cf40
added unit tests covering the new validators’ behavior
naaa760 Nov 29, 2025
23d60e1
refine the glob handling and keep the diff-aware validators passing
naaa760 Dec 1, 2025
4fefa76
switch Mastra rule bundle to the new diff-aware parameters
naaa760 Dec 1, 2025
63adcf9
document that the recommended rules now rely on the diff-aware valida…
naaa760 Dec 1, 2025
9d40d7a
add a 'diff-aware validators' section describing the new parameter
naaa760 Dec 1, 2025
ecbcbd4
add Diff-Aware Validators section
naaa760 Dec 2, 2025
b8e8154
update to reference new validators and inline sample rules instead o…
naaa760 Dec 2, 2025
37b9e60
remove problematic analytics section
naaa760 Dec 2, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 79 additions & 0 deletions docs/getting-started/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,52 @@ parameters:
excluded_branches: ["feature/*", "hotfix/*"]
```

### Diff-Aware Validators

Watchflow can now reason about pull-request diffs directly. The following parameter groups plug into diff-aware validators:

#### `diff_pattern`

Use this to require or forbid specific regex patterns inside matched files.

```yaml
parameters:
file_patterns:
- "packages/core/src/**/vector-query.ts"
require_patterns:
- "throw\\s+new\\s+Error"
forbidden_patterns:
- "console\\.log"
```

#### `related_tests`

Ensure core source changes include matching test updates.

```yaml
parameters:
source_patterns:
- "packages/core/src/**"
test_patterns:
- "tests/**"
- "packages/core/tests/**"
min_test_files: 1
```

#### `required_field_in_diff`

Verify that additions to certain files include a text fragment (for example, enforcing `description` on new agents).

```yaml
parameters:
file_patterns:
- "packages/core/src/agent/**"
required_text:
- "description"
```

These validators activate automatically when the parameters above are present, so you do not need to declare an `actions` block or manual mapping.

## Severity Levels

### Severity Configuration
Expand Down Expand Up @@ -219,6 +265,39 @@ rules:
required_teams: ["senior-engineers"]
```

## Diff-Aware Validators

Watchflow supports advanced validators that inspect actual PR diffs to enforce code-level patterns:

### diff_pattern
Enforce regex requirements or prohibitions within file patches.

```yaml
parameters:
file_patterns: ["packages/core/src/**/vector-query.ts"]
require_patterns: ["throw\\s+new\\s+Error"]
forbid_patterns: ["silent.*skip"]
```

### related_tests
Require test file updates when core code changes.

```yaml
parameters:
file_patterns: ["packages/core/src/**"]
require_test_updates: true
min_test_files: 1
```

### required_field_in_diff
Ensure new additions include required fields (e.g., agent descriptions).

```yaml
parameters:
file_patterns: ["packages/core/src/agent/**"]
required_text: "description"
```

## Best Practices

### Rule Design
Expand Down
134 changes: 134 additions & 0 deletions docs/reports/mastra-analysis.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,134 @@
# Mastra Repository Analysis

Mastra (`mastra-ai/mastra`) is a TypeScript-first agent framework for building production-grade AI assistants. The project has roughly **280 contributors**, **134 open pull requests**, and active CI coverage via GitHub Actions. This document captures the agreed-upon analysis from November 2025 so we can align on rule proposals before shipping automation.

## Repository Snapshot

- **Focus**: AI agents with tooling, memory, workflows, and multi-step orchestration
- **Primary language**: TypeScript with pnpm-based monorepo
- **Governance signals**: Detailed `CONTRIBUTING.md`, CODEOWNERS, changeset automation, active doc set
- **Pain points**: Complex LLM/provider integrations, repeated validation gaps, and regression risk in shared tooling layers

## Pull Request Sample (Nov 2025)

| PR | Title | Outcome | Notes |
| --- | --- | --- | --- |
| [#10180](https://github.com/mastra-ai/mastra/pull/10180) | feat: add custom model gateway support with automatic type generation | ✅ merged | Large feature: gateway registry, TS type generation, doc updates |
| [#10269](https://github.com/mastra-ai/mastra/pull/10269) | AI SDK tripwire data chunks | ✅ merged | Fixes & changeset for SDK data chunking bug |
| [#10141](https://github.com/mastra-ai/mastra/pull/10141) | fix: throw on invalid filter instead of silently skipping filtering | ✅ merged | Addressed regression where invalid filters returned unfiltered data |
| [#10300](https://github.com/mastra-ai/mastra/pull/10300) | Add description to type | ✅ merged | Unblocked Agent profile UI by exposing description metadata |
| [#9880](https://github.com/mastra-ai/mastra/pull/9880) | Fix clientjs clientTools execution | ✅ merged | Fixed client-side tool streaming regressions |
| [#9941](https://github.com/mastra-ai/mastra/pull/9941) | fix(core): input tool validation with no schema | ✅ merged | Restored validation for schema-less tool inputs |

## Pattern Summary

- **Validation & safety gaps (≈40%)** – invalid filters or schema-less tools silently bypassed safeguards.
- **Tooling & integration regressions (≈33%)** – clientTools streaming, AI SDK data chunking, URL handling.
- **Experience polish gaps (≈17%)** – missing agent descriptions prevented UI consistency.
- **High merge velocity** – most fixes merged quickly; reinforces need for automated guardrails so regressions are caught before release.

## Recommended Watchflow Rules

Rules intentionally avoid the optional `actions:` block so they remain compatible with the current loader. Enforcement intent is described in each `description` and reflected in `severity`.

```yaml
rules:
- description: "Block merges when PRs change filter validation logic without failing on invalid inputs"
enabled: true
severity: "high"
event_types: ["pull_request"]
parameters:
file_patterns:
- "packages/core/src/**/vector-query.ts"
- "packages/core/src/**/graph-rag.ts"
- "packages/core/src/**/filters/*.ts"
require_patterns:
- "throw\\s+new\\s+Error"
- "raise\\s+ValueError"
forbidden_patterns:
- "return\\s+.*filter\\s*$"
how_to_fix: "Ensure invalid filters raise descriptive errors instead of silently returning unfiltered results."

- description: "Require regression tests when modifying tool schema validation or client tool execution"
enabled: true
severity: "medium"
event_types: ["pull_request"]
parameters:
source_patterns:
- "packages/core/src/**/tool*.ts"
- "packages/core/src/agent/**"
- "packages/client/**"
test_patterns:
- "packages/core/tests/**"
- "tests/**"
min_test_files: 1
rationale: "Tool invocation changes have previously caused regressions in clientTools streaming."

- description: "Ensure every agent exposes a user-facing description for UI profiles"
enabled: true
severity: "low"
event_types: ["pull_request"]
parameters:
file_patterns:
- "packages/core/src/agent/**"
required_text:
- "description"
message: "Add or update the agent description so downstream UIs can render capabilities."

- description: "Block merges when URL or asset handling changes bypass provider capability checks"
enabled: true
severity: "high"
event_types: ["pull_request"]
parameters:
file_patterns:
- "packages/core/src/agent/message-list/**"
- "packages/core/src/llm/**"
require_patterns:
- "isUrlSupportedByModel"
forbidden_patterns:
- "downloadAssetsFromMessages\\(messages\\)"
how_to_fix: "Preserve remote URLs for providers that support them natively; only download assets for unsupported providers."
```

These concrete rules rely on the diff-aware validators recently added to Watchflow:

- `diff_pattern` ensures critical patches keep throwing exceptions or performing capability checks.
- `related_tests` requires PRs touching core modules to include matching test updates.
- `required_field_in_diff` verifies additions to agent definitions include a `description` so downstream UIs stay in sync.

Because the PR processor now passes normalized diffs into the engine, these validators operate deterministically without LLM fallbacks.

## PR Template Snippet

```markdown
## Repository Analysis Complete

We've analyzed your repository and identified key quality patterns based on recent PR history.

### Key Findings
- 40% of recent fixes patched validation or data-safety gaps (filters, schema-less tools).
- 33% addressed tool/LLM integration regressions (clientTools, AI SDK, URL handling).
- Tests/documentation often lag behind critical fixes, creating follow-up churn.

### Recommended Rules
- Block filter-validation changes that stop throwing on invalid inputs.
- Require regression tests when modifying tool schemas or clientTools execution.
- Enforce agent descriptions so UI consumers can present profiles.
- Block URL/asset handling changes that skip provider capability checks.

### Installation
1. Install the Watchflow GitHub App and grant access to `mastra-ai/mastra`.
2. Add `.watchflow/rules.yaml` with the rules above (see snippet).
3. Watchflow will start reporting violations through status checks immediately.

Questions? Reach out to the Watchflow team.
```

## Validation Plan

1. Keep the rule definitions in `docs/samples/mastra-watchflow-rules.yaml`.
2. Run `pytest tests/unit/test_mastra_rules_sample.py` to ensure every rule loads via `Rule.model_validate`.
3. (Optional) Use the repository analysis agent once PR-diff ingestion ships to simulate Mastra commits before opening an automated PR with these rules.

This keeps the deliverable lightweight, fully tested, and ready for the PR template automation flow discussed with Dimitris.

57 changes: 57 additions & 0 deletions docs/samples/mastra-watchflow-rules.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
rules:
- description: "Block merges when PRs change filter validation logic without failing on invalid inputs"
enabled: true
severity: "high"
event_types: ["pull_request"]
parameters:
file_patterns:
- "packages/core/src/**/vector-query.ts"
- "packages/core/src/**/graph-rag.ts"
- "packages/core/src/**/filters/*.ts"
require_patterns:
- "throw\\s+new\\s+Error"
- "raise\\s+ValueError"
forbidden_patterns:
- "return\\s+.*filter\\s*$"
how_to_fix: "Ensure invalid filters raise descriptive errors instead of silently returning unfiltered results."

- description: "Require regression tests when modifying tool schema validation or client tool execution"
enabled: true
severity: "medium"
event_types: ["pull_request"]
parameters:
source_patterns:
- "packages/core/src/**/tool*.ts"
- "packages/core/src/agent/**"
- "packages/client/**"
test_patterns:
- "packages/core/tests/**"
- "tests/**"
min_test_files: 1
rationale: "Tool invocation changes have previously caused regressions in clientTools streaming."

- description: "Ensure every agent exposes a user-facing description for UI profiles"
enabled: true
severity: "low"
event_types: ["pull_request"]
parameters:
file_patterns:
- "packages/core/src/agent/**"
required_text:
- "description"
message: "Add or update the agent description so downstream UIs can render capabilities."

- description: "Block merges when URL or asset handling changes bypass provider capability checks"
enabled: true
severity: "high"
event_types: ["pull_request"]
parameters:
file_patterns:
- "packages/core/src/agent/message-list/**"
- "packages/core/src/llm/**"
require_patterns:
- "isUrlSupportedByModel"
forbidden_patterns:
- "downloadAssetsFromMessages\\(messages\\)"
how_to_fix: "Preserve remote URLs for providers that support them natively; only download assets for unsupported providers."

5 changes: 2 additions & 3 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,8 @@ nav:
- Comparative Analysis: benchmarks.md
- Architecture:
- Overview: concepts/overview.md
- Case Studies:
- Mastra Repository Analysis: reports/mastra-analysis.md

# Plugins
plugins:
Expand Down Expand Up @@ -132,6 +134,3 @@ extra:
social:
- icon: fontawesome/brands/github
link: https://github.com/warestack/watchflow
analytics:
provider: google
property: ${GOOGLE_ANALYTICS_KEY}
13 changes: 12 additions & 1 deletion src/agents/engine_agent/prompts.py
Original file line number Diff line number Diff line change
Expand Up @@ -177,7 +177,7 @@ def _extract_event_context(event_data: dict, event_type: str) -> str:
context_parts = []

if event_type == "pull_request":
pr = event_data.get("pull_request", {})
pr = event_data.get("pull_request_details") or event_data.get("pull_request") or {}
context_parts.extend(
[
f"Title: {pr.get('title', 'N/A')}",
Expand All @@ -188,6 +188,17 @@ def _extract_event_context(event_data: dict, event_type: str) -> str:
]
)

files = event_data.get("files", [])
if files:
top_files = [file.get("filename") for file in files[:5] if file.get("filename")]
context_parts.append(
f"Changed Files ({len(files)} total): {top_files if top_files else '[filenames unavailable]'}"
)

diff_summary = event_data.get("diff_summary")
if diff_summary:
context_parts.append(f"Diff Summary:\n{diff_summary}")

elif event_type == "push":
context_parts.extend(
[
Expand Down
6 changes: 5 additions & 1 deletion src/agents/factory.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
from src.agents.base import BaseAgent
from src.agents.engine_agent import RuleEngineAgent
from src.agents.feasibility_agent import RuleFeasibilityAgent
from src.agents.repository_analysis_agent import RepositoryAnalysisAgent

logger = logging.getLogger(__name__)

Expand All @@ -34,6 +35,7 @@ def get_agent(agent_type: str, **kwargs: Any) -> BaseAgent:
>>> engine_agent = get_agent("engine")
>>> feasibility_agent = get_agent("feasibility")
>>> acknowledgment_agent = get_agent("acknowledgment")
>>> analysis_agent = get_agent("repository_analysis")
"""
agent_type = agent_type.lower()

Expand All @@ -43,6 +45,8 @@ def get_agent(agent_type: str, **kwargs: Any) -> BaseAgent:
return RuleFeasibilityAgent(**kwargs)
elif agent_type == "acknowledgment":
return AcknowledgmentAgent(**kwargs)
elif agent_type == "repository_analysis":
return RepositoryAnalysisAgent(**kwargs)
else:
supported = ", ".join(["engine", "feasibility", "acknowledgment"])
supported = ", ".join(["engine", "feasibility", "acknowledgment", "repository_analysis"])
raise ValueError(f"Unsupported agent type: {agent_type}. Supported: {supported}")
Loading