add metrics #212
Open
add metrics #212
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
| Sample | Status | Result |
|---|---|---|
| Python OpenAI | ❌ | failure |
| Python Agent Framework | ❌ | failure |
| Node.js OpenAI | ❌ | failure |
| Node.js LangChain | ❌ | failure |
| .NET Semantic Kernel | ❌ | failure |
| .NET Agent Framework | ❌ | failure |
📦 SDK Versions: View the workflow summary for detailed SDK version information.
Contributor
There was a problem hiding this comment.
Pull request overview
This pull request adds a comprehensive E2E test metrics collection and visualization system for the Agent 365 Samples repository. The system automatically tracks SDK version compatibility issues caught by E2E tests across different testing stages (pre-release, pre-checkin, post-checkin, release) and displays them on an interactive dashboard.
Changes:
- Adds automated metrics collection scripts that process E2E test results, validate SDK versions, and create GitHub issues for failures
- Implements a GitHub Actions workflow to collect metrics after E2E tests complete and update a metrics history
- Creates an interactive HTML dashboard to visualize SDK compatibility issues, test results, and error trends over time
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 28 comments.
Show a summary per file
| File | Description |
|---|---|
| scripts/e2e/Validate-SdkVersions.ps1 | Validates sample SDK versions against latest available from package registries (NuGet, PyPI, npm) |
| scripts/e2e/Emit-TestMetrics.ps1 | Collects test results and SDK versions into structured JSON metrics |
| scripts/e2e/Create-GitHubIssue.ps1 | Automatically creates categorized GitHub issues for E2E test failures |
| scripts/e2e/Aggregate-Metrics.ps1 | Aggregates individual metrics into a consolidated history file for trend analysis |
| docs/metrics/index.html | Interactive dashboard displaying metrics, charts, and SDK compatibility issues |
| docs/metrics/history.json | Initial empty metrics history file |
| .github/workflows/e2e-metrics.yml | Workflow to collect and publish metrics after E2E tests complete |
rahuldevikar761
approved these changes
Feb 10, 2026
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.OpenSSF Scorecard
Scanned Files
|
rahuldevikar761
requested changes
Feb 11, 2026
Collaborator
rahuldevikar761
left a comment
rahuldevikar761
left a comment
There was a problem hiding this comment.
Giving a second review on this, I don't think this will work from here. The metrics needs to be collected somewhere. Where are you collecting?
- Add metrics dashboard (docs/metrics/index.html) with: - SDK version validation against latest (including pre-release) - Error categorization by type (SDK, Test, Infrastructure, Agent) - Bugs caught by stage (pre-release, pre-checkin, post-checkin, release) - GitHub issue linking and PR tracking - Sample data for demonstration - Add metrics collection scripts: - Emit-TestMetrics.ps1: Collect test results and SDK versions - Aggregate-Metrics.ps1: Consolidate metrics into history.json - Validate-SdkVersions.ps1: Check samples use latest SDK versions - Create-GitHubIssue.ps1: Auto-create issues for test failures - Add e2e-metrics.yml workflow to collect metrics after E2E tests
- Dashboard shows SDK issues caught at each testing stage - Filters to only Agent 365 SDK packages - Error categorization with linked GitHub issues - Sample test results with success rates - Related PRs section with fix links - Animated counters and polished dark theme - Embedded demo data for offline preview - Empty history.json ready for real E2E data
- Use environment variables instead of direct interpolation in workflow - Fix 3 critical security issues in e2e-metrics.yml: 1. Line 52: github.event.workflow_run.head_branch in stage detection 2. Line 110: github context variables in PowerShell metrics placeholders 3. Line 155: github.event.workflow_run.head_branch in Emit-TestMetrics call - Pass context through env: block to prevent untrusted code execution
- Separate tables for each language ecosystem - Color-coded headers (green/purple/yellow) - Language-specific empty state messages - Updated package detection logic for each platform
PowerShell Scripts: - Add copyright headers to Emit-TestMetrics.ps1, Aggregate-Metrics.ps1 - Fix null reference for errorMessage with safer substring handling - Fix null handling in JSON parsing for history file - Use GITHUB_REPOSITORY env var as default for Repository param - Make dashboard URL configurable via METRICS_DASHBOARD_URL env var - Fix temp file cleanup with proper error handling - Fix null reference for regex Matches[3] in requirements parsing - Add comment about version comparison being simple equality - Improve gh issue create error handling Workflow: - Add concurrency group to prevent race conditions - Add documentation comment about permissions Dashboard: - Add escapeHtml() function to prevent XSS vulnerabilities - Escape user-controlled content (testName, errorMessage, PR title/author) - Add aria-label attributes to emojis and canvas elements for accessibility - Handle null lastUpdated date gracefully
- Checkout Agent365-metrics repo in workflow - Update paths to write metrics to metrics-repo/docs/ - Push history.json to Agent365-metrics instead of Agent365-Samples - Update dashboard link to Azure Function endpoint
- Remove artifact-based approach (E2E tests don't upload artifacts) - Use GitHub API to fetch workflow run job statuses directly - Create one entry per sample in dashboard-expected format - Simplify workflow by removing PowerShell scripts dependency - Push metrics to Agent365-metrics repo via METRICS_REPO_TOKEN
abdulanu0
force-pushed
the
users/anabdul/add-metrics
branch
from
34000af to
337e0da
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.