Skip to content

Build evaluation report in CI and when enabled #5903

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 12 commits into from
Feb 14, 2025
Merged

Build evaluation report in CI and when enabled #5903

merged 12 commits into from
Feb 14, 2025

Conversation

peterwald
Copy link
Member

@peterwald peterwald commented Feb 13, 2025 •
edited
Loading

  • Disable evaluation report builds by default.
  • Enable evaluation report builds in CI.
  • Extract custom targets in M.E.AI.Evaluation.Reporting to Directory.Build.targets

Fixes #5900

Reporting build can be enabled with dotnet build /p:EnableEvaluationReportBuild=true or by setting the property in src/Libraries/Microsoft.Extensions.AI.Evaluation.Reporting/Directory.Build.props

Microsoft Reviewers: Open in CodeFlow

@peterwald peterwald added the area-ai-eval Microsoft.Extensions.AI.Evaluation and related label Feb 13, 2025
@peterwald peterwald self-assigned this Feb 13, 2025
@peterwald peterwald requested review from a team as code owners February 13, 2025 20:51
@shyamnamboodiripad shyamnamboodiripad requested a review from a team as a code owner February 13, 2025 22:35
stephentoub
stephentoub approved these changes Feb 13, 2025
View reviewed changes
Copy link
Member

@stephentoub stephentoub left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you

@dotnet-comment-bot
Copy link
Collaborator

‼️ Found issues ‼️

Project Coverage Type Expected Actual
Microsoft.Extensions.Caching.Hybrid Line 86 82.92 🔻
Microsoft.Extensions.AI.OpenAI Line 77 68.04 🔻
Microsoft.Extensions.AI.OpenAI Branch 77 49.8 🔻
Microsoft.Extensions.AI.Evaluation Line 88 58.67 🔻
Microsoft.Extensions.AI.Evaluation Branch 88 56.67 🔻
Microsoft.Extensions.AI.Evaluation.Quality Line 88 7.57 🔻
Microsoft.Extensions.AI.Evaluation.Quality Branch 88 16.42 🔻
Microsoft.Extensions.AI.Evaluation.Reporting Line 88 72.06 🔻
Microsoft.Extensions.AI.Evaluation.Reporting Branch 88 64.8 🔻
Microsoft.Extensions.AI.Evaluation.Console Line 88 8.26 🔻
Microsoft.Extensions.AI.Evaluation.Console Branch 88 17.07 🔻
Microsoft.Gen.MetadataExtractor Line 98 57.35 🔻
Microsoft.Gen.MetadataExtractor Branch 98 62.5 🔻
Microsoft.Extensions.AI.Ollama Line 80 78.2 🔻

🎉 Good job! The coverage increased 🎉
Update MinCodeCoverage in the project files.

Project Expected Actual
Microsoft.Extensions.AI.AzureAIInference 91 92
Microsoft.Extensions.AI 88 89

Full code coverage report: https://dev.azure.com/dnceng-public/public/_build/results?buildId=951939&view=codecoverage-tab

@dotnet-comment-bot
Copy link
Collaborator

‼️ Found issues ‼️

Project Coverage Type Expected Actual
Microsoft.Extensions.Caching.Hybrid Line 86 82.77 🔻
Microsoft.Extensions.AI.OpenAI Line 77 68.04 🔻
Microsoft.Extensions.AI.OpenAI Branch 77 49.8 🔻
Microsoft.Extensions.AI.Evaluation.Quality Line 88 7.57 🔻
Microsoft.Extensions.AI.Evaluation.Quality Branch 88 16.42 🔻
Microsoft.Extensions.AI.Evaluation Line 88 58.67 🔻
Microsoft.Extensions.AI.Evaluation Branch 88 56.67 🔻
Microsoft.Extensions.AI.Ollama Line 80 78.2 🔻
Microsoft.Gen.MetadataExtractor Line 98 57.35 🔻
Microsoft.Gen.MetadataExtractor Branch 98 62.5 🔻
Microsoft.Extensions.AI.Evaluation.Reporting Line 88 72.06 🔻
Microsoft.Extensions.AI.Evaluation.Reporting Branch 88 64.8 🔻
Microsoft.Extensions.AI.Evaluation.Console Line 88 8.26 🔻
Microsoft.Extensions.AI.Evaluation.Console Branch 88 17.07 🔻

🎉 Good job! The coverage increased 🎉
Update MinCodeCoverage in the project files.

Project Expected Actual
Microsoft.Extensions.AI.AzureAIInference 91 92
Microsoft.Extensions.AI 88 89

Full code coverage report: https://dev.azure.com/dnceng-public/public/_build/results?buildId=951976&view=codecoverage-tab

@shyamnamboodiripad shyamnamboodiripad merged commit 51a5d43 into dotnet:main Feb 14, 2025
6 checks passed
@peterwald peterwald deleted the ext-build branch February 14, 2025 01:17
@rnelson rnelson mentioned this pull request Mar 12, 2025
Merged
@github-actions github-actions bot locked and limited conversation to collaborators Mar 16, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@shyamnamboodiripad shyamnamboodiripad shyamnamboodiripad approved these changes

@stephentoub stephentoub stephentoub approved these changes

+1 more reviewer

@RussKie RussKie RussKie approved these changes

Reviewers whose approvals may not affect merge requirements
Assignees

@peterwald peterwald

Labels
area-ai-eval Microsoft.Extensions.AI.Evaluation and related
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Solution no longer builds after merging M.E.AI.Evaluation source
5 participants
@peterwald @dotnet-comment-bot @stephentoub @RussKie @shyamnamboodiripad