Skip to content

Build evaluation report in CI and when enabled - #5903

Merged
shyamnamboodiripad merged 12 commits into
dotnet:mainfrom
peterwald:ext-build
Feb 14, 2025
Merged

Build evaluation report in CI and when enabled#5903
shyamnamboodiripad merged 12 commits into
dotnet:mainfrom
peterwald:ext-build

Conversation

@peterwald

@peterwaldpeterwald commented Feb 13, 2025•
edited
Loading

Copy link
Copy Markdown
Member
  • Disable evaluation report builds by default.
  • Enable evaluation report builds in CI.
  • Extract custom targets in M.E.AI.Evaluation.Reporting to Directory.Build.targets

Fixes#5900

Reporting build can be enabled with dotnet build /p:EnableEvaluationReportBuild=true or by setting the property in src/Libraries/Microsoft.Extensions.AI.Evaluation.Reporting/Directory.Build.props

Microsoft Reviewers: Open in CodeFlow

@peterwaldpeterwald added the area-ai-eval Microsoft.Extensions.AI.Evaluation and related label Feb 13, 2025
@peterwaldpeterwald self-assigned this Feb 13, 2025
@peterwald
peterwald requested review from a team as code ownersFebruary 13, 2025 20:51
shyamnamboodiripad
shyamnamboodiripad reviewed Feb 13, 2025
View reviewed changes
Comment threadsrc/Libraries/Microsoft.Extensions.AI.Evaluation.Reporting/Directory.Build.props Outdated
@shyamnamboodiripad
shyamnamboodiripad requested a review from a team as a code ownerFebruary 13, 2025 22:35
stephentoub
stephentoub approved these changes Feb 13, 2025
View reviewed changes

@stephentoubstephentoub left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you

@dotnet-comment-bot

dotnet-comment-bot commented Feb 13, 2025

Copy link
Copy Markdown
Collaborator

‼️Found issues‼️

ProjectCoverage TypeExpectedActual
Microsoft.Extensions.Caching.HybridLine8682.92 🔻
Microsoft.Extensions.AI.OpenAILine7768.04 🔻
Microsoft.Extensions.AI.OpenAIBranch7749.8 🔻
Microsoft.Extensions.AI.EvaluationLine8858.67 🔻
Microsoft.Extensions.AI.EvaluationBranch8856.67 🔻
Microsoft.Extensions.AI.Evaluation.QualityLine887.57 🔻
Microsoft.Extensions.AI.Evaluation.QualityBranch8816.42 🔻
Microsoft.Extensions.AI.Evaluation.ReportingLine8872.06 🔻
Microsoft.Extensions.AI.Evaluation.ReportingBranch8864.8 🔻
Microsoft.Extensions.AI.Evaluation.ConsoleLine888.26 🔻
Microsoft.Extensions.AI.Evaluation.ConsoleBranch8817.07 🔻
Microsoft.Gen.MetadataExtractorLine9857.35 🔻
Microsoft.Gen.MetadataExtractorBranch9862.5 🔻
Microsoft.Extensions.AI.OllamaLine8078.2 🔻

🎉 Good job! The coverage increased 🎉
Update MinCodeCoverage in the project files.

ProjectExpectedActual
Microsoft.Extensions.AI.AzureAIInference9192
Microsoft.Extensions.AI8889

Full code coverage report: https://dev.azure.com/dnceng-public/public/_build/results?buildId=951939&view=codecoverage-tab

@dotnet-comment-bot

dotnet-comment-bot commented Feb 13, 2025

Copy link
Copy Markdown
Collaborator

‼️Found issues‼️

ProjectCoverage TypeExpectedActual
Microsoft.Extensions.Caching.HybridLine8682.77 🔻
Microsoft.Extensions.AI.OpenAILine7768.04 🔻
Microsoft.Extensions.AI.OpenAIBranch7749.8 🔻
Microsoft.Extensions.AI.Evaluation.QualityLine887.57 🔻
Microsoft.Extensions.AI.Evaluation.QualityBranch8816.42 🔻
Microsoft.Extensions.AI.EvaluationLine8858.67 🔻
Microsoft.Extensions.AI.EvaluationBranch8856.67 🔻
Microsoft.Extensions.AI.OllamaLine8078.2 🔻
Microsoft.Gen.MetadataExtractorLine9857.35 🔻
Microsoft.Gen.MetadataExtractorBranch9862.5 🔻
Microsoft.Extensions.AI.Evaluation.ReportingLine8872.06 🔻
Microsoft.Extensions.AI.Evaluation.ReportingBranch8864.8 🔻
Microsoft.Extensions.AI.Evaluation.ConsoleLine888.26 🔻
Microsoft.Extensions.AI.Evaluation.ConsoleBranch8817.07 🔻

🎉 Good job! The coverage increased 🎉
Update MinCodeCoverage in the project files.

ProjectExpectedActual
Microsoft.Extensions.AI.AzureAIInference9192
Microsoft.Extensions.AI8889

Full code coverage report: https://dev.azure.com/dnceng-public/public/_build/results?buildId=951976&view=codecoverage-tab

@shyamnamboodiripad
shyamnamboodiripad merged commit 51a5d43 into dotnet:mainFeb 14, 2025
@peterwald
peterwald deleted the ext-build branch February 14, 2025 01:17
@rnelsonrnelson mentioned this pull request Mar 12, 2025
Merged
@github-actionsgithub-actionsBot locked and limited conversation to collaborators Mar 16, 2025
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@shyamnamboodiripadshyamnamboodiripadshyamnamboodiripad approved these changes
@stephentoubstephentoubstephentoub approved these changes
+1 more reviewer
@RussKieRussKieRussKie approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

@peterwaldpeterwald

Labels

area-ai-evalMicrosoft.Extensions.AI.Evaluation and related

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Solution no longer builds after merging M.E.AI.Evaluation source

5 participants

@peterwald@dotnet-comment-bot@stephentoub@RussKie@shyamnamboodiripad