Test templates by RussKie · Pull Request #6086 · dotnet/extensions

RussKie · 2025-03-12T10:12:19Z

Leveraging template testing infra for a basic test of the AI template. More tests can be added later.
During a test the template is getting installed locally and then a new app is created, which is then gets snapshot-verified using Verify tooling.

| [2025-03-14T00:13:58] TestRun Information: > dotnet new install D:\Development\dotnet-extensions\src\ProjectTemplates\Microsoft.Extensions.AI.Templates\src\ChatWithCustomData --debug:custom-hive C:\Users\user\AppData\Local\Temp\hkgl3nh4.eq4\home
| [2025-03-14T00:13:58] TestRun Information: The following template packages will be installed:
| D:\Development\dotnet-extensions\src\ProjectTemplates\Microsoft.Extensions.AI.Templates\src\ChatWithCustomData
| Success: D:\Development\dotnet-extensions\src\ProjectTemplates\Microsoft.Extensions.AI.Templates\src\ChatWithCustomData installed the following templates:
| Template Name Short Name Language Tags | --------------- ---------- -------- -------------
| AI Chat Web App aichatweb [C#] Common/AI/Web
| [2025-03-14T00:13:59] TestRun Information: > dotnet new aichatweb -n aichatweb -o aichatweb --debug:custom-hive C:\Users\user\AppData\Local\Temp\hkgl3nh4.eq4\home --no-update-check
| [2025-03-14T00:13:59] TestRun Information: The template "AI Chat Web App" was created successfully.

For the local devex, it is very much worth installing Verify DiffEngineTray to approve all incoming snapshots (for the initial iteration).
It is also possible that git block staging the snapshot files due to line ending issues, and git config core.safecrlf false could be used as a jailbreak card.

Microsoft Reviewers: Open in CodeFlow

RussKie · 2025-03-12T23:55:51Z

I'm surprised it worked, but it did.

MackinnonBuck

Looks great! Just a few questions.

MackinnonBuck · 2025-03-13T23:52:35Z

+ <ItemGroup>
+<!--#if (IsOllama)
+ <PackageReference Include="OllamaSharp" Version="${OllamaSharpVersion}" />
+#elif (IsGHModels)
+ <PackageReference Include="OpenAI" Version="${OpenAIVersion}" />
+ <PackageReference Include="Microsoft.Extensions.AI.OpenAI" Version="${MicrosoftExtensionsAIVersion}" />
+#elif (IsAzureAiFoundry)
+ <PackageReference Include="Azure.AI.Projects" Version="${AzureAIProjectsVersion}" />
+ <PackageReference Include="Azure.AI.OpenAI" Version="${AzureAIOpenAIVersion}" />
+ <PackageReference Include="Microsoft.Extensions.AI.OpenAI" Version="${MicrosoftExtensionsAIVersion}" />
+#else -->
+ <PackageReference Include="Azure.AI.OpenAI" Version="${AzureAIOpenAIVersion}" />
+ <PackageReference Include="Microsoft.Extensions.AI.OpenAI" Version="${MicrosoftExtensionsAIVersion}" />
+<!--#endif -->
+<!--#if (UseManagedIdentity) -->
+ <PackageReference Include="Azure.Identity" Version="${AzureIdentityVersion}" />
+<!--#endif -->
+ <PackageReference Include="Microsoft.EntityFrameworkCore.Sqlite" Version="9.0.2" />
+ <PackageReference Include="Microsoft.Extensions.AI" Version="${MicrosoftExtensionsAIVersion}" />
+ <PackageReference Include="Microsoft.SemanticKernel.Core" Version="${MicrosoftSemanticKernelCoreVersion}" />
+ <PackageReference Include="PdfPig" Version="${PdfPigVersion}" />
+ <PackageReference Include="System.Linq.Async" Version="${SystemLinqAsyncVersion}" />
+<!--#if (UseAzureAISearch)
+ <PackageReference Include="Azure.Search.Documents" Version="${AzureSearchDocumentsVersion}" />
+ <PackageReference Include="Microsoft.SemanticKernel.Connectors.AzureAISearch" Version="${MicrosoftSemanticKernelConnectorsAzureAISearchVersion}" />
+#endif -->
+ </ItemGroup>


Why is the .in file included in the snapshot? Does this mean that the .in file gets included in the generated project?

That's a good question which I can't answer :)
I guess it does...

@MackinnonBuck - yes, the *.in file is included in the generated content.
I followed the steps the test performs:
PS D:\Development> dotnet new install D:\Development\dotnet-extensions\src\ProjectTemplates\Microsoft.Extensions.AI.Templates\src\ChatWithCustomData --debug:custom-hive C:\Users\user\AppData\Local\Temp\hkgl3nh4.eq4\home The following template packages will be installed: D:\Development\dotnet-extensions\src\ProjectTemplates\Microsoft.Extensions.AI.Templates\src\ChatWithCustomData Success: D:\Development\dotnet-extensions\src\ProjectTemplates\Microsoft.Extensions.AI.Templates\src\ChatWithCustomData installed the following templates: Template Name Short Name Language Tags --------------- ---------- -------- ------------- AI Chat Web App aichatweb [C#] Common/AI/Web PS D:\Development>pushd C:\Users\user\AppData\Local\Temp\TemplateEngine.Tests\test PS C:\Users\user\AppData\Local\Temp\TemplateEngine.Tests\test> dotnet new aichatweb -n aichatweb -o aichatweb --debug:custom-hive C:\Users\user\AppData\Local\Temp\hkgl3nh4.eq4\home --no-update-check The template "AI Chat Web App" was created successfully. PS C:\Users\user\AppData\Local\Temp\TemplateEngine.Tests\test> ls .\aichatweb\ Directory: C:\Users\user\AppData\Local\Temp\TemplateEngine.Tests\test\aichatweb Mode LastWriteTime Length Name ---- ------------- ------ ---- d----- 14/03/2025 11:17 AM Components d----- 14/03/2025 11:17 AM Properties d----- 14/03/2025 11:17 AM Services d----- 14/03/2025 11:17 AM wwwroot -a---- 14/03/2025 11:17 AM 889 aichatweb.csproj -a---- 14/03/2025 11:17 AM 2053 aichatweb.csproj.in -a---- 14/03/2025 11:17 AM 178 appsettings.Development.json -a---- 14/03/2025 11:17 AM 202 appsettings.json -a---- 14/03/2025 11:17 AM 2678 Program.cs -a---- 14/03/2025 11:17 AM 1194 README.md

Oh, it's probably because the test is looking at files directly inside the repo and treating them as template content. This ignores any preprocessing of such content that happens as part of building the Microsoft.Extensions.AI.Templates project (for example, here).

We don't have to block the PR on it, but it might be good to figure out how to fix that for the future. Maybe the Microsfot.Extensions.AI.Templates project copies its content to an output folder at build time, and then that's what gets used by the tests.

Oops, your comment didn't show up for me until I refreshed the page. If you produce a NuGet package from Microsfot.Extensions.AI.Templates, I'm guessing there's no .in file there, right? That's what I've seen from my local testing.

Correct - no *.in files in the nupkg

Leveraging [template testing infra](https://github.com/dotnet/templating/wiki/Templates-Testing-Tooling) for a basic test of the AI template. More tests can be added later. During a test the template is getting installed locally and then a new app is created, which is then gets snapshot-verified using Verify tooling. For the local devex, it is very much worth installing [Verify DiffEngineTray](https://github.com/VerifyTests/DiffEngine/blob/main/docs/tray.md) to approve all incoming snapshots (for the initial iteration). It is also possible that git block staging the snapshot files due to line ending issues, and "git config core.safecrlf false" could be used as a way to work around this.

github-actionsBot added the area-ai Microsoft.Extensions.AI libraries label Mar 12, 2025

dotnet-policy-serviceBot assigned RussKie Mar 12, 2025

RussKie force-pushed the test_templates branch from 9fa7681 to fa57b1dCompare March 12, 2025 10:13

jeffhandley reviewed Mar 12, 2025
View reviewed changes

Comment thread...rojectTemplates/Microsoft.Extensions.AI.Templates.InegrationTests/AichatwebTemplatesTests.cs

RussKie force-pushed the test_templates branch 2 times, most recently from a71ffcc to 00ef90eCompare March 12, 2025 23:15

RussKie marked this pull request as ready for review March 12, 2025 23:55

RussKie requested a review from a team as a code owner March 12, 2025 23:55

RussKie requested review from MackinnonBuck and jmatthiesen March 12, 2025 23:56

RussKie added area-ai-templates Microsoft.Extensions.AI.Templates and removed area-ai Microsoft.Extensions.AI libraries labels Mar 13, 2025

RussKie force-pushed the test_templates branch from 00ef90e to 855f1ecCompare March 13, 2025 02:45

RussKie added 2 commits March 13, 2025 14:09

Test templates
0d9b3a0

fixup! Test templates
496daa2

RussKie force-pushed the test_templates branch from 855f1ec to 496daa2Compare March 13, 2025 03:10

fixup! Test templates
dde346c

RussKie requested review from a team as code owners March 13, 2025 03:32

RussKie added 2 commits March 13, 2025 15:08

fixup! Test templates
573e1d3

fixup! Test templates
78dea9f

evgenyfedorov2 approved these changes Mar 13, 2025
View reviewed changes

MackinnonBuck approved these changes Mar 13, 2025
View reviewed changes

fixup! Test templates
90760c5

RussKie enabled auto-merge (squash) March 14, 2025 00:24

RussKie merged commit 0794cd9 into dotnet:mainMar 14, 2025

RussKie deleted the test_templates branch March 14, 2025 01:00

github-actionsBot locked and limited conversation to collaborators Apr 13, 2025

Navigation Menu

Uh oh!

Test templates - #6086

Test templates#6086
RussKie merged 6 commits into
dotnet:mainfrom
RussKie:test_templates

RussKie commented Mar 12, 2025•
edited
Loading

Uh oh!

Uh oh!

RussKie commented Mar 12, 2025

Uh oh!

MackinnonBuck left a comment

Uh oh!

Uh oh!

Uh oh!

MackinnonBuckMar 13, 2025

Uh oh!

RussKieMar 14, 2025

Uh oh!

RussKieMar 14, 2025•
edited
Loading

Uh oh!

MackinnonBuckMar 14, 2025

Uh oh!

MackinnonBuckMar 14, 2025

Uh oh!

MackinnonBuckMar 14, 2025

Uh oh!

RussKieMar 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

RussKie commented Mar 12, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Microsoft Reviewers: Open in CodeFlow

Uh oh!

Uh oh!

RussKie commented Mar 12, 2025

Uh oh!

MackinnonBuck left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

MackinnonBuckMar 13, 2025

Choose a reason for hiding this comment

Uh oh!

RussKieMar 14, 2025

Choose a reason for hiding this comment

Uh oh!

RussKieMar 14, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MackinnonBuckMar 14, 2025

Choose a reason for hiding this comment

Uh oh!

MackinnonBuckMar 14, 2025

Choose a reason for hiding this comment

Uh oh!

MackinnonBuckMar 14, 2025

Choose a reason for hiding this comment

Uh oh!

RussKieMar 14, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

RussKie commented Mar 12, 2025•
edited
Loading

RussKieMar 14, 2025•
edited
Loading