Flaky test: IncrementalBuildTest.AllProjectsHaveSameOutputDirectory fails on Windows with UnauthorizedAccessException ('Access to the path 'App1.dll' is denied')

[simonrozsival]

Summary

IncrementalBuildTest.AllProjectsHaveSameOutputDirectory intermittently fails on the Windows Xamarin.Android.Build.Tests leg with:

System.UnauthorizedAccessException : Access to the path 'App1.dll' is denied.
 at System.IO.FileSystem.RemoveDirectoryRecursive(...)
 at System.IO.FileSystem.RemoveDirectory(String fullPath, Boolean recursive)
 at Xamarin.ProjectTools.ProjectBuilder.Save(...) in .../Xamarin.ProjectTools/Common/ProjectBuilder.cs:line 109
 at Xamarin.ProjectTools.SolutionBuilder.Save() in .../Xamarin.ProjectTools/Common/SolutionBuilder.cs:line 29
 at Xamarin.ProjectTools.SolutionBuilder.ReBuild(String[] parameters) in .../Xamarin.ProjectTools/Common/SolutionBuilder.cs:line 85
 at Xamarin.Android.Build.Tests.IncrementalBuildTest.AllProjectsHaveSameOutputDirectory(AndroidRuntime runtime) in .../IncrementalBuildTest.cs:line 380

This is CI infrastructure flakiness (Windows IO race), not a product regression.

Recent occurrences (dnceng-public dotnet-android, definition 333)

Scanned 78 failed builds over 2026-06-23 → 2026-06-29; this test failed in 5, all on unrelated PRs:

Build	PR	Date (UTC)
1484378	#11669	2026-06-27
1483891	#11763	2026-06-27
1480690	#11617	2026-06-25
1480000	#11617	2026-06-24
1477513	#11669	2026-06-23

Both (CoreCLR) and (NativeAOT) parametrizations have been observed, with NativeAOT the more frequent.

Root cause

The failing line is ProjectBuilder.cs:109 → Directory.Delete (ProjectDirectory, true), reached via SolutionBuilder.ReBuild() → Save():

• SolutionBuilder.Save() constructs a fresh ProjectBuilder per project on every call, so BuiltBefore == false each time. As a result ProjectBuilder.Save() always takes the recursive-delete-and-repopulate branch — even on ReBuild().
• That recursive delete targets …/App1/, which contains obj/…/App1.dll just produced by the preceding Build().
• On Windows, a handle to the freshly-written DLL can still be open by the Roslyn shared-compilation server (VBCSCompiler) — which is not disabled in the test build args and lingers ~15 min — and/or Windows Defender / the search indexer scanning the new file. A handle opened without FILE_SHARE_DELETE makes DeleteFile return ERROR_ACCESS_DENIED, surfacing as UnauthorizedAccessException: Access to the path 'App1.dll' is denied.

MSBuild node reuse is already disabled (-nodeReuse:false), so a worker node is not the holder. NativeAOT flakes more because Release + ILC writes larger/more outputs and runs longer, widening the window in which a handle is still held when the next Save() deletes.

The team already half-acknowledges this race: SolutionBuilder.Dispose() swallows the same delete failure with the comment "This happens on CI occasionally, let's not fail the test".

Proposed fix

Robust, low-risk (recommended): add a delete-with-retry/back-off helper and use it where we recursively delete just-built directories (ProjectBuilder.Save:109 and ProjectBuilder.Cleanup):

// FileSystemUtils
public static void DeleteDirectoryWithRetry (string directory, int retries = 10)
{
 if (!Directory.Exists (directory))
 return;
 SetDirectoryWriteable (directory);
 for (int i = 0; ; i++) {
 try {
 Directory.Delete (directory, true);
 return;
 } catch (Exception e) when ((e is UnauthorizedAccessException || e is IOException) && i < retries) {
 Thread.Sleep (200 * (i + 1)); // back off; let AV/Roslyn release the handle
 SetDirectoryWriteable (directory);
 }
 }
}

This neutralizes the transient-lock race regardless of which process holds the handle, with no change to test semantics.

Complementary options:

• Structural (more correct long-term): have SolutionBuilder keep persistent ProjectBuilder instances so ReBuild()'s Save() takes the in-place UpdateProjectFiles path (BuiltBefore == true) and never recursively deletes a just-built directory. Larger change; touches other SolutionBuilder tests.
• Shrink the window: pass /p:UseSharedCompilation=false in the test build args so VBCSCompiler does not linger holding the DLL. Cheap, but does not cover Defender.

Affected files

• src/Xamarin.Android.Build.Tasks/Tests/Xamarin.ProjectTools/Common/ProjectBuilder.cs (line ~109, and Cleanup)
• src/Xamarin.Android.Build.Tasks/Tests/Xamarin.ProjectTools/Utilities/FileSystemUtils.cs
• src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/IncrementalBuildTest.cs (AllProjectsHaveSameOutputDirectory)