Re-enable Microsoft.Extensions.Hosting shutdown functional coverage in CI and harden shutdown startup/Helix/NET481 compatibility - #128252

Merged
rosebyte merged 44 commits into
mainfrom
copilot/enable-functional-tests-on-hosting
Jun 24, 2026
Merged

Re-enable Microsoft.Extensions.Hosting shutdown functional coverage in CI and harden shutdown startup/Helix/NET481 compatibility#128252
rosebyte merged 44 commits into
mainfrom
copilot/enable-functional-tests-on-hosting

Conversation

CopilotAI commented May 15, 2026

Copy link
Copy Markdown
Contributor

This PR now includes the full follow-up hardening required by CI feedback after re-enabling Microsoft.Extensions.Hosting functional shutdown coverage.

Final scope of changes

  • Re-enabled Hosting FunctionalTests in CI (removed CI gating/active issue suppression).
  • Updated shutdown test deployment to run from deployed output (AppContext.BaseDirectory) instead of source-tree project discovery.
  • Removed the load-bearing PreservePublishedApplicationForDebugging dependency by using a borrowed published app path that does not delete test output on cleanup. PublishedApplication.Dispose() is now virtual; BorrowedPublishedApplication overrides it as a no-op so teardown never deletes the test output directory.
  • Replaced external kill process usage in shutdown signaling with libc signal calls for Helix/minimal-image compatibility.
  • Centralized the libckill P/Invoke in ProcessExtensions.SendSignal(int pid, int signal); removed the duplicate declaration from ShutdownTests so both signal paths share a single interop entry point. The underlying P/Invoke method is named Kill (PascalCase) with EntryPoint = "kill" to match C# naming conventions while preserving the correct libc symbol binding. SendSignal explicitly throws PlatformNotSupportedException on Windows so callers get a clear, actionable error rather than a DllNotFoundException from the libc P/Invoke.
  • Resolved Unix SIGINT/SIGTERM numbers at runtime via SystemNative_GetPlatformSignalNumber instead of hard-coding them, matching the runtime's platform-specific signal mapping approach. Signal number initialization is gated behind an OS check so Windows does not attempt the P/Invoke during type initialization.
  • Kept Unix cleanup best-effort and resilient:
    • graceful termination first (SIGTERM via libc)
    • hard-kill fallback (Process.Kill()) when graceful termination times out or fails
    • tolerance for missing process tools / constrained environments
  • Increased startup/shutdown wait windows to 180 seconds for stress/JIT-stress stability.
  • Increased the final shutdown exit grace period before escalation to Process.Kill() to 30 seconds, reducing false failures when the app has finished shutdown logging but needs a little longer to exit cleanly under load.
  • Fixed the shutdown test startup race by wiring output observation before DeployAsync() starts process output capture via SelfHostDeployer.OutputReceived (now internal), so the initial Started line cannot be missed.
  • Fixed net481 compatibility by using Marshal.GetLastWin32Error().
  • Fixed WaitForExitOrKill race: now uses the boolean result from WaitForExit(timeout) directly and wraps Process.Kill() in a catch (InvalidOperationException) to tolerate the window where the process exits between the wait and the kill call.

Validation

  • ./build.sh clr+libs -rc release succeeds.
  • ./build.sh clr+libs+host -rc release -lc release succeeds.
  • Microsoft.Extensions.Hosting.Functional.Tests builds successfully for net481.
  • dotnet build /t:test Microsoft.Extensions.Hosting.Functional.Tests.csproj -c Release -f net11.0 /p:TestFilter=Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests passes (2/2).
  • CI-feedback-driven shutdown/process-cleanup paths were updated to match Helix/runtime constraints and avoid prior failure modes.

CopilotAI requested review from Copilot and removed request for CopilotMay 15, 2026 12:32
CopilotAI linked an issue May 15, 2026 that may be closed by this pull request
@github-actionsgithub-actionsBot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label May 15, 2026
CopilotAI requested review from Copilot and removed request for CopilotMay 15, 2026 13:03
CopilotAI changed the title [WIP] Fix FunctionalTests/TestApp on Extensions HostingRe-enable Microsoft.Extensions.Hosting FunctionalTests/TestApp shutdown coverageMay 15, 2026
CopilotAI requested a review from rosebyteMay 15, 2026 13:07
@github-actionsgithub-actionsBot added area-Extensions-Hosting and removed needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners labels May 15, 2026
@rosebyte

Copy link
Copy Markdown
Member

@copilot , the shutdown tests are failing with this error on helix machines:

Comments
{ "HelixJobId": "bf5b8a6f-0bdd-40b3-9fbc-f0eec7148f40", "HelixWorkItemName": "Microsoft.Extensions.Hosting.Functional.Tests" }

Error message
System.IO.DirectoryNotFoundException : Could not find Microsoft.Extensions.Hosting.TestApp.csproj from base directory '/root/helix/work/workitem/e/'.

Stack trace
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.GetApplicationPath() in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 173
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ExecuteShutdownTest(String testName, String shutdownMechanic) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 53
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestRun() in /_/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 35
--- End of stack trace from previous location ---

CopilotAI requested review from Copilot and removed request for CopilotMay 18, 2026 11:55

CopilotAI commented May 18, 2026

Copy link
Copy Markdown
ContributorAuthor

@copilot , the shutdown tests are failing with this error on helix machines:

...

Fixed in b9cd702. I removed the source-tree lookup and switched the shutdown tests to run TestApp from the deployed test output (AppContext.BaseDirectory) using a custom ApplicationPublisher, so Helix no longer requires Microsoft.Extensions.Hosting.TestApp.csproj to exist in the workitem.

CopilotAI changed the title Re-enable Microsoft.Extensions.Hosting FunctionalTests/TestApp shutdown coverageRe-enable Microsoft.Extensions.Hosting FunctionalTests/TestApp shutdown coverage across local and Helix environmentsMay 18, 2026
@rosebyte

Copy link
Copy Markdown
Member

@copilot , this time it failed with this error:

Test Failures (700 tests failed)
🔹 [All failing tests from runtime]

⚠️ 698 failing tests not displayed for this pipeline due to space limitations

❌ Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestRun [Console][Details][Artifacts][0.70% failure rate]
[ 🚧 Create issue against infrastructure team][ 📄 Create issue in this repo]
Failing Configuration
AzureLinux.3.0.ArmArch.Open
Exception Message
System.ComponentModel.Win32Exception : An error occurred trying to start process 'pgrep' with working directory '/root/helix/work/workitem/e'. No such file or directory
CallStack
at Microsoft.Win32.SafeHandles.SafeProcessHandle.ForkAndExecProcess(ProcessStartInfo startInfo, String resolvedFilename, String[] argv, IDictionary2 env, String cwd, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, Boolean usesTerminal, SafeHandle[] inheritedHandles, Holder& waitStateHolder, Boolean throwOnNoExec) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 369 at Microsoft.Win32.SafeHandles.SafeProcessHandle.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles, Holder& waitStateHolder) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 213 at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Unix.cs:line 363 at System.Diagnostics.Process.Start() in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1272 at System.Diagnostics.Process.Start(ProcessStartInfo startInfo) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1383 at Microsoft.Extensions.Internal.ProcessExtensions.RunProcessAndWaitForExit(String fileName, String arguments, TimeSpan timeout, String& stdout) in /_/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 93 at Microsoft.Extensions.Internal.ProcessExtensions.GetAllChildIdsUnix(Int32 parentId, ISet1 children, TimeSpan timeout) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 44
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process, TimeSpan timeout) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 33
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 17
at Microsoft.Extensions.Hosting.IntegrationTesting.ApplicationDeployer.ShutDownIfAnyHostProcess(Process hostProcess) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/ApplicationDeployer.cs:line 126
at Microsoft.Extensions.Hosting.IntegrationTesting.SelfHostDeployer.Dispose() in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/SelfHostDeployer.cs:line 179
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ExecuteShutdownTest(String testName, String shutdownMechanic) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 107
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestRun() in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 34
--- End of stack trace from previous location ---
❌ Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestWaitForShutdown [Console][Details][Artifacts][0.69% failure rate]
[ 🚧 Create issue against infrastructure team][ 📄 Create issue in this repo]
Failing Configuration
AzureLinux.3.0.ArmArch.Open
Exception Message
System.ComponentModel.Win32Exception : An error occurred trying to start process 'pgrep' with working directory '/root/helix/work/workitem/e'. No such file or directory
CallStack
at Microsoft.Win32.SafeHandles.SafeProcessHandle.ForkAndExecProcess(ProcessStartInfo startInfo, String resolvedFilename, String[] argv, IDictionary2 env, String cwd, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, Boolean usesTerminal, SafeHandle[] inheritedHandles, Holder& waitStateHolder, Boolean throwOnNoExec) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 369 at Microsoft.Win32.SafeHandles.SafeProcessHandle.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles, Holder& waitStateHolder) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 213 at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Unix.cs:line 363 at System.Diagnostics.Process.Start() in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1272 at System.Diagnostics.Process.Start(ProcessStartInfo startInfo) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1383 at Microsoft.Extensions.Internal.ProcessExtensions.RunProcessAndWaitForExit(String fileName, String arguments, TimeSpan timeout, String& stdout) in /_/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 93 at Microsoft.Extensions.Internal.ProcessExtensions.GetAllChildIdsUnix(Int32 parentId, ISet1 children, TimeSpan timeout) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 44
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process, TimeSpan timeout) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 33
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 17
at Microsoft.Extensions.Hosting.IntegrationTesting.ApplicationDeployer.ShutDownIfAnyHostProcess(Process hostProcess) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/ApplicationDeployer.cs:line 126
at Microsoft.Extensions.Hosting.IntegrationTesting.SelfHostDeployer.Dispose() in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/SelfHostDeployer.cs:line 179
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ExecuteShutdownTest(String testName, String shutdownMechanic) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 107
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestWaitForShutdown() in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 41
--- End of stack trace from previous location ---

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

…plicate in ShutdownTests
Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

CopilotAIand others added 4 commits June 15, 2026 15:53
Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>
…on-hosting' into copilot/enable-functional-tests-on-hosting
Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>
Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

rosebyteand others added 2 commits June 16, 2026 23:04
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:83

  • OutputReceived is raised from background threads, but output is being mutated via output += ... without synchronization. This can lead to lost/garbled output (and flaky assertions) when multiple lines arrive quickly. Consider using a thread-safe accumulator (e.g., a StringBuilder guarded by a lock, or enqueue lines into a concurrent collection and join at the end) rather than mutating a shared string from the event callback.
 var started = new TaskCompletionSource<int>(TaskCreationOptions.RunContinuationsAsynchronously);
var completed = new TaskCompletionSource<int>(TaskCreationOptions.RunContinuationsAsynchronously);
var output = string.Empty;
deployer.OutputReceived += (sender, args) =>
{
if (!string.IsNullOrEmpty(args.Data) && args.Data.StartsWith(StartedMessage, StringComparison.Ordinal))
{
output += args.Data.Substring(StartedMessage.Length) + '\n';
started.TrySetResult(0);
}

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Enable FunctionalTests/TestApp on Extensions Hosting

5 participants

@rosebyte@svick@mrek-msft
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

Re-enable Microsoft.Extensions.Hosting shutdown functional coverage in CI and harden shutdown startup/Helix/NET481 compatibility - #128252

Merged
rosebyte merged 44 commits into
mainfrom
copilot/enable-functional-tests-on-hosting
Jun 24, 2026
Merged

Re-enable Microsoft.Extensions.Hosting shutdown functional coverage in CI and harden shutdown startup/Helix/NET481 compatibility#128252
rosebyte merged 44 commits into
mainfrom
copilot/enable-functional-tests-on-hosting

Conversation

CopilotAI commented May 15, 2026

Copy link
Copy Markdown
Contributor

This PR now includes the full follow-up hardening required by CI feedback after re-enabling Microsoft.Extensions.Hosting functional shutdown coverage.

Final scope of changes

  • Re-enabled Hosting FunctionalTests in CI (removed CI gating/active issue suppression).
  • Updated shutdown test deployment to run from deployed output (AppContext.BaseDirectory) instead of source-tree project discovery.
  • Removed the load-bearing PreservePublishedApplicationForDebugging dependency by using a borrowed published app path that does not delete test output on cleanup. PublishedApplication.Dispose() is now virtual; BorrowedPublishedApplication overrides it as a no-op so teardown never deletes the test output directory.
  • Replaced external kill process usage in shutdown signaling with libc signal calls for Helix/minimal-image compatibility.
  • Centralized the libckill P/Invoke in ProcessExtensions.SendSignal(int pid, int signal); removed the duplicate declaration from ShutdownTests so both signal paths share a single interop entry point. The underlying P/Invoke method is named Kill (PascalCase) with EntryPoint = "kill" to match C# naming conventions while preserving the correct libc symbol binding. SendSignal explicitly throws PlatformNotSupportedException on Windows so callers get a clear, actionable error rather than a DllNotFoundException from the libc P/Invoke.
  • Resolved Unix SIGINT/SIGTERM numbers at runtime via SystemNative_GetPlatformSignalNumber instead of hard-coding them, matching the runtime's platform-specific signal mapping approach. Signal number initialization is gated behind an OS check so Windows does not attempt the P/Invoke during type initialization.
  • Kept Unix cleanup best-effort and resilient:
    • graceful termination first (SIGTERM via libc)
    • hard-kill fallback (Process.Kill()) when graceful termination times out or fails
    • tolerance for missing process tools / constrained environments
  • Increased startup/shutdown wait windows to 180 seconds for stress/JIT-stress stability.
  • Increased the final shutdown exit grace period before escalation to Process.Kill() to 30 seconds, reducing false failures when the app has finished shutdown logging but needs a little longer to exit cleanly under load.
  • Fixed the shutdown test startup race by wiring output observation before DeployAsync() starts process output capture via SelfHostDeployer.OutputReceived (now internal), so the initial Started line cannot be missed.
  • Fixed net481 compatibility by using Marshal.GetLastWin32Error().
  • Fixed WaitForExitOrKill race: now uses the boolean result from WaitForExit(timeout) directly and wraps Process.Kill() in a catch (InvalidOperationException) to tolerate the window where the process exits between the wait and the kill call.

Validation

  • ./build.sh clr+libs -rc release succeeds.
  • ./build.sh clr+libs+host -rc release -lc release succeeds.
  • Microsoft.Extensions.Hosting.Functional.Tests builds successfully for net481.
  • dotnet build /t:test Microsoft.Extensions.Hosting.Functional.Tests.csproj -c Release -f net11.0 /p:TestFilter=Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests passes (2/2).
  • CI-feedback-driven shutdown/process-cleanup paths were updated to match Helix/runtime constraints and avoid prior failure modes.

CopilotAI requested review from Copilot and removed request for CopilotMay 15, 2026 12:32
CopilotAI linked an issue May 15, 2026 that may be closed by this pull request
@github-actionsgithub-actionsBot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label May 15, 2026
CopilotAI requested review from Copilot and removed request for CopilotMay 15, 2026 13:03
CopilotAI changed the title [WIP] Fix FunctionalTests/TestApp on Extensions HostingRe-enable Microsoft.Extensions.Hosting FunctionalTests/TestApp shutdown coverageMay 15, 2026
CopilotAI requested a review from rosebyteMay 15, 2026 13:07
@github-actionsgithub-actionsBot added area-Extensions-Hosting and removed needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners labels May 15, 2026
@rosebyte

Copy link
Copy Markdown
Member

@copilot , the shutdown tests are failing with this error on helix machines:

Comments
{ "HelixJobId": "bf5b8a6f-0bdd-40b3-9fbc-f0eec7148f40", "HelixWorkItemName": "Microsoft.Extensions.Hosting.Functional.Tests" }

Error message
System.IO.DirectoryNotFoundException : Could not find Microsoft.Extensions.Hosting.TestApp.csproj from base directory '/root/helix/work/workitem/e/'.

Stack trace
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.GetApplicationPath() in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 173
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ExecuteShutdownTest(String testName, String shutdownMechanic) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 53
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestRun() in /_/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 35
--- End of stack trace from previous location ---

CopilotAI requested review from Copilot and removed request for CopilotMay 18, 2026 11:55

CopilotAI commented May 18, 2026

Copy link
Copy Markdown
ContributorAuthor

@copilot , the shutdown tests are failing with this error on helix machines:

...

Fixed in b9cd702. I removed the source-tree lookup and switched the shutdown tests to run TestApp from the deployed test output (AppContext.BaseDirectory) using a custom ApplicationPublisher, so Helix no longer requires Microsoft.Extensions.Hosting.TestApp.csproj to exist in the workitem.

CopilotAI changed the title Re-enable Microsoft.Extensions.Hosting FunctionalTests/TestApp shutdown coverageRe-enable Microsoft.Extensions.Hosting FunctionalTests/TestApp shutdown coverage across local and Helix environmentsMay 18, 2026
@rosebyte

Copy link
Copy Markdown
Member

@copilot , this time it failed with this error:

Test Failures (700 tests failed)
🔹 [All failing tests from runtime]

⚠️ 698 failing tests not displayed for this pipeline due to space limitations

❌ Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestRun [Console][Details][Artifacts][0.70% failure rate]
[ 🚧 Create issue against infrastructure team][ 📄 Create issue in this repo]
Failing Configuration
AzureLinux.3.0.ArmArch.Open
Exception Message
System.ComponentModel.Win32Exception : An error occurred trying to start process 'pgrep' with working directory '/root/helix/work/workitem/e'. No such file or directory
CallStack
at Microsoft.Win32.SafeHandles.SafeProcessHandle.ForkAndExecProcess(ProcessStartInfo startInfo, String resolvedFilename, String[] argv, IDictionary2 env, String cwd, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, Boolean usesTerminal, SafeHandle[] inheritedHandles, Holder& waitStateHolder, Boolean throwOnNoExec) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 369 at Microsoft.Win32.SafeHandles.SafeProcessHandle.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles, Holder& waitStateHolder) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 213 at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Unix.cs:line 363 at System.Diagnostics.Process.Start() in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1272 at System.Diagnostics.Process.Start(ProcessStartInfo startInfo) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1383 at Microsoft.Extensions.Internal.ProcessExtensions.RunProcessAndWaitForExit(String fileName, String arguments, TimeSpan timeout, String& stdout) in /_/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 93 at Microsoft.Extensions.Internal.ProcessExtensions.GetAllChildIdsUnix(Int32 parentId, ISet1 children, TimeSpan timeout) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 44
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process, TimeSpan timeout) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 33
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 17
at Microsoft.Extensions.Hosting.IntegrationTesting.ApplicationDeployer.ShutDownIfAnyHostProcess(Process hostProcess) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/ApplicationDeployer.cs:line 126
at Microsoft.Extensions.Hosting.IntegrationTesting.SelfHostDeployer.Dispose() in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/SelfHostDeployer.cs:line 179
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ExecuteShutdownTest(String testName, String shutdownMechanic) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 107
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestRun() in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 34
--- End of stack trace from previous location ---
❌ Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestWaitForShutdown [Console][Details][Artifacts][0.69% failure rate]
[ 🚧 Create issue against infrastructure team][ 📄 Create issue in this repo]
Failing Configuration
AzureLinux.3.0.ArmArch.Open
Exception Message
System.ComponentModel.Win32Exception : An error occurred trying to start process 'pgrep' with working directory '/root/helix/work/workitem/e'. No such file or directory
CallStack
at Microsoft.Win32.SafeHandles.SafeProcessHandle.ForkAndExecProcess(ProcessStartInfo startInfo, String resolvedFilename, String[] argv, IDictionary2 env, String cwd, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, Boolean usesTerminal, SafeHandle[] inheritedHandles, Holder& waitStateHolder, Boolean throwOnNoExec) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 369 at Microsoft.Win32.SafeHandles.SafeProcessHandle.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles, Holder& waitStateHolder) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 213 at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Unix.cs:line 363 at System.Diagnostics.Process.Start() in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1272 at System.Diagnostics.Process.Start(ProcessStartInfo startInfo) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1383 at Microsoft.Extensions.Internal.ProcessExtensions.RunProcessAndWaitForExit(String fileName, String arguments, TimeSpan timeout, String& stdout) in /_/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 93 at Microsoft.Extensions.Internal.ProcessExtensions.GetAllChildIdsUnix(Int32 parentId, ISet1 children, TimeSpan timeout) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 44
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process, TimeSpan timeout) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 33
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 17
at Microsoft.Extensions.Hosting.IntegrationTesting.ApplicationDeployer.ShutDownIfAnyHostProcess(Process hostProcess) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/ApplicationDeployer.cs:line 126
at Microsoft.Extensions.Hosting.IntegrationTesting.SelfHostDeployer.Dispose() in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/SelfHostDeployer.cs:line 179
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ExecuteShutdownTest(String testName, String shutdownMechanic) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 107
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestWaitForShutdown() in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 41
--- End of stack trace from previous location ---

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

…plicate in ShutdownTests
Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

CopilotAIand others added 4 commits June 15, 2026 15:53
Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>
…on-hosting' into copilot/enable-functional-tests-on-hosting
Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>
Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

rosebyteand others added 2 commits June 16, 2026 23:04
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:83

  • OutputReceived is raised from background threads, but output is being mutated via output += ... without synchronization. This can lead to lost/garbled output (and flaky assertions) when multiple lines arrive quickly. Consider using a thread-safe accumulator (e.g., a StringBuilder guarded by a lock, or enqueue lines into a concurrent collection and join at the end) rather than mutating a shared string from the event callback.
 var started = new TaskCompletionSource<int>(TaskCreationOptions.RunContinuationsAsynchronously);
var completed = new TaskCompletionSource<int>(TaskCreationOptions.RunContinuationsAsynchronously);
var output = string.Empty;
deployer.OutputReceived += (sender, args) =>
{
if (!string.IsNullOrEmpty(args.Data) && args.Data.StartsWith(StartedMessage, StringComparison.Ordinal))
{
output += args.Data.Substring(StartedMessage.Length) + '\n';
started.TrySetResult(0);
}

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Enable FunctionalTests/TestApp on Extensions Hosting

5 participants

@rosebyte@svick@mrek-msft
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Re-enable Microsoft.Extensions.Hosting shutdown functional coverage in CI and harden shutdown startup/Helix/NET481 compatibility - #128252

Merged
rosebyte merged 44 commits into
mainfrom
copilot/enable-functional-tests-on-hosting
Jun 24, 2026
Merged

Re-enable Microsoft.Extensions.Hosting shutdown functional coverage in CI and harden shutdown startup/Helix/NET481 compatibility#128252
rosebyte merged 44 commits into
mainfrom
copilot/enable-functional-tests-on-hosting

Conversation

CopilotAI commented May 15, 2026

Copy link
Copy Markdown
Contributor

This PR now includes the full follow-up hardening required by CI feedback after re-enabling Microsoft.Extensions.Hosting functional shutdown coverage.

Final scope of changes

  • Re-enabled Hosting FunctionalTests in CI (removed CI gating/active issue suppression).
  • Updated shutdown test deployment to run from deployed output (AppContext.BaseDirectory) instead of source-tree project discovery.
  • Removed the load-bearing PreservePublishedApplicationForDebugging dependency by using a borrowed published app path that does not delete test output on cleanup. PublishedApplication.Dispose() is now virtual; BorrowedPublishedApplication overrides it as a no-op so teardown never deletes the test output directory.
  • Replaced external kill process usage in shutdown signaling with libc signal calls for Helix/minimal-image compatibility.
  • Centralized the libckill P/Invoke in ProcessExtensions.SendSignal(int pid, int signal); removed the duplicate declaration from ShutdownTests so both signal paths share a single interop entry point. The underlying P/Invoke method is named Kill (PascalCase) with EntryPoint = "kill" to match C# naming conventions while preserving the correct libc symbol binding. SendSignal explicitly throws PlatformNotSupportedException on Windows so callers get a clear, actionable error rather than a DllNotFoundException from the libc P/Invoke.
  • Resolved Unix SIGINT/SIGTERM numbers at runtime via SystemNative_GetPlatformSignalNumber instead of hard-coding them, matching the runtime's platform-specific signal mapping approach. Signal number initialization is gated behind an OS check so Windows does not attempt the P/Invoke during type initialization.
  • Kept Unix cleanup best-effort and resilient:
    • graceful termination first (SIGTERM via libc)
    • hard-kill fallback (Process.Kill()) when graceful termination times out or fails
    • tolerance for missing process tools / constrained environments
  • Increased startup/shutdown wait windows to 180 seconds for stress/JIT-stress stability.
  • Increased the final shutdown exit grace period before escalation to Process.Kill() to 30 seconds, reducing false failures when the app has finished shutdown logging but needs a little longer to exit cleanly under load.
  • Fixed the shutdown test startup race by wiring output observation before DeployAsync() starts process output capture via SelfHostDeployer.OutputReceived (now internal), so the initial Started line cannot be missed.
  • Fixed net481 compatibility by using Marshal.GetLastWin32Error().
  • Fixed WaitForExitOrKill race: now uses the boolean result from WaitForExit(timeout) directly and wraps Process.Kill() in a catch (InvalidOperationException) to tolerate the window where the process exits between the wait and the kill call.

Validation

  • ./build.sh clr+libs -rc release succeeds.
  • ./build.sh clr+libs+host -rc release -lc release succeeds.
  • Microsoft.Extensions.Hosting.Functional.Tests builds successfully for net481.
  • dotnet build /t:test Microsoft.Extensions.Hosting.Functional.Tests.csproj -c Release -f net11.0 /p:TestFilter=Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests passes (2/2).
  • CI-feedback-driven shutdown/process-cleanup paths were updated to match Helix/runtime constraints and avoid prior failure modes.

CopilotAI requested review from Copilot and removed request for CopilotMay 15, 2026 12:32
CopilotAI linked an issue May 15, 2026 that may be closed by this pull request
@github-actionsgithub-actionsBot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label May 15, 2026
CopilotAI requested review from Copilot and removed request for CopilotMay 15, 2026 13:03
CopilotAI changed the title [WIP] Fix FunctionalTests/TestApp on Extensions HostingRe-enable Microsoft.Extensions.Hosting FunctionalTests/TestApp shutdown coverageMay 15, 2026
CopilotAI requested a review from rosebyteMay 15, 2026 13:07
@github-actionsgithub-actionsBot added area-Extensions-Hosting and removed needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners labels May 15, 2026
@rosebyte

Copy link
Copy Markdown
Member

@copilot , the shutdown tests are failing with this error on helix machines:

Comments
{ "HelixJobId": "bf5b8a6f-0bdd-40b3-9fbc-f0eec7148f40", "HelixWorkItemName": "Microsoft.Extensions.Hosting.Functional.Tests" }

Error message
System.IO.DirectoryNotFoundException : Could not find Microsoft.Extensions.Hosting.TestApp.csproj from base directory '/root/helix/work/workitem/e/'.

Stack trace
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.GetApplicationPath() in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 173
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ExecuteShutdownTest(String testName, String shutdownMechanic) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 53
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestRun() in /_/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 35
--- End of stack trace from previous location ---

CopilotAI requested review from Copilot and removed request for CopilotMay 18, 2026 11:55

CopilotAI commented May 18, 2026

Copy link
Copy Markdown
ContributorAuthor

@copilot , the shutdown tests are failing with this error on helix machines:

...

Fixed in b9cd702. I removed the source-tree lookup and switched the shutdown tests to run TestApp from the deployed test output (AppContext.BaseDirectory) using a custom ApplicationPublisher, so Helix no longer requires Microsoft.Extensions.Hosting.TestApp.csproj to exist in the workitem.

CopilotAI changed the title Re-enable Microsoft.Extensions.Hosting FunctionalTests/TestApp shutdown coverageRe-enable Microsoft.Extensions.Hosting FunctionalTests/TestApp shutdown coverage across local and Helix environmentsMay 18, 2026
@rosebyte

Copy link
Copy Markdown
Member

@copilot , this time it failed with this error:

Test Failures (700 tests failed)
🔹 [All failing tests from runtime]

⚠️ 698 failing tests not displayed for this pipeline due to space limitations

❌ Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestRun [Console][Details][Artifacts][0.70% failure rate]
[ 🚧 Create issue against infrastructure team][ 📄 Create issue in this repo]
Failing Configuration
AzureLinux.3.0.ArmArch.Open
Exception Message
System.ComponentModel.Win32Exception : An error occurred trying to start process 'pgrep' with working directory '/root/helix/work/workitem/e'. No such file or directory
CallStack
at Microsoft.Win32.SafeHandles.SafeProcessHandle.ForkAndExecProcess(ProcessStartInfo startInfo, String resolvedFilename, String[] argv, IDictionary2 env, String cwd, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, Boolean usesTerminal, SafeHandle[] inheritedHandles, Holder& waitStateHolder, Boolean throwOnNoExec) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 369 at Microsoft.Win32.SafeHandles.SafeProcessHandle.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles, Holder& waitStateHolder) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 213 at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Unix.cs:line 363 at System.Diagnostics.Process.Start() in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1272 at System.Diagnostics.Process.Start(ProcessStartInfo startInfo) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1383 at Microsoft.Extensions.Internal.ProcessExtensions.RunProcessAndWaitForExit(String fileName, String arguments, TimeSpan timeout, String& stdout) in /_/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 93 at Microsoft.Extensions.Internal.ProcessExtensions.GetAllChildIdsUnix(Int32 parentId, ISet1 children, TimeSpan timeout) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 44
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process, TimeSpan timeout) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 33
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 17
at Microsoft.Extensions.Hosting.IntegrationTesting.ApplicationDeployer.ShutDownIfAnyHostProcess(Process hostProcess) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/ApplicationDeployer.cs:line 126
at Microsoft.Extensions.Hosting.IntegrationTesting.SelfHostDeployer.Dispose() in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/SelfHostDeployer.cs:line 179
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ExecuteShutdownTest(String testName, String shutdownMechanic) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 107
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestRun() in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 34
--- End of stack trace from previous location ---
❌ Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestWaitForShutdown [Console][Details][Artifacts][0.69% failure rate]
[ 🚧 Create issue against infrastructure team][ 📄 Create issue in this repo]
Failing Configuration
AzureLinux.3.0.ArmArch.Open
Exception Message
System.ComponentModel.Win32Exception : An error occurred trying to start process 'pgrep' with working directory '/root/helix/work/workitem/e'. No such file or directory
CallStack
at Microsoft.Win32.SafeHandles.SafeProcessHandle.ForkAndExecProcess(ProcessStartInfo startInfo, String resolvedFilename, String[] argv, IDictionary2 env, String cwd, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, Boolean usesTerminal, SafeHandle[] inheritedHandles, Holder& waitStateHolder, Boolean throwOnNoExec) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 369 at Microsoft.Win32.SafeHandles.SafeProcessHandle.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles, Holder& waitStateHolder) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 213 at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Unix.cs:line 363 at System.Diagnostics.Process.Start() in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1272 at System.Diagnostics.Process.Start(ProcessStartInfo startInfo) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1383 at Microsoft.Extensions.Internal.ProcessExtensions.RunProcessAndWaitForExit(String fileName, String arguments, TimeSpan timeout, String& stdout) in /_/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 93 at Microsoft.Extensions.Internal.ProcessExtensions.GetAllChildIdsUnix(Int32 parentId, ISet1 children, TimeSpan timeout) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 44
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process, TimeSpan timeout) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 33
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 17
at Microsoft.Extensions.Hosting.IntegrationTesting.ApplicationDeployer.ShutDownIfAnyHostProcess(Process hostProcess) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/ApplicationDeployer.cs:line 126
at Microsoft.Extensions.Hosting.IntegrationTesting.SelfHostDeployer.Dispose() in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/SelfHostDeployer.cs:line 179
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ExecuteShutdownTest(String testName, String shutdownMechanic) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 107
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestWaitForShutdown() in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 41
--- End of stack trace from previous location ---

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

…plicate in ShutdownTests
Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

CopilotAIand others added 4 commits June 15, 2026 15:53
Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>
…on-hosting' into copilot/enable-functional-tests-on-hosting
Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>
Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

rosebyteand others added 2 commits June 16, 2026 23:04
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:83

  • OutputReceived is raised from background threads, but output is being mutated via output += ... without synchronization. This can lead to lost/garbled output (and flaky assertions) when multiple lines arrive quickly. Consider using a thread-safe accumulator (e.g., a StringBuilder guarded by a lock, or enqueue lines into a concurrent collection and join at the end) rather than mutating a shared string from the event callback.
 var started = new TaskCompletionSource<int>(TaskCreationOptions.RunContinuationsAsynchronously);
var completed = new TaskCompletionSource<int>(TaskCreationOptions.RunContinuationsAsynchronously);
var output = string.Empty;
deployer.OutputReceived += (sender, args) =>
{
if (!string.IsNullOrEmpty(args.Data) && args.Data.StartsWith(StartedMessage, StringComparison.Ordinal))
{
output += args.Data.Substring(StartedMessage.Length) + '\n';
started.TrySetResult(0);
}

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Enable FunctionalTests/TestApp on Extensions Hosting

5 participants

@rosebyte@svick@mrek-msft
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Re-enable Microsoft.Extensions.Hosting shutdown functional coverage in CI and harden shutdown startup/Helix/NET481 compatibility - #128252

Merged
rosebyte merged 44 commits into
mainfrom
copilot/enable-functional-tests-on-hosting
Jun 24, 2026
Merged

Re-enable Microsoft.Extensions.Hosting shutdown functional coverage in CI and harden shutdown startup/Helix/NET481 compatibility#128252
rosebyte merged 44 commits into
mainfrom
copilot/enable-functional-tests-on-hosting

Conversation

CopilotAI commented May 15, 2026

Copy link
Copy Markdown
Contributor

This PR now includes the full follow-up hardening required by CI feedback after re-enabling Microsoft.Extensions.Hosting functional shutdown coverage.

Final scope of changes

  • Re-enabled Hosting FunctionalTests in CI (removed CI gating/active issue suppression).
  • Updated shutdown test deployment to run from deployed output (AppContext.BaseDirectory) instead of source-tree project discovery.
  • Removed the load-bearing PreservePublishedApplicationForDebugging dependency by using a borrowed published app path that does not delete test output on cleanup. PublishedApplication.Dispose() is now virtual; BorrowedPublishedApplication overrides it as a no-op so teardown never deletes the test output directory.
  • Replaced external kill process usage in shutdown signaling with libc signal calls for Helix/minimal-image compatibility.
  • Centralized the libckill P/Invoke in ProcessExtensions.SendSignal(int pid, int signal); removed the duplicate declaration from ShutdownTests so both signal paths share a single interop entry point. The underlying P/Invoke method is named Kill (PascalCase) with EntryPoint = "kill" to match C# naming conventions while preserving the correct libc symbol binding. SendSignal explicitly throws PlatformNotSupportedException on Windows so callers get a clear, actionable error rather than a DllNotFoundException from the libc P/Invoke.
  • Resolved Unix SIGINT/SIGTERM numbers at runtime via SystemNative_GetPlatformSignalNumber instead of hard-coding them, matching the runtime's platform-specific signal mapping approach. Signal number initialization is gated behind an OS check so Windows does not attempt the P/Invoke during type initialization.
  • Kept Unix cleanup best-effort and resilient:
    • graceful termination first (SIGTERM via libc)
    • hard-kill fallback (Process.Kill()) when graceful termination times out or fails
    • tolerance for missing process tools / constrained environments
  • Increased startup/shutdown wait windows to 180 seconds for stress/JIT-stress stability.
  • Increased the final shutdown exit grace period before escalation to Process.Kill() to 30 seconds, reducing false failures when the app has finished shutdown logging but needs a little longer to exit cleanly under load.
  • Fixed the shutdown test startup race by wiring output observation before DeployAsync() starts process output capture via SelfHostDeployer.OutputReceived (now internal), so the initial Started line cannot be missed.
  • Fixed net481 compatibility by using Marshal.GetLastWin32Error().
  • Fixed WaitForExitOrKill race: now uses the boolean result from WaitForExit(timeout) directly and wraps Process.Kill() in a catch (InvalidOperationException) to tolerate the window where the process exits between the wait and the kill call.

Validation

  • ./build.sh clr+libs -rc release succeeds.
  • ./build.sh clr+libs+host -rc release -lc release succeeds.
  • Microsoft.Extensions.Hosting.Functional.Tests builds successfully for net481.
  • dotnet build /t:test Microsoft.Extensions.Hosting.Functional.Tests.csproj -c Release -f net11.0 /p:TestFilter=Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests passes (2/2).
  • CI-feedback-driven shutdown/process-cleanup paths were updated to match Helix/runtime constraints and avoid prior failure modes.

CopilotAI requested review from Copilot and removed request for CopilotMay 15, 2026 12:32
CopilotAI linked an issue May 15, 2026 that may be closed by this pull request
@github-actionsgithub-actionsBot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label May 15, 2026
CopilotAI requested review from Copilot and removed request for CopilotMay 15, 2026 13:03
CopilotAI changed the title [WIP] Fix FunctionalTests/TestApp on Extensions HostingRe-enable Microsoft.Extensions.Hosting FunctionalTests/TestApp shutdown coverageMay 15, 2026
CopilotAI requested a review from rosebyteMay 15, 2026 13:07
@github-actionsgithub-actionsBot added area-Extensions-Hosting and removed needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners labels May 15, 2026
@rosebyte

Copy link
Copy Markdown
Member

@copilot , the shutdown tests are failing with this error on helix machines:

Comments
{ "HelixJobId": "bf5b8a6f-0bdd-40b3-9fbc-f0eec7148f40", "HelixWorkItemName": "Microsoft.Extensions.Hosting.Functional.Tests" }

Error message
System.IO.DirectoryNotFoundException : Could not find Microsoft.Extensions.Hosting.TestApp.csproj from base directory '/root/helix/work/workitem/e/'.

Stack trace
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.GetApplicationPath() in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 173
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ExecuteShutdownTest(String testName, String shutdownMechanic) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 53
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestRun() in /_/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 35
--- End of stack trace from previous location ---

CopilotAI requested review from Copilot and removed request for CopilotMay 18, 2026 11:55

CopilotAI commented May 18, 2026

Copy link
Copy Markdown
ContributorAuthor

@copilot , the shutdown tests are failing with this error on helix machines:

...

Fixed in b9cd702. I removed the source-tree lookup and switched the shutdown tests to run TestApp from the deployed test output (AppContext.BaseDirectory) using a custom ApplicationPublisher, so Helix no longer requires Microsoft.Extensions.Hosting.TestApp.csproj to exist in the workitem.

CopilotAI changed the title Re-enable Microsoft.Extensions.Hosting FunctionalTests/TestApp shutdown coverageRe-enable Microsoft.Extensions.Hosting FunctionalTests/TestApp shutdown coverage across local and Helix environmentsMay 18, 2026
@rosebyte

Copy link
Copy Markdown
Member

@copilot , this time it failed with this error:

Test Failures (700 tests failed)
🔹 [All failing tests from runtime]

⚠️ 698 failing tests not displayed for this pipeline due to space limitations

❌ Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestRun [Console][Details][Artifacts][0.70% failure rate]
[ 🚧 Create issue against infrastructure team][ 📄 Create issue in this repo]
Failing Configuration
AzureLinux.3.0.ArmArch.Open
Exception Message
System.ComponentModel.Win32Exception : An error occurred trying to start process 'pgrep' with working directory '/root/helix/work/workitem/e'. No such file or directory
CallStack
at Microsoft.Win32.SafeHandles.SafeProcessHandle.ForkAndExecProcess(ProcessStartInfo startInfo, String resolvedFilename, String[] argv, IDictionary2 env, String cwd, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, Boolean usesTerminal, SafeHandle[] inheritedHandles, Holder& waitStateHolder, Boolean throwOnNoExec) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 369 at Microsoft.Win32.SafeHandles.SafeProcessHandle.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles, Holder& waitStateHolder) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 213 at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Unix.cs:line 363 at System.Diagnostics.Process.Start() in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1272 at System.Diagnostics.Process.Start(ProcessStartInfo startInfo) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1383 at Microsoft.Extensions.Internal.ProcessExtensions.RunProcessAndWaitForExit(String fileName, String arguments, TimeSpan timeout, String& stdout) in /_/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 93 at Microsoft.Extensions.Internal.ProcessExtensions.GetAllChildIdsUnix(Int32 parentId, ISet1 children, TimeSpan timeout) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 44
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process, TimeSpan timeout) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 33
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 17
at Microsoft.Extensions.Hosting.IntegrationTesting.ApplicationDeployer.ShutDownIfAnyHostProcess(Process hostProcess) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/ApplicationDeployer.cs:line 126
at Microsoft.Extensions.Hosting.IntegrationTesting.SelfHostDeployer.Dispose() in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/SelfHostDeployer.cs:line 179
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ExecuteShutdownTest(String testName, String shutdownMechanic) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 107
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestRun() in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 34
--- End of stack trace from previous location ---
❌ Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestWaitForShutdown [Console][Details][Artifacts][0.69% failure rate]
[ 🚧 Create issue against infrastructure team][ 📄 Create issue in this repo]
Failing Configuration
AzureLinux.3.0.ArmArch.Open
Exception Message
System.ComponentModel.Win32Exception : An error occurred trying to start process 'pgrep' with working directory '/root/helix/work/workitem/e'. No such file or directory
CallStack
at Microsoft.Win32.SafeHandles.SafeProcessHandle.ForkAndExecProcess(ProcessStartInfo startInfo, String resolvedFilename, String[] argv, IDictionary2 env, String cwd, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, Boolean usesTerminal, SafeHandle[] inheritedHandles, Holder& waitStateHolder, Boolean throwOnNoExec) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 369 at Microsoft.Win32.SafeHandles.SafeProcessHandle.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles, Holder& waitStateHolder) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 213 at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Unix.cs:line 363 at System.Diagnostics.Process.Start() in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1272 at System.Diagnostics.Process.Start(ProcessStartInfo startInfo) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1383 at Microsoft.Extensions.Internal.ProcessExtensions.RunProcessAndWaitForExit(String fileName, String arguments, TimeSpan timeout, String& stdout) in /_/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 93 at Microsoft.Extensions.Internal.ProcessExtensions.GetAllChildIdsUnix(Int32 parentId, ISet1 children, TimeSpan timeout) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 44
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process, TimeSpan timeout) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 33
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 17
at Microsoft.Extensions.Hosting.IntegrationTesting.ApplicationDeployer.ShutDownIfAnyHostProcess(Process hostProcess) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/ApplicationDeployer.cs:line 126
at Microsoft.Extensions.Hosting.IntegrationTesting.SelfHostDeployer.Dispose() in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/SelfHostDeployer.cs:line 179
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ExecuteShutdownTest(String testName, String shutdownMechanic) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 107
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestWaitForShutdown() in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 41
--- End of stack trace from previous location ---

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

…plicate in ShutdownTests
Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

CopilotAIand others added 4 commits June 15, 2026 15:53
Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>
…on-hosting' into copilot/enable-functional-tests-on-hosting
Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>
Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

rosebyteand others added 2 commits June 16, 2026 23:04
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:83

  • OutputReceived is raised from background threads, but output is being mutated via output += ... without synchronization. This can lead to lost/garbled output (and flaky assertions) when multiple lines arrive quickly. Consider using a thread-safe accumulator (e.g., a StringBuilder guarded by a lock, or enqueue lines into a concurrent collection and join at the end) rather than mutating a shared string from the event callback.
 var started = new TaskCompletionSource<int>(TaskCreationOptions.RunContinuationsAsynchronously);
var completed = new TaskCompletionSource<int>(TaskCreationOptions.RunContinuationsAsynchronously);
var output = string.Empty;
deployer.OutputReceived += (sender, args) =>
{
if (!string.IsNullOrEmpty(args.Data) && args.Data.StartsWith(StartedMessage, StringComparison.Ordinal))
{
output += args.Data.Substring(StartedMessage.Length) + '\n';
started.TrySetResult(0);
}

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Enable FunctionalTests/TestApp on Extensions Hosting

5 participants

@rosebyte@svick@mrek-msft
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

Re-enable Microsoft.Extensions.Hosting shutdown functional coverage in CI and harden shutdown startup/Helix/NET481 compatibility - #128252

Merged
rosebyte merged 44 commits into
mainfrom
copilot/enable-functional-tests-on-hosting
Jun 24, 2026
Merged

Re-enable Microsoft.Extensions.Hosting shutdown functional coverage in CI and harden shutdown startup/Helix/NET481 compatibility#128252
rosebyte merged 44 commits into
mainfrom
copilot/enable-functional-tests-on-hosting

Conversation

CopilotAI commented May 15, 2026

Copy link
Copy Markdown
Contributor

This PR now includes the full follow-up hardening required by CI feedback after re-enabling Microsoft.Extensions.Hosting functional shutdown coverage.

Final scope of changes

  • Re-enabled Hosting FunctionalTests in CI (removed CI gating/active issue suppression).
  • Updated shutdown test deployment to run from deployed output (AppContext.BaseDirectory) instead of source-tree project discovery.
  • Removed the load-bearing PreservePublishedApplicationForDebugging dependency by using a borrowed published app path that does not delete test output on cleanup. PublishedApplication.Dispose() is now virtual; BorrowedPublishedApplication overrides it as a no-op so teardown never deletes the test output directory.
  • Replaced external kill process usage in shutdown signaling with libc signal calls for Helix/minimal-image compatibility.
  • Centralized the libckill P/Invoke in ProcessExtensions.SendSignal(int pid, int signal); removed the duplicate declaration from ShutdownTests so both signal paths share a single interop entry point. The underlying P/Invoke method is named Kill (PascalCase) with EntryPoint = "kill" to match C# naming conventions while preserving the correct libc symbol binding. SendSignal explicitly throws PlatformNotSupportedException on Windows so callers get a clear, actionable error rather than a DllNotFoundException from the libc P/Invoke.
  • Resolved Unix SIGINT/SIGTERM numbers at runtime via SystemNative_GetPlatformSignalNumber instead of hard-coding them, matching the runtime's platform-specific signal mapping approach. Signal number initialization is gated behind an OS check so Windows does not attempt the P/Invoke during type initialization.
  • Kept Unix cleanup best-effort and resilient:
    • graceful termination first (SIGTERM via libc)
    • hard-kill fallback (Process.Kill()) when graceful termination times out or fails
    • tolerance for missing process tools / constrained environments
  • Increased startup/shutdown wait windows to 180 seconds for stress/JIT-stress stability.
  • Increased the final shutdown exit grace period before escalation to Process.Kill() to 30 seconds, reducing false failures when the app has finished shutdown logging but needs a little longer to exit cleanly under load.
  • Fixed the shutdown test startup race by wiring output observation before DeployAsync() starts process output capture via SelfHostDeployer.OutputReceived (now internal), so the initial Started line cannot be missed.
  • Fixed net481 compatibility by using Marshal.GetLastWin32Error().
  • Fixed WaitForExitOrKill race: now uses the boolean result from WaitForExit(timeout) directly and wraps Process.Kill() in a catch (InvalidOperationException) to tolerate the window where the process exits between the wait and the kill call.

Validation

  • ./build.sh clr+libs -rc release succeeds.
  • ./build.sh clr+libs+host -rc release -lc release succeeds.
  • Microsoft.Extensions.Hosting.Functional.Tests builds successfully for net481.
  • dotnet build /t:test Microsoft.Extensions.Hosting.Functional.Tests.csproj -c Release -f net11.0 /p:TestFilter=Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests passes (2/2).
  • CI-feedback-driven shutdown/process-cleanup paths were updated to match Helix/runtime constraints and avoid prior failure modes.

CopilotAI requested review from Copilot and removed request for CopilotMay 15, 2026 12:32
CopilotAI linked an issue May 15, 2026 that may be closed by this pull request
@github-actionsgithub-actionsBot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label May 15, 2026
CopilotAI requested review from Copilot and removed request for CopilotMay 15, 2026 13:03
CopilotAI changed the title [WIP] Fix FunctionalTests/TestApp on Extensions HostingRe-enable Microsoft.Extensions.Hosting FunctionalTests/TestApp shutdown coverageMay 15, 2026
CopilotAI requested a review from rosebyteMay 15, 2026 13:07
@github-actionsgithub-actionsBot added area-Extensions-Hosting and removed needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners labels May 15, 2026
@rosebyte

Copy link
Copy Markdown
Member

@copilot , the shutdown tests are failing with this error on helix machines:

Comments
{ "HelixJobId": "bf5b8a6f-0bdd-40b3-9fbc-f0eec7148f40", "HelixWorkItemName": "Microsoft.Extensions.Hosting.Functional.Tests" }

Error message
System.IO.DirectoryNotFoundException : Could not find Microsoft.Extensions.Hosting.TestApp.csproj from base directory '/root/helix/work/workitem/e/'.

Stack trace
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.GetApplicationPath() in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 173
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ExecuteShutdownTest(String testName, String shutdownMechanic) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 53
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestRun() in /_/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 35
--- End of stack trace from previous location ---

CopilotAI requested review from Copilot and removed request for CopilotMay 18, 2026 11:55

CopilotAI commented May 18, 2026

Copy link
Copy Markdown
ContributorAuthor

@copilot , the shutdown tests are failing with this error on helix machines:

...

Fixed in b9cd702. I removed the source-tree lookup and switched the shutdown tests to run TestApp from the deployed test output (AppContext.BaseDirectory) using a custom ApplicationPublisher, so Helix no longer requires Microsoft.Extensions.Hosting.TestApp.csproj to exist in the workitem.

CopilotAI changed the title Re-enable Microsoft.Extensions.Hosting FunctionalTests/TestApp shutdown coverageRe-enable Microsoft.Extensions.Hosting FunctionalTests/TestApp shutdown coverage across local and Helix environmentsMay 18, 2026
@rosebyte

Copy link
Copy Markdown
Member

@copilot , this time it failed with this error:

Test Failures (700 tests failed)
🔹 [All failing tests from runtime]

⚠️ 698 failing tests not displayed for this pipeline due to space limitations

❌ Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestRun [Console][Details][Artifacts][0.70% failure rate]
[ 🚧 Create issue against infrastructure team][ 📄 Create issue in this repo]
Failing Configuration
AzureLinux.3.0.ArmArch.Open
Exception Message
System.ComponentModel.Win32Exception : An error occurred trying to start process 'pgrep' with working directory '/root/helix/work/workitem/e'. No such file or directory
CallStack
at Microsoft.Win32.SafeHandles.SafeProcessHandle.ForkAndExecProcess(ProcessStartInfo startInfo, String resolvedFilename, String[] argv, IDictionary2 env, String cwd, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, Boolean usesTerminal, SafeHandle[] inheritedHandles, Holder& waitStateHolder, Boolean throwOnNoExec) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 369 at Microsoft.Win32.SafeHandles.SafeProcessHandle.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles, Holder& waitStateHolder) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 213 at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Unix.cs:line 363 at System.Diagnostics.Process.Start() in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1272 at System.Diagnostics.Process.Start(ProcessStartInfo startInfo) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1383 at Microsoft.Extensions.Internal.ProcessExtensions.RunProcessAndWaitForExit(String fileName, String arguments, TimeSpan timeout, String& stdout) in /_/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 93 at Microsoft.Extensions.Internal.ProcessExtensions.GetAllChildIdsUnix(Int32 parentId, ISet1 children, TimeSpan timeout) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 44
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process, TimeSpan timeout) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 33
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 17
at Microsoft.Extensions.Hosting.IntegrationTesting.ApplicationDeployer.ShutDownIfAnyHostProcess(Process hostProcess) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/ApplicationDeployer.cs:line 126
at Microsoft.Extensions.Hosting.IntegrationTesting.SelfHostDeployer.Dispose() in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/SelfHostDeployer.cs:line 179
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ExecuteShutdownTest(String testName, String shutdownMechanic) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 107
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestRun() in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 34
--- End of stack trace from previous location ---
❌ Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestWaitForShutdown [Console][Details][Artifacts][0.69% failure rate]
[ 🚧 Create issue against infrastructure team][ 📄 Create issue in this repo]
Failing Configuration
AzureLinux.3.0.ArmArch.Open
Exception Message
System.ComponentModel.Win32Exception : An error occurred trying to start process 'pgrep' with working directory '/root/helix/work/workitem/e'. No such file or directory
CallStack
at Microsoft.Win32.SafeHandles.SafeProcessHandle.ForkAndExecProcess(ProcessStartInfo startInfo, String resolvedFilename, String[] argv, IDictionary2 env, String cwd, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, Boolean usesTerminal, SafeHandle[] inheritedHandles, Holder& waitStateHolder, Boolean throwOnNoExec) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 369 at Microsoft.Win32.SafeHandles.SafeProcessHandle.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles, Holder& waitStateHolder) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 213 at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Unix.cs:line 363 at System.Diagnostics.Process.Start() in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1272 at System.Diagnostics.Process.Start(ProcessStartInfo startInfo) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1383 at Microsoft.Extensions.Internal.ProcessExtensions.RunProcessAndWaitForExit(String fileName, String arguments, TimeSpan timeout, String& stdout) in /_/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 93 at Microsoft.Extensions.Internal.ProcessExtensions.GetAllChildIdsUnix(Int32 parentId, ISet1 children, TimeSpan timeout) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 44
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process, TimeSpan timeout) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 33
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 17
at Microsoft.Extensions.Hosting.IntegrationTesting.ApplicationDeployer.ShutDownIfAnyHostProcess(Process hostProcess) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/ApplicationDeployer.cs:line 126
at Microsoft.Extensions.Hosting.IntegrationTesting.SelfHostDeployer.Dispose() in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/SelfHostDeployer.cs:line 179
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ExecuteShutdownTest(String testName, String shutdownMechanic) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 107
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestWaitForShutdown() in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 41
--- End of stack trace from previous location ---

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

…plicate in ShutdownTests
Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

CopilotAIand others added 4 commits June 15, 2026 15:53
Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>
…on-hosting' into copilot/enable-functional-tests-on-hosting
Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>
Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

rosebyteand others added 2 commits June 16, 2026 23:04
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:83

  • OutputReceived is raised from background threads, but output is being mutated via output += ... without synchronization. This can lead to lost/garbled output (and flaky assertions) when multiple lines arrive quickly. Consider using a thread-safe accumulator (e.g., a StringBuilder guarded by a lock, or enqueue lines into a concurrent collection and join at the end) rather than mutating a shared string from the event callback.
 var started = new TaskCompletionSource<int>(TaskCreationOptions.RunContinuationsAsynchronously);
var completed = new TaskCompletionSource<int>(TaskCreationOptions.RunContinuationsAsynchronously);
var output = string.Empty;
deployer.OutputReceived += (sender, args) =>
{
if (!string.IsNullOrEmpty(args.Data) && args.Data.StartsWith(StartedMessage, StringComparison.Ordinal))
{
output += args.Data.Substring(StartedMessage.Length) + '\n';
started.TrySetResult(0);
}

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Enable FunctionalTests/TestApp on Extensions Hosting

5 participants

@rosebyte@svick@mrek-msft
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Re-enable Microsoft.Extensions.Hosting shutdown functional coverage in CI and harden shutdown startup/Helix/NET481 compatibility - #128252

Merged
rosebyte merged 44 commits into
mainfrom
copilot/enable-functional-tests-on-hosting
Jun 24, 2026
Merged

Re-enable Microsoft.Extensions.Hosting shutdown functional coverage in CI and harden shutdown startup/Helix/NET481 compatibility#128252
rosebyte merged 44 commits into
mainfrom
copilot/enable-functional-tests-on-hosting

Conversation

CopilotAI commented May 15, 2026

Copy link
Copy Markdown
Contributor

This PR now includes the full follow-up hardening required by CI feedback after re-enabling Microsoft.Extensions.Hosting functional shutdown coverage.

Final scope of changes

  • Re-enabled Hosting FunctionalTests in CI (removed CI gating/active issue suppression).
  • Updated shutdown test deployment to run from deployed output (AppContext.BaseDirectory) instead of source-tree project discovery.
  • Removed the load-bearing PreservePublishedApplicationForDebugging dependency by using a borrowed published app path that does not delete test output on cleanup. PublishedApplication.Dispose() is now virtual; BorrowedPublishedApplication overrides it as a no-op so teardown never deletes the test output directory.
  • Replaced external kill process usage in shutdown signaling with libc signal calls for Helix/minimal-image compatibility.
  • Centralized the libckill P/Invoke in ProcessExtensions.SendSignal(int pid, int signal); removed the duplicate declaration from ShutdownTests so both signal paths share a single interop entry point. The underlying P/Invoke method is named Kill (PascalCase) with EntryPoint = "kill" to match C# naming conventions while preserving the correct libc symbol binding. SendSignal explicitly throws PlatformNotSupportedException on Windows so callers get a clear, actionable error rather than a DllNotFoundException from the libc P/Invoke.
  • Resolved Unix SIGINT/SIGTERM numbers at runtime via SystemNative_GetPlatformSignalNumber instead of hard-coding them, matching the runtime's platform-specific signal mapping approach. Signal number initialization is gated behind an OS check so Windows does not attempt the P/Invoke during type initialization.
  • Kept Unix cleanup best-effort and resilient:
    • graceful termination first (SIGTERM via libc)
    • hard-kill fallback (Process.Kill()) when graceful termination times out or fails
    • tolerance for missing process tools / constrained environments
  • Increased startup/shutdown wait windows to 180 seconds for stress/JIT-stress stability.
  • Increased the final shutdown exit grace period before escalation to Process.Kill() to 30 seconds, reducing false failures when the app has finished shutdown logging but needs a little longer to exit cleanly under load.
  • Fixed the shutdown test startup race by wiring output observation before DeployAsync() starts process output capture via SelfHostDeployer.OutputReceived (now internal), so the initial Started line cannot be missed.
  • Fixed net481 compatibility by using Marshal.GetLastWin32Error().
  • Fixed WaitForExitOrKill race: now uses the boolean result from WaitForExit(timeout) directly and wraps Process.Kill() in a catch (InvalidOperationException) to tolerate the window where the process exits between the wait and the kill call.

Validation

  • ./build.sh clr+libs -rc release succeeds.
  • ./build.sh clr+libs+host -rc release -lc release succeeds.
  • Microsoft.Extensions.Hosting.Functional.Tests builds successfully for net481.
  • dotnet build /t:test Microsoft.Extensions.Hosting.Functional.Tests.csproj -c Release -f net11.0 /p:TestFilter=Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests passes (2/2).
  • CI-feedback-driven shutdown/process-cleanup paths were updated to match Helix/runtime constraints and avoid prior failure modes.

CopilotAI requested review from Copilot and removed request for CopilotMay 15, 2026 12:32
CopilotAI linked an issue May 15, 2026 that may be closed by this pull request
@github-actionsgithub-actionsBot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label May 15, 2026
CopilotAI requested review from Copilot and removed request for CopilotMay 15, 2026 13:03
CopilotAI changed the title [WIP] Fix FunctionalTests/TestApp on Extensions HostingRe-enable Microsoft.Extensions.Hosting FunctionalTests/TestApp shutdown coverageMay 15, 2026
CopilotAI requested a review from rosebyteMay 15, 2026 13:07
@github-actionsgithub-actionsBot added area-Extensions-Hosting and removed needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners labels May 15, 2026
@rosebyte

Copy link
Copy Markdown
Member

@copilot , the shutdown tests are failing with this error on helix machines:

Comments
{ "HelixJobId": "bf5b8a6f-0bdd-40b3-9fbc-f0eec7148f40", "HelixWorkItemName": "Microsoft.Extensions.Hosting.Functional.Tests" }

Error message
System.IO.DirectoryNotFoundException : Could not find Microsoft.Extensions.Hosting.TestApp.csproj from base directory '/root/helix/work/workitem/e/'.

Stack trace
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.GetApplicationPath() in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 173
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ExecuteShutdownTest(String testName, String shutdownMechanic) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 53
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestRun() in /_/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 35
--- End of stack trace from previous location ---

CopilotAI requested review from Copilot and removed request for CopilotMay 18, 2026 11:55

CopilotAI commented May 18, 2026

Copy link
Copy Markdown
ContributorAuthor

@copilot , the shutdown tests are failing with this error on helix machines:

...

Fixed in b9cd702. I removed the source-tree lookup and switched the shutdown tests to run TestApp from the deployed test output (AppContext.BaseDirectory) using a custom ApplicationPublisher, so Helix no longer requires Microsoft.Extensions.Hosting.TestApp.csproj to exist in the workitem.

CopilotAI changed the title Re-enable Microsoft.Extensions.Hosting FunctionalTests/TestApp shutdown coverageRe-enable Microsoft.Extensions.Hosting FunctionalTests/TestApp shutdown coverage across local and Helix environmentsMay 18, 2026
@rosebyte

Copy link
Copy Markdown
Member

@copilot , this time it failed with this error:

Test Failures (700 tests failed)
🔹 [All failing tests from runtime]

⚠️ 698 failing tests not displayed for this pipeline due to space limitations

❌ Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestRun [Console][Details][Artifacts][0.70% failure rate]
[ 🚧 Create issue against infrastructure team][ 📄 Create issue in this repo]
Failing Configuration
AzureLinux.3.0.ArmArch.Open
Exception Message
System.ComponentModel.Win32Exception : An error occurred trying to start process 'pgrep' with working directory '/root/helix/work/workitem/e'. No such file or directory
CallStack
at Microsoft.Win32.SafeHandles.SafeProcessHandle.ForkAndExecProcess(ProcessStartInfo startInfo, String resolvedFilename, String[] argv, IDictionary2 env, String cwd, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, Boolean usesTerminal, SafeHandle[] inheritedHandles, Holder& waitStateHolder, Boolean throwOnNoExec) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 369 at Microsoft.Win32.SafeHandles.SafeProcessHandle.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles, Holder& waitStateHolder) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 213 at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Unix.cs:line 363 at System.Diagnostics.Process.Start() in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1272 at System.Diagnostics.Process.Start(ProcessStartInfo startInfo) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1383 at Microsoft.Extensions.Internal.ProcessExtensions.RunProcessAndWaitForExit(String fileName, String arguments, TimeSpan timeout, String& stdout) in /_/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 93 at Microsoft.Extensions.Internal.ProcessExtensions.GetAllChildIdsUnix(Int32 parentId, ISet1 children, TimeSpan timeout) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 44
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process, TimeSpan timeout) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 33
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 17
at Microsoft.Extensions.Hosting.IntegrationTesting.ApplicationDeployer.ShutDownIfAnyHostProcess(Process hostProcess) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/ApplicationDeployer.cs:line 126
at Microsoft.Extensions.Hosting.IntegrationTesting.SelfHostDeployer.Dispose() in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/SelfHostDeployer.cs:line 179
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ExecuteShutdownTest(String testName, String shutdownMechanic) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 107
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestRun() in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 34
--- End of stack trace from previous location ---
❌ Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestWaitForShutdown [Console][Details][Artifacts][0.69% failure rate]
[ 🚧 Create issue against infrastructure team][ 📄 Create issue in this repo]
Failing Configuration
AzureLinux.3.0.ArmArch.Open
Exception Message
System.ComponentModel.Win32Exception : An error occurred trying to start process 'pgrep' with working directory '/root/helix/work/workitem/e'. No such file or directory
CallStack
at Microsoft.Win32.SafeHandles.SafeProcessHandle.ForkAndExecProcess(ProcessStartInfo startInfo, String resolvedFilename, String[] argv, IDictionary2 env, String cwd, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, Boolean usesTerminal, SafeHandle[] inheritedHandles, Holder& waitStateHolder, Boolean throwOnNoExec) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 369 at Microsoft.Win32.SafeHandles.SafeProcessHandle.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles, Holder& waitStateHolder) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 213 at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Unix.cs:line 363 at System.Diagnostics.Process.Start() in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1272 at System.Diagnostics.Process.Start(ProcessStartInfo startInfo) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1383 at Microsoft.Extensions.Internal.ProcessExtensions.RunProcessAndWaitForExit(String fileName, String arguments, TimeSpan timeout, String& stdout) in /_/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 93 at Microsoft.Extensions.Internal.ProcessExtensions.GetAllChildIdsUnix(Int32 parentId, ISet1 children, TimeSpan timeout) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 44
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process, TimeSpan timeout) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 33
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 17
at Microsoft.Extensions.Hosting.IntegrationTesting.ApplicationDeployer.ShutDownIfAnyHostProcess(Process hostProcess) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/ApplicationDeployer.cs:line 126
at Microsoft.Extensions.Hosting.IntegrationTesting.SelfHostDeployer.Dispose() in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/SelfHostDeployer.cs:line 179
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ExecuteShutdownTest(String testName, String shutdownMechanic) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 107
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestWaitForShutdown() in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 41
--- End of stack trace from previous location ---

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

…plicate in ShutdownTests
Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

CopilotAIand others added 4 commits June 15, 2026 15:53
Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>
…on-hosting' into copilot/enable-functional-tests-on-hosting
Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>
Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

rosebyteand others added 2 commits June 16, 2026 23:04
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:83

  • OutputReceived is raised from background threads, but output is being mutated via output += ... without synchronization. This can lead to lost/garbled output (and flaky assertions) when multiple lines arrive quickly. Consider using a thread-safe accumulator (e.g., a StringBuilder guarded by a lock, or enqueue lines into a concurrent collection and join at the end) rather than mutating a shared string from the event callback.
 var started = new TaskCompletionSource<int>(TaskCreationOptions.RunContinuationsAsynchronously);
var completed = new TaskCompletionSource<int>(TaskCreationOptions.RunContinuationsAsynchronously);
var output = string.Empty;
deployer.OutputReceived += (sender, args) =>
{
if (!string.IsNullOrEmpty(args.Data) && args.Data.StartsWith(StartedMessage, StringComparison.Ordinal))
{
output += args.Data.Substring(StartedMessage.Length) + '\n';
started.TrySetResult(0);
}

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Enable FunctionalTests/TestApp on Extensions Hosting

5 participants

@rosebyte@svick@mrek-msft
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Re-enable Microsoft.Extensions.Hosting shutdown functional coverage in CI and harden shutdown startup/Helix/NET481 compatibility - #128252

Merged
rosebyte merged 44 commits into
mainfrom
copilot/enable-functional-tests-on-hosting
Jun 24, 2026
Merged

Re-enable Microsoft.Extensions.Hosting shutdown functional coverage in CI and harden shutdown startup/Helix/NET481 compatibility#128252
rosebyte merged 44 commits into
mainfrom
copilot/enable-functional-tests-on-hosting

Conversation

CopilotAI commented May 15, 2026

Copy link
Copy Markdown
Contributor

This PR now includes the full follow-up hardening required by CI feedback after re-enabling Microsoft.Extensions.Hosting functional shutdown coverage.

Final scope of changes

  • Re-enabled Hosting FunctionalTests in CI (removed CI gating/active issue suppression).
  • Updated shutdown test deployment to run from deployed output (AppContext.BaseDirectory) instead of source-tree project discovery.
  • Removed the load-bearing PreservePublishedApplicationForDebugging dependency by using a borrowed published app path that does not delete test output on cleanup. PublishedApplication.Dispose() is now virtual; BorrowedPublishedApplication overrides it as a no-op so teardown never deletes the test output directory.
  • Replaced external kill process usage in shutdown signaling with libc signal calls for Helix/minimal-image compatibility.
  • Centralized the libckill P/Invoke in ProcessExtensions.SendSignal(int pid, int signal); removed the duplicate declaration from ShutdownTests so both signal paths share a single interop entry point. The underlying P/Invoke method is named Kill (PascalCase) with EntryPoint = "kill" to match C# naming conventions while preserving the correct libc symbol binding. SendSignal explicitly throws PlatformNotSupportedException on Windows so callers get a clear, actionable error rather than a DllNotFoundException from the libc P/Invoke.
  • Resolved Unix SIGINT/SIGTERM numbers at runtime via SystemNative_GetPlatformSignalNumber instead of hard-coding them, matching the runtime's platform-specific signal mapping approach. Signal number initialization is gated behind an OS check so Windows does not attempt the P/Invoke during type initialization.
  • Kept Unix cleanup best-effort and resilient:
    • graceful termination first (SIGTERM via libc)
    • hard-kill fallback (Process.Kill()) when graceful termination times out or fails
    • tolerance for missing process tools / constrained environments
  • Increased startup/shutdown wait windows to 180 seconds for stress/JIT-stress stability.
  • Increased the final shutdown exit grace period before escalation to Process.Kill() to 30 seconds, reducing false failures when the app has finished shutdown logging but needs a little longer to exit cleanly under load.
  • Fixed the shutdown test startup race by wiring output observation before DeployAsync() starts process output capture via SelfHostDeployer.OutputReceived (now internal), so the initial Started line cannot be missed.
  • Fixed net481 compatibility by using Marshal.GetLastWin32Error().
  • Fixed WaitForExitOrKill race: now uses the boolean result from WaitForExit(timeout) directly and wraps Process.Kill() in a catch (InvalidOperationException) to tolerate the window where the process exits between the wait and the kill call.

Validation

  • ./build.sh clr+libs -rc release succeeds.
  • ./build.sh clr+libs+host -rc release -lc release succeeds.
  • Microsoft.Extensions.Hosting.Functional.Tests builds successfully for net481.
  • dotnet build /t:test Microsoft.Extensions.Hosting.Functional.Tests.csproj -c Release -f net11.0 /p:TestFilter=Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests passes (2/2).
  • CI-feedback-driven shutdown/process-cleanup paths were updated to match Helix/runtime constraints and avoid prior failure modes.

CopilotAI requested review from Copilot and removed request for CopilotMay 15, 2026 12:32
CopilotAI linked an issue May 15, 2026 that may be closed by this pull request
@github-actionsgithub-actionsBot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label May 15, 2026
CopilotAI requested review from Copilot and removed request for CopilotMay 15, 2026 13:03
CopilotAI changed the title [WIP] Fix FunctionalTests/TestApp on Extensions HostingRe-enable Microsoft.Extensions.Hosting FunctionalTests/TestApp shutdown coverageMay 15, 2026
CopilotAI requested a review from rosebyteMay 15, 2026 13:07
@github-actionsgithub-actionsBot added area-Extensions-Hosting and removed needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners labels May 15, 2026
@rosebyte

Copy link
Copy Markdown
Member

@copilot , the shutdown tests are failing with this error on helix machines:

Comments
{ "HelixJobId": "bf5b8a6f-0bdd-40b3-9fbc-f0eec7148f40", "HelixWorkItemName": "Microsoft.Extensions.Hosting.Functional.Tests" }

Error message
System.IO.DirectoryNotFoundException : Could not find Microsoft.Extensions.Hosting.TestApp.csproj from base directory '/root/helix/work/workitem/e/'.

Stack trace
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.GetApplicationPath() in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 173
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ExecuteShutdownTest(String testName, String shutdownMechanic) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 53
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestRun() in /_/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 35
--- End of stack trace from previous location ---

CopilotAI requested review from Copilot and removed request for CopilotMay 18, 2026 11:55

CopilotAI commented May 18, 2026

Copy link
Copy Markdown
ContributorAuthor

@copilot , the shutdown tests are failing with this error on helix machines:

...

Fixed in b9cd702. I removed the source-tree lookup and switched the shutdown tests to run TestApp from the deployed test output (AppContext.BaseDirectory) using a custom ApplicationPublisher, so Helix no longer requires Microsoft.Extensions.Hosting.TestApp.csproj to exist in the workitem.

CopilotAI changed the title Re-enable Microsoft.Extensions.Hosting FunctionalTests/TestApp shutdown coverageRe-enable Microsoft.Extensions.Hosting FunctionalTests/TestApp shutdown coverage across local and Helix environmentsMay 18, 2026
@rosebyte

Copy link
Copy Markdown
Member

@copilot , this time it failed with this error:

Test Failures (700 tests failed)
🔹 [All failing tests from runtime]

⚠️ 698 failing tests not displayed for this pipeline due to space limitations

❌ Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestRun [Console][Details][Artifacts][0.70% failure rate]
[ 🚧 Create issue against infrastructure team][ 📄 Create issue in this repo]
Failing Configuration
AzureLinux.3.0.ArmArch.Open
Exception Message
System.ComponentModel.Win32Exception : An error occurred trying to start process 'pgrep' with working directory '/root/helix/work/workitem/e'. No such file or directory
CallStack
at Microsoft.Win32.SafeHandles.SafeProcessHandle.ForkAndExecProcess(ProcessStartInfo startInfo, String resolvedFilename, String[] argv, IDictionary2 env, String cwd, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, Boolean usesTerminal, SafeHandle[] inheritedHandles, Holder& waitStateHolder, Boolean throwOnNoExec) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 369 at Microsoft.Win32.SafeHandles.SafeProcessHandle.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles, Holder& waitStateHolder) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 213 at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Unix.cs:line 363 at System.Diagnostics.Process.Start() in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1272 at System.Diagnostics.Process.Start(ProcessStartInfo startInfo) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1383 at Microsoft.Extensions.Internal.ProcessExtensions.RunProcessAndWaitForExit(String fileName, String arguments, TimeSpan timeout, String& stdout) in /_/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 93 at Microsoft.Extensions.Internal.ProcessExtensions.GetAllChildIdsUnix(Int32 parentId, ISet1 children, TimeSpan timeout) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 44
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process, TimeSpan timeout) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 33
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 17
at Microsoft.Extensions.Hosting.IntegrationTesting.ApplicationDeployer.ShutDownIfAnyHostProcess(Process hostProcess) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/ApplicationDeployer.cs:line 126
at Microsoft.Extensions.Hosting.IntegrationTesting.SelfHostDeployer.Dispose() in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/SelfHostDeployer.cs:line 179
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ExecuteShutdownTest(String testName, String shutdownMechanic) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 107
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestRun() in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 34
--- End of stack trace from previous location ---
❌ Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestWaitForShutdown [Console][Details][Artifacts][0.69% failure rate]
[ 🚧 Create issue against infrastructure team][ 📄 Create issue in this repo]
Failing Configuration
AzureLinux.3.0.ArmArch.Open
Exception Message
System.ComponentModel.Win32Exception : An error occurred trying to start process 'pgrep' with working directory '/root/helix/work/workitem/e'. No such file or directory
CallStack
at Microsoft.Win32.SafeHandles.SafeProcessHandle.ForkAndExecProcess(ProcessStartInfo startInfo, String resolvedFilename, String[] argv, IDictionary2 env, String cwd, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, Boolean usesTerminal, SafeHandle[] inheritedHandles, Holder& waitStateHolder, Boolean throwOnNoExec) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 369 at Microsoft.Win32.SafeHandles.SafeProcessHandle.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles, Holder& waitStateHolder) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 213 at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Unix.cs:line 363 at System.Diagnostics.Process.Start() in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1272 at System.Diagnostics.Process.Start(ProcessStartInfo startInfo) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1383 at Microsoft.Extensions.Internal.ProcessExtensions.RunProcessAndWaitForExit(String fileName, String arguments, TimeSpan timeout, String& stdout) in /_/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 93 at Microsoft.Extensions.Internal.ProcessExtensions.GetAllChildIdsUnix(Int32 parentId, ISet1 children, TimeSpan timeout) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 44
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process, TimeSpan timeout) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 33
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 17
at Microsoft.Extensions.Hosting.IntegrationTesting.ApplicationDeployer.ShutDownIfAnyHostProcess(Process hostProcess) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/ApplicationDeployer.cs:line 126
at Microsoft.Extensions.Hosting.IntegrationTesting.SelfHostDeployer.Dispose() in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/SelfHostDeployer.cs:line 179
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ExecuteShutdownTest(String testName, String shutdownMechanic) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 107
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestWaitForShutdown() in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 41
--- End of stack trace from previous location ---

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

…plicate in ShutdownTests
Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

CopilotAIand others added 4 commits June 15, 2026 15:53
Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>
…on-hosting' into copilot/enable-functional-tests-on-hosting
Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>
Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

rosebyteand others added 2 commits June 16, 2026 23:04
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:83

  • OutputReceived is raised from background threads, but output is being mutated via output += ... without synchronization. This can lead to lost/garbled output (and flaky assertions) when multiple lines arrive quickly. Consider using a thread-safe accumulator (e.g., a StringBuilder guarded by a lock, or enqueue lines into a concurrent collection and join at the end) rather than mutating a shared string from the event callback.
 var started = new TaskCompletionSource<int>(TaskCreationOptions.RunContinuationsAsynchronously);
var completed = new TaskCompletionSource<int>(TaskCreationOptions.RunContinuationsAsynchronously);
var output = string.Empty;
deployer.OutputReceived += (sender, args) =>
{
if (!string.IsNullOrEmpty(args.Data) && args.Data.StartsWith(StartedMessage, StringComparison.Ordinal))
{
output += args.Data.Substring(StartedMessage.Length) + '\n';
started.TrySetResult(0);
}

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Enable FunctionalTests/TestApp on Extensions Hosting

5 participants

@rosebyte@svick@mrek-msft
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

Re-enable Microsoft.Extensions.Hosting shutdown functional coverage in CI and harden shutdown startup/Helix/NET481 compatibility - #128252

Merged
rosebyte merged 44 commits into
mainfrom
copilot/enable-functional-tests-on-hosting
Jun 24, 2026
Merged

Re-enable Microsoft.Extensions.Hosting shutdown functional coverage in CI and harden shutdown startup/Helix/NET481 compatibility#128252
rosebyte merged 44 commits into
mainfrom
copilot/enable-functional-tests-on-hosting

Conversation

CopilotAI commented May 15, 2026

Copy link
Copy Markdown
Contributor

This PR now includes the full follow-up hardening required by CI feedback after re-enabling Microsoft.Extensions.Hosting functional shutdown coverage.

Final scope of changes

  • Re-enabled Hosting FunctionalTests in CI (removed CI gating/active issue suppression).
  • Updated shutdown test deployment to run from deployed output (AppContext.BaseDirectory) instead of source-tree project discovery.
  • Removed the load-bearing PreservePublishedApplicationForDebugging dependency by using a borrowed published app path that does not delete test output on cleanup. PublishedApplication.Dispose() is now virtual; BorrowedPublishedApplication overrides it as a no-op so teardown never deletes the test output directory.
  • Replaced external kill process usage in shutdown signaling with libc signal calls for Helix/minimal-image compatibility.
  • Centralized the libckill P/Invoke in ProcessExtensions.SendSignal(int pid, int signal); removed the duplicate declaration from ShutdownTests so both signal paths share a single interop entry point. The underlying P/Invoke method is named Kill (PascalCase) with EntryPoint = "kill" to match C# naming conventions while preserving the correct libc symbol binding. SendSignal explicitly throws PlatformNotSupportedException on Windows so callers get a clear, actionable error rather than a DllNotFoundException from the libc P/Invoke.
  • Resolved Unix SIGINT/SIGTERM numbers at runtime via SystemNative_GetPlatformSignalNumber instead of hard-coding them, matching the runtime's platform-specific signal mapping approach. Signal number initialization is gated behind an OS check so Windows does not attempt the P/Invoke during type initialization.
  • Kept Unix cleanup best-effort and resilient:
    • graceful termination first (SIGTERM via libc)
    • hard-kill fallback (Process.Kill()) when graceful termination times out or fails
    • tolerance for missing process tools / constrained environments
  • Increased startup/shutdown wait windows to 180 seconds for stress/JIT-stress stability.
  • Increased the final shutdown exit grace period before escalation to Process.Kill() to 30 seconds, reducing false failures when the app has finished shutdown logging but needs a little longer to exit cleanly under load.
  • Fixed the shutdown test startup race by wiring output observation before DeployAsync() starts process output capture via SelfHostDeployer.OutputReceived (now internal), so the initial Started line cannot be missed.
  • Fixed net481 compatibility by using Marshal.GetLastWin32Error().
  • Fixed WaitForExitOrKill race: now uses the boolean result from WaitForExit(timeout) directly and wraps Process.Kill() in a catch (InvalidOperationException) to tolerate the window where the process exits between the wait and the kill call.

Validation

  • ./build.sh clr+libs -rc release succeeds.
  • ./build.sh clr+libs+host -rc release -lc release succeeds.
  • Microsoft.Extensions.Hosting.Functional.Tests builds successfully for net481.
  • dotnet build /t:test Microsoft.Extensions.Hosting.Functional.Tests.csproj -c Release -f net11.0 /p:TestFilter=Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests passes (2/2).
  • CI-feedback-driven shutdown/process-cleanup paths were updated to match Helix/runtime constraints and avoid prior failure modes.

CopilotAI requested review from Copilot and removed request for CopilotMay 15, 2026 12:32
CopilotAI linked an issue May 15, 2026 that may be closed by this pull request
@github-actionsgithub-actionsBot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label May 15, 2026
CopilotAI requested review from Copilot and removed request for CopilotMay 15, 2026 13:03
CopilotAI changed the title [WIP] Fix FunctionalTests/TestApp on Extensions HostingRe-enable Microsoft.Extensions.Hosting FunctionalTests/TestApp shutdown coverageMay 15, 2026
CopilotAI requested a review from rosebyteMay 15, 2026 13:07
@github-actionsgithub-actionsBot added area-Extensions-Hosting and removed needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners labels May 15, 2026
@rosebyte

Copy link
Copy Markdown
Member

@copilot , the shutdown tests are failing with this error on helix machines:

Comments
{ "HelixJobId": "bf5b8a6f-0bdd-40b3-9fbc-f0eec7148f40", "HelixWorkItemName": "Microsoft.Extensions.Hosting.Functional.Tests" }

Error message
System.IO.DirectoryNotFoundException : Could not find Microsoft.Extensions.Hosting.TestApp.csproj from base directory '/root/helix/work/workitem/e/'.

Stack trace
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.GetApplicationPath() in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 173
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ExecuteShutdownTest(String testName, String shutdownMechanic) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 53
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestRun() in /_/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 35
--- End of stack trace from previous location ---

CopilotAI requested review from Copilot and removed request for CopilotMay 18, 2026 11:55

CopilotAI commented May 18, 2026

Copy link
Copy Markdown
ContributorAuthor

@copilot , the shutdown tests are failing with this error on helix machines:

...

Fixed in b9cd702. I removed the source-tree lookup and switched the shutdown tests to run TestApp from the deployed test output (AppContext.BaseDirectory) using a custom ApplicationPublisher, so Helix no longer requires Microsoft.Extensions.Hosting.TestApp.csproj to exist in the workitem.

CopilotAI changed the title Re-enable Microsoft.Extensions.Hosting FunctionalTests/TestApp shutdown coverageRe-enable Microsoft.Extensions.Hosting FunctionalTests/TestApp shutdown coverage across local and Helix environmentsMay 18, 2026
@rosebyte

Copy link
Copy Markdown
Member

@copilot , this time it failed with this error:

Test Failures (700 tests failed)
🔹 [All failing tests from runtime]

⚠️ 698 failing tests not displayed for this pipeline due to space limitations

❌ Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestRun [Console][Details][Artifacts][0.70% failure rate]
[ 🚧 Create issue against infrastructure team][ 📄 Create issue in this repo]
Failing Configuration
AzureLinux.3.0.ArmArch.Open
Exception Message
System.ComponentModel.Win32Exception : An error occurred trying to start process 'pgrep' with working directory '/root/helix/work/workitem/e'. No such file or directory
CallStack
at Microsoft.Win32.SafeHandles.SafeProcessHandle.ForkAndExecProcess(ProcessStartInfo startInfo, String resolvedFilename, String[] argv, IDictionary2 env, String cwd, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, Boolean usesTerminal, SafeHandle[] inheritedHandles, Holder& waitStateHolder, Boolean throwOnNoExec) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 369 at Microsoft.Win32.SafeHandles.SafeProcessHandle.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles, Holder& waitStateHolder) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 213 at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Unix.cs:line 363 at System.Diagnostics.Process.Start() in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1272 at System.Diagnostics.Process.Start(ProcessStartInfo startInfo) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1383 at Microsoft.Extensions.Internal.ProcessExtensions.RunProcessAndWaitForExit(String fileName, String arguments, TimeSpan timeout, String& stdout) in /_/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 93 at Microsoft.Extensions.Internal.ProcessExtensions.GetAllChildIdsUnix(Int32 parentId, ISet1 children, TimeSpan timeout) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 44
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process, TimeSpan timeout) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 33
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 17
at Microsoft.Extensions.Hosting.IntegrationTesting.ApplicationDeployer.ShutDownIfAnyHostProcess(Process hostProcess) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/ApplicationDeployer.cs:line 126
at Microsoft.Extensions.Hosting.IntegrationTesting.SelfHostDeployer.Dispose() in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/SelfHostDeployer.cs:line 179
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ExecuteShutdownTest(String testName, String shutdownMechanic) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 107
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestRun() in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 34
--- End of stack trace from previous location ---
❌ Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestWaitForShutdown [Console][Details][Artifacts][0.69% failure rate]
[ 🚧 Create issue against infrastructure team][ 📄 Create issue in this repo]
Failing Configuration
AzureLinux.3.0.ArmArch.Open
Exception Message
System.ComponentModel.Win32Exception : An error occurred trying to start process 'pgrep' with working directory '/root/helix/work/workitem/e'. No such file or directory
CallStack
at Microsoft.Win32.SafeHandles.SafeProcessHandle.ForkAndExecProcess(ProcessStartInfo startInfo, String resolvedFilename, String[] argv, IDictionary2 env, String cwd, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, Boolean usesTerminal, SafeHandle[] inheritedHandles, Holder& waitStateHolder, Boolean throwOnNoExec) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 369 at Microsoft.Win32.SafeHandles.SafeProcessHandle.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles, Holder& waitStateHolder) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 213 at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Unix.cs:line 363 at System.Diagnostics.Process.Start() in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1272 at System.Diagnostics.Process.Start(ProcessStartInfo startInfo) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1383 at Microsoft.Extensions.Internal.ProcessExtensions.RunProcessAndWaitForExit(String fileName, String arguments, TimeSpan timeout, String& stdout) in /_/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 93 at Microsoft.Extensions.Internal.ProcessExtensions.GetAllChildIdsUnix(Int32 parentId, ISet1 children, TimeSpan timeout) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 44
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process, TimeSpan timeout) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 33
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process) in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 17
at Microsoft.Extensions.Hosting.IntegrationTesting.ApplicationDeployer.ShutDownIfAnyHostProcess(Process hostProcess) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/ApplicationDeployer.cs:line 126
at Microsoft.Extensions.Hosting.IntegrationTesting.SelfHostDeployer.Dispose() in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/SelfHostDeployer.cs:line 179
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ExecuteShutdownTest(String testName, String shutdownMechanic) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 107
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestWaitForShutdown() in /
/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 41
--- End of stack trace from previous location ---

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

…plicate in ShutdownTests
Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

CopilotAIand others added 4 commits June 15, 2026 15:53
Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>
…on-hosting' into copilot/enable-functional-tests-on-hosting
Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>
Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

rosebyteand others added 2 commits June 16, 2026 23:04
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:83

  • OutputReceived is raised from background threads, but output is being mutated via output += ... without synchronization. This can lead to lost/garbled output (and flaky assertions) when multiple lines arrive quickly. Consider using a thread-safe accumulator (e.g., a StringBuilder guarded by a lock, or enqueue lines into a concurrent collection and join at the end) rather than mutating a shared string from the event callback.
 var started = new TaskCompletionSource<int>(TaskCreationOptions.RunContinuationsAsynchronously);
var completed = new TaskCompletionSource<int>(TaskCreationOptions.RunContinuationsAsynchronously);
var output = string.Empty;
deployer.OutputReceived += (sender, args) =>
{
if (!string.IsNullOrEmpty(args.Data) && args.Data.StartsWith(StartedMessage, StringComparison.Ordinal))
{
output += args.Data.Substring(StartedMessage.Length) + '\n';
started.TrySetResult(0);
}

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Enable FunctionalTests/TestApp on Extensions Hosting

5 participants

@rosebyte@svick@mrek-msft