test(readiness): fix the four identifiable readiness-gate flakes at the root - #335
Merged
Conversation
…he root The full-platform-readiness gate was red on roughly 43% of master pushes from four flakes that each had a real cause; none of them needed a retry loop until green. Signatures, so future reds can be matched: 1. EndpointHealthMonitorTests (Nexo.Tests.Orchestration) Signature: "Expected registry.Updates to contain a single item, but the collection is empty" in ExecuteAsync_HealthyProbe_UpdatesRegistryTrue / ExecuteAsync_UnhealthyProbe_* . Cause: a fixed Task.Delay(300) before StopAsync, while the first probe is a real gRPC round-trip (or a connection refusal) whose latency depends on the host; StopAsync then cancels the probe before UpdateHealth ran. Fix: bounded poll (50 ms, <= 5 s) on registry.Updates before StopAsync; assertions unchanged. 2. AgentTransportServiceImplReflectionGapCoverageTests (Nexo.Tests.Transport) Signature: "Expected process.WaitForExit(10000) to be true" or an IOException from Directory.Delete in CreateCertificateWithSan. Cause: the SAN certificate was produced by shelling out to openssl (PATH dependent, 10 s cap, Process never disposed, and the temp-dir delete in finally masked the timeout). Fix: generate the certificate in-process with CertificateRequest + SubjectAlternativeNameBuilder (same shape as HttpBarrierContextMiddleware tests). No openssl at all. 3. ApiDevelopmentHostDiTests / API host under a non-Full profile Signature: "System.InvalidOperationException: Unable to resolve service for type 'Nexo.Transport.Grpc.GrpcAgentTransport' while attempting to activate 'Nexo.Runtime.Routing.EndpointHealthMonitor'." (Hosting failed to start). Cause (test): AutonomyCompositionTests, McpA2AProtocolIngressProdStyleTests and HostingE2ESmokeTests set the process-wide NEXO_DEPLOYMENT_PROFILE outside the serialized "EnvironmentVariables" collection while xunit runs collections in parallel, so a concurrent API host build saw airgapped/edge. Cause (product): AddNexoRuntimeRouting registers EndpointHealthMonitor unconditionally while the kernel registers its GrpcAgentTransport dependency only for profiles with runtime transport (Full/Server) -- every host that calls AddNexoRuntimeRouting + AddNexo (API, CLI, daemon, fleet host) failed DI under airgapped/edge/system. Fix (test): the env-setting tests now live in [Collection( "EnvironmentVariables")] (AutonomyCompositionAirGappedProfileTests; the MCP refusal moved to AirGappedProfileApiHostProdStyleTests since a class can be in only one collection; HostingE2ESmokeTests joined the collection). Fix (product): EndpointHealthMonitor gained a transport-less constructor and stays idle when no GrpcAgentTransport is registered -- the same condition the kernel gates the transport on, evaluated at resolution time, so it holds for every host, not just Nexo.API. Existing 4-arg constructor and its null guard are unchanged; DI picks it whenever the transport is registered. New ProdStyle test boots the real Nexo.API host under airgapped and hits /health (verified failing with the signature above before the fix). 4. CertifiedBrickHotSwapHostTests.SecondSwap_ReplacesGeneration_AndOldContextIsCollected Signature: "Expected second.PreviousGenerationCollected to be true because the drained generation-1 context must be collected once its bricks are dropped, but found False" (provenance shows GenerationLeakSuspected). Cause: WaitForContextRelease ran 10 back-to-back GC passes on the thread that is often the one that just completed the last invocation of the retiring generation; that thread's frames still root the finished invocation's state machine (and a brick instance from the old context) until they unwind, and no number of collections frees a stack-rooted object. Fix: WaitForContextReleaseAsync -- up to 20 passes with a real Task.Delay(10 ms) yield between passes, which gives the stack back so the next pass succeeds. The retry has to live in the host, not the test: the flag the test asserts is what the host observed, so a test-side sweep can never turn it true (comment added at the assertion; the assertion is kept). 5. VirtualProductionNcrRoutingTests (macOS, run 31982502428) Signature: "System.Net.HttpListenerException : Address already in use" from HttpListener.Close() in RunPodLoopbackApiServer.Dispose during teardown of NcrCapabilityPoller_exposes_environment_hardware_profiler_snapshot. Cause: the helper already binds an ephemeral port (TcpListener probe then release); on macOS/Linux the managed HttpListener re-binds the prefix's endpoint while tearing it down and fails when the just-released port is in TIME_WAIT. Close() was the one unguarded call. Fix: Close() now swallows HttpListenerException/ObjectDisposedException (teardown only; assertions run before it); VirtualProductionNcrRoutingHost restores its process-wide env overrides in a finally, and the test class joined the "EnvironmentVariables" collection because it sets NEXO_TOTAL_VRAM_BYTES / NEXO_AVAILABLE_VRAM_BYTES. Verification: built Nexo.Tests.Orchestration, Nexo.Tests.Transport (net8.0) and Nexo.Tests.Infrastructure (net8.0 + net9.0), 0 warnings; ran EndpointHealthMonitorTests (7/7), the cert tests (9/9), HostingDeploymentProfileTests + HostingE2ESmokeTests + the moved airgapped tests + AirGappedProfileApiHostProdStyleTests + CertifiedBrickHotSwapHostTests + VirtualProductionNcrRoutingTests + McpA2AProtocolIngressProdStyleTests + ApiDevelopmentHostDiTests + RuntimeServiceCollectionExtensions* on net9.0 (83/83) and the net8.0-compilable subset (75/75); wider HotSwap|Autonomy|Ncr| Routing sweep on net9.0 151/151. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…er; fix a stale comment The air-gapped MCP/A2A enable-refusal test moved out of McpA2AProtocolIngressProdStyleTests into AirGappedProfileApiHostProdStyleTests (a class can be in only one xunit collection), so mcp-a2a-gate.yml's name filter no longer ran it. Add the new class to the filter. Also: both the EnvironmentVariables and Integration collections are DisableParallelization=true; in xunit 2.9 non-parallel collections run after the parallel ones, one at a time, so an env-setting host boot in one never overlaps a host boot in the other. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
…utonomyCompositionTests)
Uh oh!
There was an error while loading. Please reload this page.
IanFrelinger pushed a commit
that referenced
this pull request
Aug 17, 2026
…rograms historical; document spikes/ docs/DocsIndex.md: Start Here #1 is the tester quickstart, README #2, GettingStarted #3; IntegratorGuide and consumer-template/CONSUMING.md are listed for the first time; RELEASE.md, the setup-matrix scripts and the bootstrap helpers move out of the numbered list into an "escape hatches" paragraph (RELEASE stays under Operator); a new "Trust loop / certification (experimental, hold-mode)" group makes docs/trust-loop/*, the evidence ledger, governed-pipeline, SELF-EXTEND-AUDIT, the sample objective and spikes/ reachable; "Planning & Roadmap" becomes "Planning history (historical as of 2026-08-16)" because it described a finished program. Stale status docs are date-stamped rather than deleted: - ProtocolIntegration-MCP-A2A.md said the Nexo.API wiring was "planned"; it landed in #269/#270 (Program.cs registers all four adapters and maps /api/mcp and /api/a2a/{agentId}; IngressCatalog rows McpServer/A2AAgents). - ci-pre-existing-failures.md said the Full Platform Readiness Gate was RED as of 06-21; it went green with #317-#320/#335. Kept because docs/planning/* still cite it. - GapAnalysis.md and NorthStarGapAnalysis.md predate the trust loop; both get a historical banner naming their still-open rows and pointing at the ledger, ProjectTiers and CHANGELOG. GettingStarted.md #5 contradicted DocsIndex on "installer" scripts (the scripts/install/* helpers exist but bootstrap the container lane); it now says so and points first-run readers at the quickstart. The pointer to a README section "First Successful Pipeline Run" (which does not exist) now names the sections that do. IntegratorGuide.md recommended `dotnet build Nexo.sln` (the docs guard forbids it in README); it now builds Nexo.Kernel.sln and the CLI project, with hello-brick as the smallest test example, and the compatibility matrix names the solutions that actually apply. spikes/README.md: the 27 tracked files under spikes/ were cited as PASS evidence by the ledger but undocumented. It says what each spike is, every flag of the flight runner, which ledger rows cite each spike, and that none of it is a supported entry point. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Aug 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Production-readiness audit, H10. Ledger: https://claude.ai/code/artifact/15fa4198-97b8-4da6-bea0-0c24d364a7f3
Why
The full-platform-readiness gate has been red on ~43% of master pushes from identifiable flakes, plus a fourth one on macOS in run 31982502428. Each had a real cause; none is fixed with a retry-until-green. Signatures are listed so future reds can be matched.
What
1. EndpointHealthMonitorTests (Nexo.Tests.Orchestration) — signature:
Expected registry.Updates to contain a single item, but the collection is empty. A fixedTask.Delay(300)raced a real gRPC round-trip / connection refusal, andStopAsynccancelled the probe beforeUpdateHealthran. Now a bounded 50 ms poll (<= 5 s) onregistry.UpdatesbeforeStopAsync; assertions unchanged. Also addsExecuteAsync_WithoutGrpcTransport_StaysIdle_AndDoesNotTouchRegistryfor the product change in (3).2. AgentTransportServiceImplReflectionGapCoverageTests (Nexo.Tests.Transport) — signature:
Expected process.WaitForExit(10000) to be true/ IOException fromDirectory.Delete. The SAN cert was produced by shelling out toopenssl(PATH-dependent, 10 s cap, undisposed Process, temp-dir delete masking the timeout). Now generated in-process withCertificateRequest+SubjectAlternativeNameBuilder(mirrors HttpBarrierContextMiddlewareTests). No openssl at all.3. API host DI under non-Full profile — signature:
InvalidOperationException: Unable to resolve service for type 'Nexo.Transport.Grpc.GrpcAgentTransport' while attempting to activate 'Nexo.Runtime.Routing.EndpointHealthMonitor'(Hosting failed to start).AutonomyCompositionTests,McpA2AProtocolIngressProdStyleTestsandHostingE2ESmokeTestsset the process-wideNEXO_DEPLOYMENT_PROFILEoutside[Collection("EnvironmentVariables")]while xunit runs collections in parallel. Fix: the airgapped autonomy test moved toAutonomyCompositionAirGappedProfileTests(same file, in the collection, usesEnvironmentVariableScope); the MCP refusal moved to the newTests/VirtualProduction/AirGappedProfileApiHostProdStyleTests(a class can be in only one collection);HostingE2ESmokeTestsjoined the collection.AddNexoRuntimeRoutingregistersEndpointHealthMonitorunconditionally while the kernel registersGrpcAgentTransportonly when the profile includes runtime transport — every host that callsAddNexoRuntimeRouting+AddNexo(API, CLI, daemon, fleet host) failed DI under airgapped/edge/system. Fix insrc/Nexo.Runtime/Routing/EndpointHealthMonitor.cs: a transport-less constructor; DI picks the existing 4-arg constructor whenever the transport is registered (unchanged, null guard kept) and falls back otherwise, in which caseExecuteAsynclogs once and stays idle. This is the kernel's profile condition evaluated at resolution time, so it holds for every host, not just Nexo.API. New ProdStyle test boots the real Nexo.API host underairgapped, hits/health, asserts the monitor is registered and the transport is not — verified failing with the exact signature above before the fix.4. CertifiedBrickHotSwapHostTests.SecondSwap_ReplacesGeneration_AndOldContextIsCollected — signature:
Expected second.PreviousGenerationCollected to be true ... but found False(provenance showsGenerationLeakSuspected).WaitForContextReleaseran 10 back-to-back GC passes on the thread that is often the one that just completed the last invocation of the retiring generation; its frames still root that invocation's state machine (and a brick instance from the old context) until they unwind, and no number of collections frees a stack-rooted object. Fix inCertifiedBrickHotSwapHost:WaitForContextReleaseAsync— up to 20 passes with a realTask.Delay(10 ms)yield between passes. The retry lives in the host, not the test: the asserted flag is what the host observed, so a test-side sweep could never turn it true (comment added at the assertion; the assertion is kept).5. VirtualProductionNcrRoutingTests (macOS run 31982502428) — signature:
HttpListenerException: Address already in usefromHttpListener.Close()inRunPodLoopbackApiServer.Dispose. The helper already binds an ephemeral port; on macOS/Linux the managed HttpListener re-binds the prefix's endpoint during teardown and fails while the just-released port is in TIME_WAIT.Close()was the one unguarded call: now catchesHttpListenerException/ObjectDisposedException(teardown only).VirtualProductionNcrRoutingHost.DisposeAsyncrestores its env overrides in afinally, and the test class joined[Collection("EnvironmentVariables")](it setsNEXO_TOTAL_VRAM_BYTES/NEXO_AVAILABLE_VRAM_BYTES).Verification
Nexo.Tests.Orchestration,Nexo.Tests.Transport(net8.0) andNexo.Tests.Infrastructure(net8.0 + net9.0): 0 warnings, 0 errors.EndpointHealthMonitorTests7/7; cert tests 9/9.HotSwap|Autonomy|Ncr|Routingsweep 151/151.EndpointHealthMonitorchange stashed,Airgapped_profile_api_host_starts_with_an_idle_endpoint_health_monitorfails with the signature in (3).Note: the readiness gate is path-filtered and has no PR trigger; dispatch
full-platform-readiness-gate.ymlon this branch to prove it.🤖 Generated with Claude Code
Reviewer follow-ups applied on top: the air-gapped refusal test is added back to
mcp-a2a-gate.yml's filter (it moved classes); a stale hot-swap comment fixed. The reviewer's concern thatAirGappedProfileApiHostProdStyleTests(EnvironmentVariables collection) could overlap Integration-collection host boots does not hold: both collections areDisableParallelization = true, and xunit 2.9 runs non-parallel collections after the parallel ones, one at a time.