fix(deploy): curl-free healthchecks, /ready, non-root images, k8s probes, state dir under /data - #342
Merged
Merged
Conversation
…pment
/health was the only probe and it is a constant 200 by design (liveness),
so nothing told an orchestrator when the host had actually finished
starting or had begun to shut down. /ready now answers 200 only between
IHostApplicationLifetime.ApplicationStarted and ApplicationStopping, and
503 ("starting" / "stopping") otherwise, so k8s readinessProbe, compose
--wait and load balancers can drain before Kestrel closes. The decision
is a pure function (EvaluateReadiness) so the lifetime transitions can be
pinned without an in-process host, which cannot show the drain window.
Swagger UI + document were served unconditionally. They enumerate every
mapped route and schema, so they are now on only in Development or when
Nexo:Api:EnableSwagger is set (Nexo__Api__EnableSwagger=true), and the
host logs when it is enabled outside Development. appsettings.Testing.json
keeps it on for the existing ingress tests.
ProdStyle coverage under Tests/VirtualProduction hits /health and /ready
on the real pipeline, including with built-in ApiKey auth on, since probes
carry no credentials.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>LiteDB stores and rollback snapshots (nexo-patterns.db, nexo-adaptation.db, nexo-adaptation-audit.db, nexo-copilot-tasks.db, nexo-execution.db, nexo-test-failures.db, nexo-snapshots/) defaulted to the CWD / repo root in the kernel registrar, the adaptation, self-context and rollback DI extensions and every CLI command. In a container that is /app, which is lost on recreate; in the agent-server stack it littered the bind-mounted repo under /work. RepoPathResolver.ResolveStateDirectory now centralises the default: NEXO_STATE_DIR (absolute, or relative to the resolved root) else <repo or app root>/.nexo/state (already gitignored via .nexo/), created on first use. Nexo:PatternStorePath / --store-path still win and keep the sibling files co-located with them. Backward compatibility is the simplest correct thing: an install with nexo-*.db already at the root and no .nexo/state yet keeps using the root (nothing is moved or logged); creating .nexo/state, or setting NEXO_STATE_DIR, switches to the new layout. Documented in docs/Configuration.md. The portal, agent-server, private and cloud compose stacks mount a nexo-state named volume at /data/state and set NEXO_STATE_DIR to match. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… surface --wait failures Dockerfile.api, Dockerfile.quickstart and Dockerfile.fleet-host declared a curl HEALTHCHECK on mcr.microsoft.com/dotnet/aspnet:8.0, which ships no curl, so every container sat at "(unhealthy)" forever and `docker compose up --wait` could never succeed. The probe now uses bash's /dev/tcp (bash and grep are in the runtime image) to GET /health and require a 200. Dockerfile.agent-server is an SDK image and keeps curl. scripts/prod-dry-run.sh piped `up -d --wait` stderr to /dev/null, which hid exactly that failure and burned ~90 s in the fallback loop with no hint why; stderr is kept now. The runtime stages of the api, quickstart, fleet-host and cli images now run as the aspnet image's unprivileged `app` user (USER $APP_UID, uid 1654 in .NET 8+, numeric so k8s runAsNonRoot can verify it). Everything the process writes lives under /data (state at NEXO_STATE_DIR=/data/state, dailies at NEXO_DAILIES_PATH=/data/dailies), created and chowned to app; /app stays root-owned and read-only. Dockerfile.agent-server intentionally stays root: its tester/optimizer agents run dotnet build/test against a host bind mount whose ownership the image cannot know. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…orker sample The sample set NEXO_DEPLOYMENT_PROFILE=Worker, which AddNexo() rejects (valid: full, server, edge, air-gapped, system), so the pod threw at startup and CrashLoopBackOff'd. A mesh worker executes bricks over the runtime transport, which needs the Server composition (same modules as Full; Edge strips NCR + agents), so it is `server` now, with the accepted set called out inline. Adds httpGet livenessProbe on /health and readinessProbe on /ready, a named http port, NEXO_STATE_DIR=/data/state (per-replica scratch unless a PVC is mounted), a runAsNonRoot/allowPrivilegeEscalation=false security context matching the image's app user, and a commented ghcr.io image reference with the pin-a-tag advice from docs/DEPLOYMENT.md. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…yment notes Nexo.Transport.Grpc.Server.Host had no appsettings.json, forced HTTP/2 in code and documented neither its port nor TLS; the API client's Nexo:GrpcTransport defaults name /run/secrets/* files that no compose file mounts. Adds appsettings.json with the knobs the host already reads (Urls http://127.0.0.1:5001 so it does not collide with the API on 5000, Kestrel EndpointDefaults Http2, Nexo:GrpcTransport with null cert paths) and docs/GrpcHost.md covering the listen address, h2c vs TLS, mTLS, the client-side options, and a compose secrets snippet that mounts the /run/secrets/* paths the API defaults expect. docker-compose.ephemeral.yml carried a commented-out nexo service pointing at a non-existent .docker/Dockerfile.ephemeral with a `.` context that would have been deploy/compose/. "Ephemeral" is disposable Ollama (+ optional Postgres) plus a one-off `run --rm nexo ...`, so the service is kept and pointed at .docker/Dockerfile.cli with context ../..; the two doc rows now say what the file is for. docker-compose.provenance.yml drops the obsolete `version:` key. docs/DEPLOYMENT.md gains the container health / readiness / non-root / Swagger / k8s-profile notes and the runtime-state section that the other commits in this PR refer to. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…-studio stores writable under the non-root user Reviewer verified two defects on the branch: - appsettings.json 'Urls' loads AFTER host configuration, so ASPNETCORE_URLS was silently ignored (bound to loopback :5001 with TLS never engaged). The loopback default now lives in Program.cs and applies only when no URL is configured. - With USER app, stores that default to <cwd>/.nexo (/app, root-owned) threw UnauthorizedAccessException; runtime-studio roots now point under /data/state and /app/.nexo is created writable as a safety net for the rest. Also reword the ephemeral compose header (the env var never switched the profile). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
IanFrelinger
commented
Aug 17, 2026
OwnerAuthor
Docker daemon is unresponsive on the dev box, so the non-root images + |
…ullable error in RepoPathResolver The readiness run on this branch (31992535899) caught both: Nexo.API failed to start as uid 1654 with UnauthorizedAccessException on /app/config (TrustPolicyPackRegistry writes <repoRoot>/config/trust-packs), and the netstandard2.0 build of Nexo.Core.Application failed CS8602 at RepoPathResolver.cs:89 (no nullable annotation on IsNullOrWhiteSpace there). Pre-create and chown /app/config like /app/.nexo; add the null-forgiving operator. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
IanFrelinger
commented
Aug 17, 2026
OwnerAuthor
Readiness run 31992535899 caught two real defects (non-root /app/config write; netstandard2.0 CS8602) — fixed in e229462. Re-dispatched: https://github.com/IanFrelinger/Nexo/actions/runs/31996300624 |
…ter's renormalized file + this PR's state-dir edits)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Production-readiness audit, M8 + M9 + M12 + L2 + L3 + L4. Ledger: https://claude.ai/code/artifact/15fa4198-97b8-4da6-bea0-0c24d364a7f3
Why
.docker/Dockerfile.api|quickstart|fleet-hostdeclared acurlHEALTHCHECK onmcr.microsoft.com/dotnet/aspnet:8.0, which ships no curl, so every container was(unhealthy)forever andcompose up --waitcould never pass;scripts/prod-dry-run.shhid that with2>/dev/nulland burned ~90 s./healthwas a constant 200 and there was no/ready.deploy/k8s/nexo-mesh-worker-deployment.yamlsetNEXO_DEPLOYMENT_PROFILE=Worker, whichAddNexo()rejects → CrashLoopBackOff; no probes.Nexo:GrpcTransportdefaults name/run/secrets/*no compose mounts.docker-compose.ephemeral.ymlpointed at a non-existent.docker/Dockerfile.ephemeralwith the wrong context;docker-compose.provenance.ymlhad an obsoleteversion:.What
Five commits:
GET /ready: 200 only betweenApplicationStartedandApplicationStopping, 503 (starting/stopping) otherwise; decision is a pureEvaluateReadinessso lifetime transitions are testable. Swagger on only inDevelopmentorNexo:Api:EnableSwagger=true(logged when enabled outside Development);appsettings.Testing.jsonkeeps it on for existing ingress tests./readyadded toIngressCatalog. ProdStyle testTests/VirtualProduction/HealthAndReadinessProdStyleTests.cshits/health+/ready, incl. with ApiKey auth on.RepoPathResolver.ResolveStateDirectory:NEXO_STATE_DIR(absolute or root-relative) else<repo/app root>/.nexo/state(.nexo/already gitignored, created on first use). Wired into kernel registrar (copilot tasks, default pattern store), adaptation/self-context/rollback extensions,FileSnapshotStore, and the 8 CLI commands (--store-pathhelp updated). Back-compat: ifnexo-*.dbalready sit at the root and.nexo/statedoes not exist, the root keeps being used (no move, no log — documented as the simpler correct thing; create.nexo/stateor setNEXO_STATE_DIRto migrate). Portal / agent-server / private / cloud compose stacks mount anexo-statenamed volume at/data/state+NEXO_STATE_DIR. 4 unit tests inRepoPathResolverTests. Docs:Configuration.md,api/index.md,SelfHostedAgentServer.md,agent-server.env.example,private-backup-restore.md./dev/tcpGET/health+grep " 200 "on the three aspnet images (agent-server is SDK, keeps curl);prod-dry-run.shkeeps--waitstderr. Runtime stages of api/quickstart/fleet-host/cli run asUSER $APP_UID(aspnetappuser, uid 1654 — the spec's 64198 is not what the MS images use);/data/state+/data/dailiescreated and chowned toapp,NEXO_STATE_DIR/NEXO_DAILIES_PATHpoint there so nothing writes under root-owned/app.Dockerfile.agent-serverintentionally stays root (tester/optimizer agentsdotnet build/testa host bind mount of unknown ownership) — called out in the file.server(Server = Full composition; Edge strips NCR + agents, wrong for a worker that executes bricks), accepted set commented inline; httpGet liveness/health+ readiness/ready; named port;NEXO_STATE_DIR;runAsNonRoot+allowPrivilegeEscalation: false; commentedghcr.io/ianfrelinger/nexo-api:<tag>with pin advice.Nexo.Transport.Grpc.Server.Host/appsettings.json(Urlshttp://127.0.0.1:5001— avoids the API's 5000,Kestrel:EndpointDefaults:Protocols=Http2,Nexo:GrpcTransportnull cert paths) +docs/GrpcHost.md(h2c vs TLS, mTLS, client options, compose secrets snippet mounting the/run/secrets/*the API defaults expect) + DocsIndex row. Ephemeral compose: kept thenexoservice (it is what "ephemeral" is for:run --rm nexo ...against disposable Ollama/Postgres) and pointed it at.docker/Dockerfile.cliwithcontext: ../.., behind thefullprofile so compose-gate's--profile dblane is untouched; README/DEPLOYMENT rows say what it is. Droppedversion:from provenance.docs/DEPLOYMENT.mdgets the container health/readiness/non-root/Swagger/k8s-profile section and the runtime-state section.Verification
dotnet build(DOTNET_ROLL_FORWARD=Major): Nexo.API, Nexo.CLI, Nexo.Transport.Grpc.Server.Host (appsettings.json copied to output), Nexo.Tests.Application, Nexo.Tests.Infrastructure (net8.0 + net9.0) — 0 warnings, 0 errors.dotnet test -f net9.0Tests.Infrastructure:HealthAndReadinessProdStyleTests+MiddlewareIngressIntegrationTests9/9 pass;Rollback|Snapshot|Adaptation107/107 pass. Tests.ApplicationRepoPathResolverTests12/12 pass. Test run created<worktree>/.nexo/state/and it is gitignored (confirms M12 +.nexo/ignore).docker compose -f <each touched compose> config --quiet— all six valid (provenance / cloud need their required env vars, as before).docker-desktopWSL distro Stopped,docker version/docker infohang past 5 min) → could notdocker build/run the images locally or run kubeconform; kubectl client dry-run needs cluster discovery. k8s YAML validated by careful read. CI'sfull-platform-readiness-gatedocker-all-images job builds api/quickstart/agent-server and requires/health(dispatch on the branch to prove);prod-dry-run-pris workflow_dispatch.Notes / deliberate scope calls
Dockerfile.agent-serverroot (see above).Dockerfile.game-director(commercial, SDK image with curl) and Fleet.Host's unconditional Swagger left untouched — outside the listed files.nexo-step-modes.json(StepExecutionModeStore) still defaults to repo root: it is a JSON config file, not LiteDB state, and its Save() already fails soft; not moved.chown -R 1654:1654documented in DEPLOYMENT.md.🤖 Generated with Claude Code
Reviewer follow-ups applied on top (verified defects): the gRPC host's
appsettings.jsonUrlssilently overrodeASPNETCORE_URLS(loads after host config) — the loopback default now lives inProgram.csand yields to env/--urls; underUSER app, stores defaulting to<cwd>/.nexothrewUnauthorizedAccessException— runtime-studio roots now point under/data/stateand/app/.nexois writable. Docker is down on the dev box, so the non-root images are proven by dispatching full-platform-readiness on this branch (link in comments).