Quota probes run only in the orchestrator process, so a pool whose credential lives on an executor cannot be metered

docs/operating/quota.md

@@ -127,6 +127,54 @@ refused (never dispatched on an unkeyed reading) and the reason names the
 member and the unresolved pool. `/quota` reports each member's `pool` and
 `poolKind` so operators can see which members share a meter.
 
+## Executor-reported pools: metering a credential the orchestrator cannot read
+
+By default every pool is orchestrator-probed: the orchestrator holds the
+credential and probes directly in-process. When the credential for an
+account lives on an executor host the orchestrator cannot read, that
+account has no reading — so the pool can instead be metered from readings
+the executor reports:
+
+```json
+"QuotaRouter": {
+  "Pools": {
+    "edge-sub": {
+      "Kind": "ResettingWindow",
+      "ProbeSource": "ExecutorReported",
+      "ReportedReadingMaxAgeSeconds": 300,
+      "HolderHostIds": ["edge-gpu-01"]
+    }
+  }
+}
+```
+
+The holding executor probes locally with its own credential and POSTs each
+reading (pool, available percentage or balance, reset time, observed time)
+to `/executors/{hostId}/quota-reports`. The request must carry that host's
+own token: each executor gets a named `CodeyBox:ApiClients` entry whose
+`ExecutorHostId` equals its host id, and the executor uses that token (not
+the shared operator key) as its API bearer. The shared operator key and any
+token without a host binding are rejected on this endpoint with `403`, and
+a bound token reporting for a different host is rejected too — the path
+host is matched against the authenticated caller's binding before the
+registry is even consulted, so a rejected caller cannot probe which host
+ids are registered. The orchestrator validates every
+report on arrival — the pool must exist and be executor-reported, the
+reporting host must be declared in the pool's `HolderHostIds` (exact match;
+a report from any other host is rejected and the stored reading is left
+unchanged), percentages must sit within 0–100, balances must be finite and
+non-negative, and a depleting-balance pool never carries a reset instant —
+and meters the pool from the latest fresh report.
+
+The orchestrator stays the sole authority for the gate decision: an
+executor reports readings and never decides admission. A stale report
+(older than `ReportedReadingMaxAgeSeconds`) or a pool that never reported
+reads as unknown — never as healthy headroom — and flows through the same
+unknown handling as a direct probe, including the Transient / Permanent /
+NoCredential distinction and fail-closed whenever a non-zero floor is in
+force. `ReportedReadingClockSkewSeconds` (default 300) bounds how far in
+the future a report's observed time may be before it is rejected.
+
 ## Replenishment kinds: resetting windows vs depleting balances
 
 Two kinds of allowance are in use and they are not interchangeable:

docs/reference/api.md

@@ -31,6 +31,16 @@ at least 32 characters or the service refuses to start.
           "Subject": "service",
           "DisplayName": "JobTrack"
         }
+      },
+      {
+        "Name": "exec-1",
+        "TokenEnvVar": "CODEYBOX_EXEC_1_API_KEY",
+        "ExecutorHostId": "exec-1",
+        "Principal": {
+          "Issuer": "codeybox",
+          "Subject": "exec-1",
+          "DisplayName": "Executor exec-1"
+        }
       }
     ]
   }
@@ -44,6 +54,14 @@ Delegated identities are authenticated claims from the integration—not
 credentials—and are persisted with the work item for API, commit, and pull
 request attribution.
 
+A client with `ExecutorHostId` is a host-bound executor token: it may act
+only as that executor host on host-scoped executor endpoints (notably
+`POST /executors/{hostId}/quota-reports`, which rejects any other caller —
+including the operator key — so a shared bearer cannot forge another host's
+quota meter). Give each executor host its own token environment variable and
+matching `ExecutorHostId`, and put that token (not the operator key) in the
+executor's `ApiKeyEnvVar` on that host.
+
 ### GitHub App delivery credentials
 
 For team installations, configure the GitHub upstream with a GitHub App
@@ -1398,6 +1416,21 @@ Heartbeat a registered executor into the worker registry. Request body is `{ "cu
 
 Remove an executor registration (clean shutdown). Response: `200 OK` with `{ "hostId": "exec-1" }`.
 
+### `POST /executors/{hostId}/quota-reports`
+
+Report one quota reading for a pool whose credential the calling host holds
+(pool identity, availability reading, reset time, observed time). The bearer
+must be a per-executor token bound to the path host (see `ExecutorHostId`
+under [Authentication](#authentication)); the shared operator key and any
+token without a host binding are rejected, so one bearer holder cannot forge
+another host's meter. The host must also be registered and be declared in
+the pool's `HolderHostIds`. Response: `200 OK` with `{ "accepted": true,
+"pool": "<pool>" }`. `401` with no bearer, `403` on a token/host mismatch,
+`404` for an unregistered host, `400` on a rejected report (unknown pool,
+non-holder, out-of-range reading, inconsistent reset). Accepting a report
+never decides admission — the orchestrator's quota gate does that. See
+[`quota.md`](../operating/quota.md) for the executor-reported pool design.
+
 ### `GET /sandboxes/leaked`
 
 Returns the provider-owned persistent sandboxes detected as leaked on the most

docs/reference/configuration.md

@@ -718,7 +718,7 @@ Tuning knobs for the quota probe and deferred-requeue logic.
 | `EndFloorPct` | `3` | Global late-window ramp floor as reset approaches. |
 | `RampWindowSeconds` | `604800` | Global quota-window length used for the ramp calculation. |
 | `FloorByAgent` | `{}` | Optional per-agent overrides keyed by agent kind, e.g. `codex` or `claude`. Each entry may set `StartFloorPct`, `EndFloorPct`, `MinQuotaPct`, and `RampWindowSeconds`; omitted fields inherit global values, and omitted agents use the global ramp. |
-| `Pools` | `{}` | Optional quota pools keyed by pool name. Each pool names one underlying account or subscription; class members join via their `Pool` reference and share one reading, one floor, and one reservation escrow. Each entry sets `Kind` (`ResettingWindow` or `DepletingBalance`), optional `BalanceUnit`, and optional `ReservationEstimate` (native units). Hot-reloadable. See `docs/operating/quota.md`. |
+| `Pools` | `{}` | Optional quota pools keyed by pool name. Each pool names one underlying account or subscription; class members join via their `Pool` reference and share one reading, one floor, and one reservation escrow. Each entry sets `Kind` (`ResettingWindow` or `DepletingBalance`), optional `BalanceUnit`, and optional `ReservationEstimate` (native units). Each entry may also set `ProbeSource` (`OrchestratorDirect`, the default, or `ExecutorReported` for accounts whose credential lives on an executor host), `ReportedReadingMaxAgeSeconds` (staleness bound for executor reports, default `300`), and `HolderHostIds` (executor hosts authorised to report for the pool; reports from other hosts are rejected). Hot-reloadable. See `docs/operating/quota.md`. |
 | `FloorByPool` | `{}` | Optional per-pool floor overrides keyed by pool name, alongside `FloorByAgent`. For a pool member the higher of the pool-resolved and agent-resolved floors wins. Resetting-window pools use the percentage fields (`MinQuotaPct`, `StartFloorPct`, `EndFloorPct`, `RampWindowSeconds`); depleting-balance pools use absolute `MinBalance`. Mixing units is rejected at load. Hot-reloadable. |
 | `QuotaRecheckIntervalSeconds` | `300` | Seconds to wait before re-probing when all Subscription members are exhausted. |
 | `QuotaCacheTtlSeconds` | `60` | Seconds to cache a quota probe result (per probe instance). |
@@ -730,6 +730,7 @@ Tuning knobs for the quota probe and deferred-requeue logic.
 | `ExpectedResets` | `{}` | Optional per-agent expected reset declarations, keyed by agent kind. Each entry may set explicit `Timestamps` and/or a recurring `CadenceSeconds` with `CadenceAnchor`; the policy paces to the sooner of the live probe reset and the next expected reset. Hot-reloadable. |
 | `ObservedFailureWindowMinutes` | `10` | Minutes a recent quota-shaped failure blocks the same agent/model across all projects. |
 | `ObservedFailureRetentionMinutes` | `30` | Minutes observed quota failures remain in `state.db`. |
+| `ReportedReadingClockSkewSeconds` | `300` | Tolerance in seconds for clock skew between executor and orchestrator hosts when validating an executor-reported reading's observed time. A report dated further in the future than this is rejected. Hot-reloadable. |
 | `ProbeMaxRetries` | `2` | Additional retries on a transient probe failure (network error / timeout / 5xx) before recording the failure. Hot-reloadable; currently honoured by the Claude probe. |
 | `ProbeRetryInitialDelayMs` | `250` | Base retry backoff in milliseconds; doubles each attempt. Hot-reloadable. |
 | `ProbeRetryMaxDelaySeconds` | `300` | Maximum between-retry delay, including provider `Retry-After` values on Anthropic OAuth usage requests. Hot-reloadable. |

src/CodeyBox.Api/AgentConfigHotReload.cs

@@ -1267,6 +1267,11 @@ private static string SerializeQuotaRouter(QuotaRouterConfig opts)
                         Kind = kv.Value.Kind.ToString(),
                         kv.Value.BalanceUnit,
                         kv.Value.ReservationEstimate,
+                        ProbeSource = kv.Value.ProbeSource.ToString(),
+                        ReportedReadingMaxAgeSeconds = checked((int)kv.Value.ReportedReadingMaxAge.TotalSeconds),
+                        HolderHostIds = kv.Value.HolderHostIds
+                            .OrderBy(h => h, StringComparer.Ordinal)
+                            .ToArray(),
                     }),
                 FloorByPool = mapped.FloorByPool
                     .OrderBy(kv => kv.Key, StringComparer.OrdinalIgnoreCase)
@@ -1289,6 +1294,7 @@ private static string SerializeQuotaRouter(QuotaRouterConfig opts)
                 UnknownPolicy = opts.UnknownPolicy.ToString(),
                 opts.ObservedFailureWindowMinutes,
                 opts.ObservedFailureRetentionMinutes,
+                opts.ReportedReadingClockSkewSeconds,
                 opts.CapRetryIntervalSeconds,
                 opts.ColdStartFitInWindow,
                 opts.DrainAggressiveness,

src/CodeyBox.Api/ApiKeyAuth.cs

@@ -59,8 +59,9 @@ public static void Configure(WebApplicationBuilder builder)
                     throw new InvalidOperationException(
                         $"{client.TokenEnvVar} must contain at least 32 characters of high-entropy random data.");
                 ValidateInitiator(client.Principal);
+                var executorHostId = NormalizeExecutorHostId(client);
                 resolved.Add(new ResolvedApiClient(
-                    client.Name, token, client.Principal, client.CanDelegateInitiator));
+                    client.Name, token, client.Principal, client.CanDelegateInitiator, executorHostId));
             }
 
             return new ApiKeyState(Token: key, Disabled: false, Clients: resolved);
@@ -82,7 +83,7 @@ public static IApplicationBuilder UseApiKeyAuth(
             if (state.Disabled)
             {
                 ctx.Items[PrincipalItemKey] = new ApiClientPrincipal(
-                    "authentication-disabled", OperatorInitiator, CanDelegateInitiator: false);
+                    AuthenticationDisabledClientName, OperatorInitiator, CanDelegateInitiator: false);
                 await next();
                 return;
             }
@@ -133,6 +134,25 @@ public static bool IsAuthorized(HttpContext ctx, ApiKeyState state)
         return state.Disabled || TryAuthenticate(ctx, state, out _);
     }
 
+    internal static bool TryGetPrincipal(HttpContext context, out ApiClientPrincipal? principal)
+    {
+        ArgumentNullException.ThrowIfNull(context);
+        if (context.Items.TryGetValue(PrincipalItemKey, out var value)
+            && value is ApiClientPrincipal typed)
+        {
+            principal = typed;
+            return true;
+        }
+        principal = null;
+        return false;
+    }
+
+    internal static bool IsAuthenticationDisabled(ApiClientPrincipal principal)
+    {
+        ArgumentNullException.ThrowIfNull(principal);
+        return string.Equals(principal.Name, AuthenticationDisabledClientName, StringComparison.Ordinal);
+    }
+
     internal static InitiatorResolution ResolveInitiator(
         HttpContext context,
         WorkInitiator? delegated)
@@ -175,12 +195,27 @@ private static bool TryAuthenticate(
             if (!ConstantTimeEquals(presented, client.Token))
                 continue;
             principal = new ApiClientPrincipal(
-                client.Name, client.FixedInitiator, client.CanDelegateInitiator);
+                client.Name, client.FixedInitiator, client.CanDelegateInitiator, client.ExecutorHostId);
             return true;
         }
         return false;
     }
 
+    private static string? NormalizeExecutorHostId(ApiClientOptions client)
+    {
+        if (string.IsNullOrWhiteSpace(client.ExecutorHostId))
+            return null;
+        var trimmed = client.ExecutorHostId.Trim();
+        if (trimmed.Length > ExecutorRegistration.MaxHostIdLength)
+            throw new InvalidOperationException(
+                $"CodeyBox:ApiClients entry '{client.Name}': ExecutorHostId must be at most " +
+                $"{ExecutorRegistration.MaxHostIdLength} characters.");
+        if (trimmed.Any(char.IsControl))
+            throw new InvalidOperationException(
+                $"CodeyBox:ApiClients entry '{client.Name}': ExecutorHostId must not contain control characters.");
+        return trimmed;
+    }
+
     private static void ValidateInitiator(WorkInitiator initiator)
     {
         ValidateIdentityPart(initiator.Issuer, nameof(initiator.Issuer), 200);
@@ -205,6 +240,14 @@ private static void ValidateIdentityPart(string value, string name, int maximumL
     }
 
     internal const string PrincipalItemKey = "CodeyBox.ApiClientPrincipal";
+
+    /// <summary>
+    /// Client name assigned to requests served while authentication is
+    /// disabled (<c>CodeyBox:DangerouslyDisableAuth=true</c>, loopback dev
+    /// only). Such callers carry no token and therefore no executor binding;
+    /// host-scoped endpoints treat them as the local operator.
+    /// </summary>
+    internal const string AuthenticationDisabledClientName = "authentication-disabled";
     private static readonly WorkInitiator OperatorInitiator = new()
     {
         Issuer = "codeybox",
@@ -240,12 +283,14 @@ internal sealed record ResolvedApiClient(
     string Name,
     string Token,
     WorkInitiator FixedInitiator,
-    bool CanDelegateInitiator);
+    bool CanDelegateInitiator,
+    string? ExecutorHostId);
 
 internal sealed record ApiClientPrincipal(
     string Name,
     WorkInitiator FixedInitiator,
-    bool CanDelegateInitiator);
+    bool CanDelegateInitiator,
+    string? ExecutorHostId = null);
 
 internal sealed record InitiatorResolution(WorkInitiator? Value, IResult? Error);
 
@@ -255,6 +300,17 @@ public sealed class ApiClientOptions
     public string TokenEnvVar { get; set; } = string.Empty;
     public WorkInitiator? Principal { get; set; }
     public bool CanDelegateInitiator { get; set; }
+
+    /// <summary>
+    /// Optional executor host this client's token is bound to. When set, the
+    /// token may only act as that host on host-scoped executor endpoints
+    /// (notably <c>POST /executors/{hostId}/quota-reports</c>): a path host
+    /// that does not exactly equal this value is rejected, so one executor
+    /// cannot forge another host's quota meter. Tokens without a binding
+    /// (including the operator key) are rejected on quota-report ingress —
+    /// a shared bearer proves nothing about which host is calling.
+    /// </summary>
+    public string? ExecutorHostId { get; set; }
 }
 
 /// <summary>