fix(service-messaging,objectql): stop serving sys_http_delivery.headers_json on the generic read path — internal: true + a batch privileged accessor (#8118)

.changeset/delivery-headers-privileged-read.md

@@ -0,0 +1,15 @@
+---
+'@objectstack/objectql': minor
+'@objectstack/service-messaging': patch
+---
+
+Stop serving webhook/flow callout credentials through the generic data-API read of `sys_http_delivery` (#8118).
+
+**What this closes.** `sys_http_delivery.headers_json` — the authored request-header map, the ordinary place an `Authorization: Bearer …` goes — was readable by every caller the data API admits (list, get, an explicit `?select=headers_json`). The column is now declared `internal: true`, so the engine omits it from every generic read with no system carve-out (#7728). The redaction sits at the row layer, so it covers the whole delivery population: `source: 'webhook'` rows (WebhookSchema-authored headers) and `source: 'flow'` rows (per-run interpolated headers that never pass through WebhookSchema at all). Deliveries are unaffected on the wire: the dispatcher's claim path recovers the map through the engine's privileged accessor and still sends every authored header verbatim — fail-closed, a delivery never goes out missing a header, and one that cannot recover its headers refuses loudly instead of going out incomplete. `IHttpOutbox.list()` and `redeliver()` now return the redacted view (`headers: undefined`) under a redacting engine; `claim()` results carry the map verbatim.
+
+**New public API.** `ObjectQL.resolveInternalField(object, recordIds, field)` — the purpose-built privileged accessor #7728 itself named as the remedy for a legitimate system reader of an `internal: true` field: a batch, driver-level read of one flagged field, refusing (ADR-0112 `INVALID_FIELD`, status 400) any field not so declared. The sibling of `resolveSecretField`, batch-shaped because its consumer claims a batch per dispatcher tick.
+
+**What this deliberately does NOT close.**
+
+- The delivery row still holds the header map in cleartext at rest until the 30d telemetry retention ages it out. Encrypting it (`Field.secret()`) was measured and rejected on #8118: one orphan `sys_secret` row per delivery with no cascade or retention, a boot-window fail-open on the fire-and-forget enqueue, and a per-row decrypt on every dispatcher tick.
+- `sys_email.headers_json` (#7986 ①-f) has the same shape; it follows this card's decision but is not part of this change.

content/docs/automation/webhooks.mdx

@@ -141,7 +141,11 @@ at enqueue and replayed byte-for-byte by retries and redelivery, so the HMAC has
 exactly one correct value: the outbox computes it once at enqueue from the
 subscriber's `secret` and stores only the result — the same `sha256=<hex>` the
 receiver is handed on the wire. Reading a delivery row tells you what was sent,
-not how to forge it.
+not how to forge it. The same rule covers the authored headers: `headers_json`
+is the ordinary place a credential goes (`Authorization: Bearer …`), so the
+column is `internal` — omitted from every generic read, with no system
+carve-out — and only the dispatcher recovers it, through the engine's
+privileged accessor, when it claims the row for sending.
 
 | Field | Type | Notes |
 |-------------------|----------|-----------------------------------------------------------------------------|
@@ -152,7 +156,7 @@ not how to forge it.
 | `label` | text | Diagnostic label / event type — surfaced on `X-Objectstack-Event`. |
 | `url` | text | Target URL, snapshotted at enqueue so config edits do not rewrite live rows. |
 | `method` | text | HTTP method. |
-| `headers_json` | textarea | Custom headers, serialised.  |
+| `headers_json` | textarea | Custom headers, serialised. `internal` — never returned on the generic data path (list, get, an explicit `?select=`); recovered only by the dispatcher's privileged read at claim time. |
 | `signature` | text | The `X-Objectstack-Signature` value sent with this delivery, `sha256=<hex>` — computed at enqueue; the secret is not stored (see §6). |
 | `timeout_ms` | number | Per-attempt timeout. |
 | `payload_json` | textarea | The full payload that will be POSTed. |

packages/objectql/src/engine.ts

@@ -5158,6 +5158,95 @@ export class ObjectQL implements IObjectQLEngine {
  return this.resolveSecret(row[field], opts);
  }
 
+ /**
+ * Privileged: recover the stored values of ONE `internal: true` field for a
+ * batch of rows, keyed by record id.
+ *
+ * [#8118] {@link omitInternalFields} deletes a flagged field from every row
+ * the engine hands back — with NO system carve-out, by explicit design
+ * (#7728): an escape hatch nobody needs is a hole in a non-exposure
+ * guarantee. The same ruling names the shape a legitimate system reader uses
+ * when one finally appears: it reads the column through a purpose-built
+ * privileged accessor, the way {@link resolveSecret} does for `secret`. This
+ * method is that accessor. Its first consumer is the outbound-HTTP
+ * dispatcher's claim path (`SqlHttpOutbox.claim()` in
+ * `@objectstack/service-messaging`): `sys_http_delivery.headers_json` — the
+ * authored header map, the ordinary place an `Authorization: Bearer …` goes —
+ * is flagged `internal` so the generic data API returns rows without it,
+ * while the dispatcher must hand exactly that map to the wire VERBATIM (a
+ * delivery that goes out missing a header is not self-announcing: against an
+ * endpoint that does not require it, the delivery succeeds while silently
+ * deviating from the authored configuration).
+ *
+ * Batch-shaped, deliberately, where {@link resolveSecretField} is
+ * single-record: its consumer claims up to a full batch per dispatcher tick,
+ * and #8118's triage rejected the `Field.secret()` route partly BECAUSE that
+ * shape costs a driver read plus a decrypt per row per tick. One driver read
+ * serves the whole claim batch; a single record is the batch of one. This is
+ * the sibling of {@link resolveSecretField}'s pattern — guard first, then a
+ * driver-level read — not a second divergent privileged-read path.
+ *
+ * The rows are read at DRIVER level on purpose — the only layer where the
+ * value still exists — so this bypasses read hooks, field-level security and
+ * sharing: the same trust {@link resolveSecret} and {@link resolveSecretField}
+ * place in their callers, and the same reason all three are explicit,
+ * separately-named privileged verbs rather than an option on `find`. No
+ * query string reaches this, so it cannot be turned on from outside the
+ * process.
+ *
+ * Refuses (ADR-0112 `code` + `status`) any field not declared
+ * `internal: true` on the object. Without that guard this would be a generic
+ * read-protection bypass — over `password` plaintext in particular, which is
+ * masked deliberately (ADR-0100) — rather than the internal channel's
+ * dereference. A `secret`-typed field is likewise refused unless it is also
+ * flagged, and even then this returns the stored `secret:<id>` ref, never a
+ * plaintext: decryption stays with {@link resolveSecretField}.
+ *
+ * Returns the stored value per id — `null` when the column is unset. An id
+ * whose row does not exist is absent from the map; what a missing row means
+ * belongs to the caller (for the dispatcher: a row deleted mid-claim). No
+ * decrypt is involved: `internal` is a read-side omission flag, not an
+ * encrypted channel — the at-rest story is the object's own (for
+ * `sys_http_delivery`, 30d telemetry retention; encrypting the delivery row
+ * was measured and rejected on #8118).
+ */
+ async resolveInternalField(
+ object: string,
+ recordIds: readonly string[],
+ field: string,
+ ): Promise<Map<string, unknown>> {
+ const schema = this._registry.getObject(object);
+ if (!collectInternalReadFields(schema).includes(field)) {
+ const err: Error & { code?: string; status?: number; object?: string; field?: string } =
+ new Error(
+ `Cannot resolve internal field "${object}.${field}": it is not declared \`internal: true\`. `
+ + 'Only fields the engine omits from the generic read path are dereferenceable here — '
+ + 'anything else either comes back on find/findOne already, or is protected by its own '
+ + 'channel (`secret` refs via resolveSecretField; `password` is masked deliberately, '
+ + 'ADR-0100, so dereferencing one here would be a mask bypass).',
+ );
+ err.code = 'INVALID_FIELD';
+ err.status = 400;
+ err.object = object;
+ err.field = field;
+ throw err;
+ }
+ const out = new Map<string, unknown>();
+ if (recordIds.length === 0) return out;
+ const driver = this.getDriver(object);
+ const found = await driver.find(object, {
+ where: { id: { $in: [...recordIds] } },
+ fields: ['id', field],
+ });
+ for (const row of Array.isArray(found) ? found : [found]) {
+ if (!row || typeof row !== 'object') continue;
+ const id = (row as Record<string, unknown>).id;
+ if (typeof id !== 'string' && typeof id !== 'number') continue;
+ out.set(String(id), (row as Record<string, unknown>)[field] ?? null);
+ }
+ return out;
+ }
+
  /**
  * Helper to get object definition
  */

packages/objectql/src/internal-fields.test.ts

@@ -43,6 +43,12 @@ function makeStubDriver() {
  if (!where || typeof where !== 'object') return true;
  for (const [k, v] of Object.entries(where)) {
  if (k.startsWith('$')) continue;
+ // `$in` — the batch shape `resolveInternalField` reads by (#8118).
+ if (v && typeof v === 'object' && '$in' in (v as any)) {
+ const members = (v as any).$in;
+ if (!Array.isArray(members) || !members.includes(row[k] ?? null)) return false;
+ continue;
+ }
  const expected = (v && typeof v === 'object' && '$eq' in (v as any)) ? (v as any).$eq : v;
  if ((row[k] ?? null) !== (expected ?? null)) return false;
  }
@@ -421,4 +427,83 @@ describe('#7728: the `internal` field flag omits a value from the generic data p
  expect(err!.message.match(/itest_api_key\.key/g)).toHaveLength(1);
  });
  });
+
+ /**
+ * [#8118] `resolveInternalField` — the purpose-built privileged accessor
+ * #7728 itself named as the shape a legitimate system reader uses ("it reads
+ * the column through a purpose-built privileged accessor, the way
+ * `resolveSecret` does"). The omit above has NO system carve-out, so this is
+ * the ONLY supported door to a flagged value; its first consumer is the
+ * outbound-HTTP dispatcher's claim path, which must put
+ * `sys_http_delivery.headers_json` on the wire verbatim while the data API
+ * returns rows without it.
+ *
+ * Batch-shaped (ids in, `Map` out) because that consumer claims a batch per
+ * dispatcher tick — #8118's triage rejected `Field.secret()` partly for
+ * costing a per-row read on that tick, and the accessor must not re-acquire
+ * the rejected cost.
+ */
+ describe('#8118: resolveInternalField — the privileged dereference', () => {
+ it('resolves the flagged field for a batch of ids, straight from storage', async () => {
+ const a = await seed();
+ const b = await ctx.engine.insert('itest_api_key', {
+ name: 'k2', prefix: 'svc_', revoked: false, key: `${HASH}-2`,
+ }, { context: { isSystem: true } } as any);
+
+ const resolved = await ctx.engine.resolveInternalField('itest_api_key', [a.id, b.id], 'key');
+ expect(resolved.get(a.id)).toBe(HASH);
+ expect(resolved.get(b.id)).toBe(`${HASH}-2`);
+ expect(resolved.size).toBe(2);
+
+ // …while the generic read path, asked in the same breath, still omits —
+ // the accessor is a second DOOR, not a hole in the first one.
+ const viaFind = (await ctx.engine.find('itest_api_key', { where: { id: a.id } }))[0] as any;
+ expect(Object.keys(viaFind)).not.toContain('key');
+ });
+
+ it('an unset value resolves to null; a missing row is absent from the map', async () => {
+ const a = await ctx.engine.insert('itest_api_key', {
+ name: 'k-unset', prefix: 'osk_', revoked: false, key: null,
+ }, { context: { isSystem: true } } as any);
+
+ const resolved = await ctx.engine.resolveInternalField(
+ 'itest_api_key', [a.id, 'r_does_not_exist'], 'key',
+ );
+ // Unset ≠ missing: the caller can tell "row exists, nothing stored"
+ // (null) from "no such row" (absent) — the dispatcher treats the latter
+ // as a row deleted mid-claim.
+ expect(resolved.has(a.id)).toBe(true);
+ expect(resolved.get(a.id)).toBeNull();
+ expect(resolved.has('r_does_not_exist')).toBe(false);
+ });
+
+ it('an empty batch resolves to an empty map without touching the driver', async () => {
+ const resolved = await ctx.engine.resolveInternalField('itest_api_key', [], 'key');
+ expect(resolved.size).toBe(0);
+ });
+
+ it('refuses a field not declared `internal: true` — ADR-0112 code AND status', async () => {
+ const created = await seed();
+ // `prefix` comes back on every find — dereferencing it here is not a
+ // privilege, and an accessor that allowed it would be a generic
+ // read-protection bypass one field-name away from `password`.
+ const err = await ctx.engine.resolveInternalField('itest_api_key', [created.id], 'prefix').then(
+ () => null,
+ (e: unknown) => e as Error & { code?: string; status?: number; field?: string },
+ );
+ expect(err).toBeInstanceOf(Error);
+ expect(err!.code).toBe('INVALID_FIELD');
+ expect(err!.status).toBe(400);
+ expect(err!.field).toBe('prefix');
+ expect(err!.message).toContain('itest_api_key.prefix');
+ });
+
+ it('refuses on an object with no flagged fields at all (guard before fast path)', async () => {
+ // The guard outranks the empty-ids fast path on purpose: a caller that
+ // wired the wrong object name hears about it deterministically, not only
+ // on the first non-empty batch.
+ await expect(ctx.engine.resolveInternalField('itest_plain', [], 'key'))
+ .rejects.toMatchObject({ code: 'INVALID_FIELD', status: 400 });
+ });
+ });
 });