diff --git a/.changeset/secret-orphans-command.md b/.changeset/secret-orphans-command.md new file mode 100644 index 0000000000..52f87692dd --- /dev/null +++ b/.changeset/secret-orphans-command.md @@ -0,0 +1,15 @@ +--- +"@objectstack/cli": minor +--- + +New operator-run command `os secret orphans`: reports the `sys_secret` rows no producer references any more, and — only behind `--delete` — removes the ones the maintainer ruling permits. **Report-only by default: without `--delete` it writes nothing and deletes nothing.** Flags: `--delete`, `--export `, `--declared-datasources `, `--no-declared-datasources`, `--yes`/`-y`, `--database-url`, `--json`. + +It closes the residue #8030 / PR #8063 deliberately left open: that fix reaps forward-only, on the write that retires a handle, so rows orphaned by rotations that already happened on a deployed instance are untouched and nothing else ever touches them. ⛔ Not a boot-time migration and not a `lifecycle`/retention policy — nothing on any boot path reaches this command; age is not unreferencedness, and an age sweep takes the in-force oldest rows first. + +**The deletion predicate is "attributable AND unreferenced by the COMPLETE cross-producer reference union", and incompleteness refuses.** `sys_secret` has three producers and no producer column, so "unreferenced by `sys_setting`" is not "unreferenced": a live, engine-owned credential lands in the settings-scoped classifier's `orphaned` bucket, which is reproduced against the real producers in the new test rather than argued. The command therefore decides on the union of all three holder families and **refuses to delete whenever any family could not be enumerated, naming that family** — the union's three independent gap sources (the host did not declare its code-defined datasources; the engine exposes no `listDatasourceDefs()`; neither reaches) are carried through per family to the operator, to `--json` and into the export, never flattened to a boolean. Saying nothing about declared datasources is a gap, not `[]`: `--no-declared-datasources` is how a host states it has none. + +Three further refusals are mechanical rather than advisory: `unattributable` rows are never deletable; a row whose `(namespace, key)` currently resolves through a **legacy inline** `sys_setting` value is withheld (the #8063 prefix guard in the opposite direction — an inline value names no handle, so it explains the absence); and re-wrap evidence (`version` / `rotated_at`) is reported but is **never** a verdict input, because `rotateKey()` keeps the handle stable and a re-wrap is not a retirement. + +**The pre-delete export is mandatory** and carries the cipher material of every row about to be deleted: the settings audit trail records content digests rather than handles, so without it an erroneous delete could be neither named nor undone. It is CREATED owner-only or not written at all — the write is an exclusive create, so a path that filled up between the early `--export` refusal and the write itself fails rather than truncating a file whose owner and permissions it would then have inherited — and it is read back and checked against the plan before a single row is removed. + +⚠️ The operator-facing text states the measured framing, which is inverted: on the pre-fix rotation path the handle was never repointed, so the value **still in force** is the oldest one — the credential the administrator believed they had replaced — while each orphan holds a value that never took effect. Deleting orphans therefore retires nothing that is exposed, and if the administrator also rotated at the provider the newest orphan may be a credential that is currently valid there. diff --git a/content/docs/deployment/cli.mdx b/content/docs/deployment/cli.mdx index c9dc4dba1a..5f1a5b04d2 100644 --- a/content/docs/deployment/cli.mdx +++ b/content/docs/deployment/cli.mdx @@ -371,6 +371,55 @@ The two commands ultimately go through the same kernel — they just differ in w shape they accept. See [Source vs Artifact](#source-vs-artifact) below. +#### `os secret orphans` + +Reports the `sys_secret` rows no producer references any more, and — only behind +`--delete` — removes the ones it can prove are the settings subsystem's to remove. +**Report-only by default: without `--delete` it writes nothing and deletes nothing.** +It is never run for you; nothing on any boot or upgrade path invokes it. + +```bash +os secret orphans # report (writes nothing) +os secret orphans --json # the same report, machine-readable +os secret orphans --no-declared-datasources # state that this host declares none +os secret orphans --delete --export ./secrets-backup.json --no-declared-datasources +``` + +**Options:** +- `--delete` — remove the deletable rows (default off) +- `--export ` — **mandatory with `--delete`**; refuses to overwrite an existing file +- `--declared-datasources ` — JSON file (an array, or `{"datasources": [...]}`) of the datasource artefacts this host declares in code +- `--no-declared-datasources` — state that this host declares none +- `-y, --yes` — skip the confirmation prompt +- `--database-url `, `--json` + +A rotation performed before the settings subsystem learned to reap left the previous +ciphertext behind, and the reaping that fixed it only fires going forward. This command +is how an operator clears that residue deliberately. + +**A row is removed only when it is attributable to a declared encrypted settings +specifier *and* named by no family of the complete reference union.** `sys_secret` is +written by three producers and carries no producer column, so "unreferenced by +`sys_setting`" is not "unreferenced" — a live credential belonging to an object's +`secret` field or to a datasource can share a settings specifier's `(namespace, key)`. +The command therefore reads all three holder families, and **refuses to delete whenever +any of them could not be enumerated, naming the family**. A host that says nothing about +its code-declared datasources leaves that family a gap; `--no-declared-datasources` is +how you state there are none. Rows nothing attributes are never deleted, nor are rows +whose setting still resolves through a legacy inline value, nor rows that were merely +re-wrapped in place (a re-wrap keeps the handle and is not a retirement). + + +**This does not retire an exposed credential, and the export is not optional.** On the +pre-fix rotation path the handle was never repointed, so the value **still in force** is +the *oldest* one — the credential the administrator believed they had replaced — while +each orphan holds a value that never took effect. If the administrator also rotated at +the provider, the newest orphan may be a credential that is still valid there. The audit +trail records content digests rather than handles, so a row deleted in error cannot be +named afterwards: the export carries the cipher material, is written owner-only, and is +read back and checked before any row is removed. Keep it until you are certain. + + ### Build & Validate | Command | Description | diff --git a/packages/cli/src/commands/secret/orphans.guards.test.ts b/packages/cli/src/commands/secret/orphans.guards.test.ts new file mode 100644 index 0000000000..210ce0127f --- /dev/null +++ b/packages/cli/src/commands/secret/orphans.guards.test.ts @@ -0,0 +1,273 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * #8103 — the command-level guards that decide whether a delete may run at + * all, tested away from the boot they normally sit behind. + * + * The first two exist to stop an ANSWER being invented: + * + * - `readDeclaredDatasources` must never turn an unreadable file into `[]`. + * `[]` is the host stating it has no code-declared datasources; a missing or + * malformed file is nobody having answered, and the union turns that into a + * declared gap that refuses the delete. Collapsing the two would make the + * union look complete while being quieter than it is. + * - `asDeletingDriver` refuses a driver that cannot delete instead of casting + * the union's READ-ONLY driver port into a writing one. The cast compiles + * and then throws partway through the delete loop, after the export has been + * written and some rows are already gone. + * + * The third is about the export WRITE itself. `--export` refuses a path that + * already exists, but that check runs before the boot, the scan and the + * confirmation — an arbitrarily long window before the bytes are written. The + * write is therefore an EXCLUSIVE CREATE (`flag: 'wx'`), so a path that filled + * up inside the window fails instead of truncating a file it does not own; the + * `mode: 0o600` that keeps cipher material owner-only is applied by the OS only + * at CREATION, so a truncating write would have inherited the squatter's owner + * and permissions and put cipher material inside them. + */ + +import { describe, it, expect, afterEach, vi } from 'vitest'; +import { existsSync, mkdtempSync, readFileSync, rmSync, statSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { dirname, join, resolve } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import SecretOrphans, { asDeletingDriver, readDeclaredDatasources } from './orphans.js'; +import { bootSchemaStack } from '../../utils/schema-migrate.js'; +import { collectSecretReferenceUnion } from '../../utils/secret-reference-union.js'; +import type { SecretReferenceUnion } from '../../utils/secret-reference-union.js'; + +// The command's own control flow is the subject here, so only the two seams +// that would boot a database or walk a real engine are replaced. Everything +// that decides WHICH rows are deletable — `planSysSecretOrphanSweep`, +// `buildPreDeleteExport`, `collectEncryptedSpecifierRefs`, `isSecretHandle` — +// runs for real, from the same modules the command reaches at run time. +vi.mock('../../utils/schema-migrate.js', () => ({ bootSchemaStack: vi.fn() })); +vi.mock('../../utils/secret-reference-union.js', () => ({ collectSecretReferenceUnion: vi.fn() })); +// Constructed and handed to the (mocked) boot, never used — a stub keeps the +// platform-objects graph out of this file's import cost. +vi.mock('@objectstack/platform-objects/plugin', () => ({ PlatformObjectsPlugin: class {} })); + +const HERE = dirname(fileURLToPath(import.meta.url)); +const CLI_ROOT = resolve(HERE, '..', '..', '..'); + +const dirs: string[] = []; +const tempDir = () => { + const dir = mkdtempSync(join(tmpdir(), 'os-8103-')); + dirs.push(dir); + return dir; +}; +afterEach(() => { + for (const dir of dirs.splice(0)) rmSync(dir, { recursive: true, force: true }); +}); + +describe('readDeclaredDatasources — an unreadable answer is never an empty one', () => { + it('reads a bare array', () => { + const dir = tempDir(); + const file = join(dir, 'ds.json'); + writeFileSync(file, JSON.stringify([{ name: 'main', external: { credentialsRef: 'sys_secret:sec_1' } }])); + expect(readDeclaredDatasources(file)).toEqual([ + { name: 'main', external: { credentialsRef: 'sys_secret:sec_1' } }, + ]); + }); + + it('reads the { datasources: [...] } wrapper', () => { + const dir = tempDir(); + const file = join(dir, 'ds.json'); + writeFileSync(file, JSON.stringify({ datasources: [{ name: 'analytics' }] })); + expect(readDeclaredDatasources(file)).toEqual([{ name: 'analytics' }]); + }); + + it('an EMPTY array is a real answer and stays one', () => { + const dir = tempDir(); + const file = join(dir, 'ds.json'); + writeFileSync(file, '[]'); + // The control that makes this assertion mean something: the same function + // over a NON-empty file returns a non-empty list, so `[]` here is the + // file's content and not a swallowed failure. + expect(readDeclaredDatasources(file)).toEqual([]); + const other = join(dir, 'other.json'); + writeFileSync(other, JSON.stringify([{ name: 'x' }])); + expect(readDeclaredDatasources(other)).toHaveLength(1); + }); + + it('throws — never returns [] — for a missing file', () => { + expect(() => readDeclaredDatasources(join(tempDir(), 'absent.json'))).toThrow(/no such file/); + }); + + it('throws — never returns [] — for malformed JSON', () => { + const dir = tempDir(); + const file = join(dir, 'ds.json'); + writeFileSync(file, '{ not json'); + expect(() => readDeclaredDatasources(file)).toThrow(/does not parse as JSON/); + }); + + it('throws — never returns [] — for JSON that is not a datasource list', () => { + const dir = tempDir(); + const file = join(dir, 'ds.json'); + writeFileSync(file, JSON.stringify({ nope: true })); + expect(() => readDeclaredDatasources(file)).toThrow(/must hold an array/); + }); +}); + +describe('asDeletingDriver — a missing delete() is a refusal, not a cast', () => { + it('accepts a driver that declares delete()', () => { + const driver = { async find() { return []; }, async delete() { return true; } }; + expect(asDeletingDriver(driver)).toBe(driver); + }); + + it('refuses the union READ-ONLY port shape, which declares only find()', () => { + // Exactly the shape `SecretReferenceDriverLike` describes. Positive + // control: the accepting case above proves the predicate can say yes. + expect(asDeletingDriver({ async find() { return []; } })).toBeNull(); + }); + + it('refuses undefined and a non-callable delete', () => { + expect(asDeletingDriver(undefined)).toBeNull(); + expect(asDeletingDriver(null)).toBeNull(); + expect(asDeletingDriver({ delete: 'yes' })).toBeNull(); + }); +}); + +// ─────────────────────────────────────────────────────────────────────────── +// The export write: created, or not written at all +// ─────────────────────────────────────────────────────────────────────────── + +/** + * One `sys_secret` row that the real sweep will judge deletable: its + * `(namespace, key)` matches a declared ENCRYPTED settings specifier below + * (attributable) and the union names no handle at all (unreferenced). + */ +const ORPHAN_ROW = { + id: 'sec_orphan_1', + namespace: 'smtp', + key: 'retired_token', + alg: 'aes-256-gcm', + version: 2, + kms_key_id: 'kms_local', + ciphertext: 'ENC(v1:retired-token-cipher-material)', + created_at: '2026-01-01T00:00:00.000Z', + rotated_at: '2026-06-01T00:00:00.000Z', +}; + +/** What `settings.listManifests()` answers — the attribution half of the predicate. */ +const SETTINGS_MANIFESTS = [ + { namespace: 'smtp', specifiers: [{ key: 'retired_token', type: 'string', encrypted: true }] }, +]; + +/** A COMPLETE union that names no handle: the row really is unreferenced. */ +const unreferencingCompleteUnion = (): SecretReferenceUnion => ({ + handleIds: new Set(), + references: [], + families: { + 'settings': { family: 'settings', status: 'enumerated', references: [] }, + 'object-field': { family: 'object-field', status: 'enumerated', references: [] }, + 'datasource': { family: 'datasource', status: 'enumerated', references: [] }, + }, + complete: true, + gaps: [], +}); + +/** + * Drive one real `--delete --json` run against the fake seams. + * + * `duringBoot` runs inside the mocked boot, which is the window this file is + * about: after the `existsSync` pre-check, before the export is written. + */ +async function runDelete( + exportPath: string, + opts: { duringBoot?: () => void } = {}, +): Promise<{ removed: string[]; payload: Record }> { + const removed: string[] = []; + + vi.mocked(collectSecretReferenceUnion).mockResolvedValue(unreferencingCompleteUnion()); + vi.mocked(bootSchemaStack).mockImplementation(async () => { + opts.duringBoot?.(); + const objectql = { + getConfigs: () => ({}), + getDriverForObject: (object: string) => + object === 'sys_secret' + ? { + async find() { return [{ ...ORPHAN_ROW }]; }, + async delete(_object: string, id: string) { removed.push(id); return true; }, + } + : { async find() { return []; } }, + }; + return { + kernel: { + getService: (name: string) => + name === 'objectql' ? objectql + : name === 'settings' ? { listManifests: () => SETTINGS_MANIFESTS } + : undefined, + }, + shutdown: async () => { /* nothing was booted */ }, + } as never; + }); + + const chunks: string[] = []; + const stdout = vi.spyOn(process.stdout, 'write').mockImplementation( + ((chunk: unknown, ...rest: unknown[]) => { + chunks.push(String(chunk)); + const done = rest.find((a) => typeof a === 'function') as ((e?: Error | null) => void) | undefined; + done?.(null); + return true; + }) as never, + ); + // `emitJson` sets `process.exitCode` on a refusal, and this process is the + // test runner's — leaving it set would fail the whole suite from the outside. + const savedExitCode = process.exitCode; + try { + await SecretOrphans.run( + ['--delete', '--yes', '--json', '--no-declared-datasources', '--export', exportPath], + { root: CLI_ROOT }, + ); + } finally { + stdout.mockRestore(); + process.exitCode = savedExitCode; + } + + const lines = chunks.join('').split('\n').filter((l) => l.trim() !== ''); + return { removed, payload: JSON.parse(lines[lines.length - 1]) as Record }; +} + +describe('the pre-delete export is created exclusively, never written onto', () => { + it('a path that filled up inside the window is refused, and every row survives', async () => { + const file = join(tempDir(), 'export.json'); + // Distinct from anything the export can contain, in both directions: this + // string is not a substring of the export document and the export document + // is not a substring of it, so neither assertion below can pass by accident. + const squatterBytes = 'SQUATTER-PAYLOAD not-an-export-document\n'; + + const { removed, payload } = await runDelete(file, { + duringBoot: () => writeFileSync(file, squatterBytes), + }); + + // The harm first, so an ablation of the flag reports the harm rather than + // the envelope: no row is removed… + expect(removed).toEqual([]); + // …and the file that was there first is byte-for-byte intact. + expect(readFileSync(file, 'utf8')).toBe(squatterBytes); + // Then the envelope: it lands in the export_failed refusal that already + // existed — no new branch, no new error code. + expect(payload.error).toBe('export_failed'); + expect(String(payload.message)).toContain(file); + // oclif builds its whole command table on the first `run()` in a process. + }, 60_000); + + it('POSITIVE CONTROL — over an unclaimed destination the run does remove rows', async () => { + // Without this, `removed` being empty above would also be satisfied by a + // command that can never delete anything at all. + const file = join(tempDir(), 'export.json'); + expect(existsSync(file)).toBe(false); + + const { removed, payload } = await runDelete(file); + + expect(payload.error).toBeUndefined(); + expect(removed).toEqual([ORPHAN_ROW.id]); + const doc = JSON.parse(readFileSync(file, 'utf8')) as { rows: Array> }; + expect(doc.rows.map((r) => r.id)).toEqual([ORPHAN_ROW.id]); + expect(doc.rows[0].ciphertext).toBe(ORPHAN_ROW.ciphertext); + // `mode` survives the flag change: it is applied at creation, which is now + // the only way this file is ever opened. + expect(statSync(file).mode & 0o777).toBe(0o600); + }, 60_000); +}); diff --git a/packages/cli/src/commands/secret/orphans.ts b/packages/cli/src/commands/secret/orphans.ts new file mode 100644 index 0000000000..57283a482b --- /dev/null +++ b/packages/cli/src/commands/secret/orphans.ts @@ -0,0 +1,527 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +import { Command, Flags } from '@oclif/core'; +import { createInterface } from 'node:readline'; +import { existsSync, readFileSync, writeFileSync } from 'node:fs'; +import { resolve as resolvePath } from 'node:path'; +import chalk from 'chalk'; +import { + printHeader, + printSuccess, + printWarning, + printError, + printInfo, + printStep, + createTimer, + emitJson, +} from '../../utils/format.js'; +import { bootSchemaStack } from '../../utils/schema-migrate.js'; +import type { + DatasourceArtefactLike, + SecretReferenceEngineLike, +} from '../../utils/secret-reference-union.js'; +import type { + PreDeleteExport, + SecretRowSnapshot, + SettingRowSnapshot, + SysSecretSweepPlan, +} from '../../utils/sys-secret-orphan-sweep.js'; + +/** + * `os secret orphans` — the operator-run half of #8103. + * + * ## What it is, and what the maintainer ruled + * + * #8030 / PR #8063 made a settings rotation reap the ciphertext it retired, + * but only FORWARD: the reaping fires on the write that repoints a handle. + * Rows orphaned by rotations that already happened on a deployed instance are + * untouched, and nothing else ever touches them. + * + * Removing those is a destructive, irreversible delete-many over stored + * credentials, so the vehicle was a maintainer decision, taken twice: + * + * - 2026-08-26, option **B** — an EXPLICIT OPERATOR COMMAND with the dry-run + * report as its default, deletion only behind an explicit flag, a mandatory + * pre-delete export, and the `unattributable` class never deleted. + * ⛔ Option C, an automatic migration that sweeps on boot or upgrade, was + * ruled out on measured grounds. Nothing on any boot path invokes this + * command, and it must not grow such a caller. + * - 2026-08-27, option **B'** — B, but SEQUENCED behind its missing + * precondition. The deletion predicate is *"attributable AND unreferenced + * by the COMPLETE union"*, over all three `sys_secret` producer families. + * Report-only remains the recorded fallback, and ⛔ no seat downgrades to it + * silently. + * + * ⛔ It is also NOT a `lifecycle`/retention implementation. Age is not + * unreferencedness, and an age sweep takes the in-force oldest rows FIRST — + * see the exposure note below, which is why that inversion is fatal here. + * + * ## ⚠️ The exposure framing is INVERTED, and the operator text says so + * + * The tempting sentence — "this cleans up leaked old credentials" — is FALSE + * for this population, and this command never says it. On the pre-fix rotation + * path the handle was never repointed, so the value STILL IN FORCE is the + * OLDEST one — the credential the administrator believed they had replaced — + * while each orphan holds a value the administrator INTENDED to set and which + * never took effect. Deleting orphans therefore retires nothing that is + * exposed, and if the administrator also rotated at the provider, the newest + * orphan may be a credential that is CURRENTLY VALID there. + * + * ## The safety property is the completeness of the union, and it is checked + * + * `sys_secret` has three producers and no producer column, so "unreferenced by + * `sys_setting`" is not "unreferenced" — a LIVE, engine-owned credential lands + * in the settings-scoped classifier's `orphaned` bucket, reproduced against the + * real producers in `sys-secret-orphan-sweep.test.ts`. This command therefore + * decides on the cross-producer union, and **refuses to delete when any family + * could not be enumerated, naming the family**. The union models three + * independent gap sources and ⛔ this command flattens none of them: the + * per-family status is printed and carried into `--json` and into the export. + * + * ## Why the export is mandatory rather than advisable + * + * The settings audit trail records content digests, never handles, so a row + * deleted in error cannot be NAMED afterwards, let alone recovered. The export + * is the only record that survives the delete, and it therefore carries the + * cipher material: an export that named the row without its ciphertext would + * make the mistake describable and still permanent. `--delete` without + * `--export` is refused, the file is written with owner-only permissions, and + * it is READ BACK and checked against the plan before a single row is removed. + */ +export default class SecretOrphans extends Command { + static override description = + 'Report `sys_secret` rows no producer references any more, and (only with --delete) remove the ' + + 'ones attributable to settings. Report-only by default: it writes nothing and deletes nothing.'; + + static override examples = [ + '<%= config.bin %> secret orphans', + '<%= config.bin %> secret orphans --json', + '<%= config.bin %> secret orphans --declared-datasources ./datasources.json', + '<%= config.bin %> secret orphans --no-declared-datasources', + '<%= config.bin %> secret orphans --delete --export ./sys-secret-backup.json --no-declared-datasources', + ]; + + static override flags = { + 'database-url': Flags.string({ + description: 'Database URL to inspect (defaults to $OS_DATABASE_URL / the project DB)', + env: 'OS_DATABASE_URL', + }), + delete: Flags.boolean({ + description: + 'Delete the deletable rows (default is a report that writes nothing). Requires --export, ' + + 'and refuses whenever the reference union is incomplete.', + default: false, + }), + export: Flags.string({ + description: + 'Path for the MANDATORY pre-delete export. Holds the cipher material of every row about ' + + 'to be deleted, so the delete is recoverable — the audit trail records digests, not ' + + 'handles, and cannot name a deleted row afterwards. Refuses to overwrite.', + }), + 'declared-datasources': Flags.string({ + description: + 'Path to a JSON file listing the datasource artefacts this host declares IN CODE (an array, ' + + 'or {"datasources": [...]}). A datasource declared in code that nothing ever installed ' + + 'reaches neither sys_metadata nor the engine index, so only the host can answer for it.', + exclusive: ['no-declared-datasources'], + }), + 'no-declared-datasources': Flags.boolean({ + description: + 'State that this host declares NO code-defined datasources. Distinct from saying nothing: ' + + 'saying nothing leaves the datasource family a declared GAP and deletion is refused.', + default: false, + exclusive: ['declared-datasources'], + }), + yes: Flags.boolean({ char: 'y', description: 'Skip the --delete confirmation prompt', default: false }), + json: Flags.boolean({ + description: 'Output as JSON (implies non-interactive; requires --yes to delete)', + default: false, + }), + }; + + async run(): Promise { + const { flags } = await this.parse(SecretOrphans); + const timer = createTimer(); + const json = flags.json; + + if (!json) printHeader('Secret · orphaned sys_secret rows'); + + // ── The host's own answer for family 3, read BEFORE the boot ─────────── + // A malformed file must not cost a boot, and — more importantly — it must + // never degrade into `[]`. `undefined` here means "nobody answered", which + // the union turns into a declared gap. + let declaredDatasources: readonly DatasourceArtefactLike[] | undefined; + if (flags['no-declared-datasources']) { + declaredDatasources = []; + } else if (flags['declared-datasources']) { + try { + declaredDatasources = readDeclaredDatasources(flags['declared-datasources']); + } catch (error) { + const message = error instanceof Error ? error.message : String(error); + if (json) { await emitJson({ error: 'declared_datasources_unreadable', message }, 1, { compact: true }); return; } + printError(message); + this.exit(1); + return; + } + } + + if (flags.delete && !flags.export) { + const message = + 'Refusing to delete without --export. The pre-delete export is mandatory: the audit trail ' + + 'records content digests rather than handles, so a row deleted in error cannot be named ' + + 'afterwards and cannot be recovered. Name a path for it.'; + if (json) { await emitJson({ error: 'export_required', message }, 1, { compact: true }); return; } + printError(message); + this.exit(1); + return; + } + + const exportPath = flags.export ? resolvePath(flags.export) : undefined; + if (exportPath && existsSync(exportPath)) { + const message = `Refusing to overwrite an existing export at ${exportPath}. Name a new path.`; + if (json) { await emitJson({ error: 'export_exists', message, path: exportPath }, 1, { compact: true }); return; } + printError(message); + this.exit(1); + return; + } + + if (!json) printStep(flags.delete ? 'Booting (DELETE mode)…' : 'Booting (report only)…'); + + // Loaded at the point of use, never at module load: oclif `import()`s every + // command module on every invocation while building its table, and these + // chains reach objectql / service-settings / service-datasource. A static + // import would charge every `os` invocation for them (the #5726 shape). + const { collectSecretReferenceUnion } = await import('../../utils/secret-reference-union.js'); + const { buildPreDeleteExport, planSysSecretOrphanSweep, useHandlePredicate } = + await import('../../utils/sys-secret-orphan-sweep.js'); + const { collectEncryptedSpecifierRefs, isSecretHandle, SettingsServicePlugin } = + await import('@objectstack/service-settings'); + const { PlatformObjectsPlugin } = await import('@objectstack/platform-objects/plugin'); + + // The legacy-inline discriminator comes from the producer that mints the + // handles, never from a restated `sec_` prefix in the sweep module. + useHandlePredicate(isSecretHandle); + + let stack; + try { + stack = await bootSchemaStack({ + jsonOutput: json, + databaseUrl: flags['database-url'], + // Settings is registered so its REGISTERED manifests are readable: the + // attribution set is theirs, and without it nothing is attributable and + // nothing is deletable (the safe direction, reported as a note). + extraPlugins: [new PlatformObjectsPlugin(), new SettingsServicePlugin({ registerRoutes: false })], + }); + } catch (error) { + const message = error instanceof Error ? error.message : String(error); + if (json) { await emitJson({ error: 'boot_failed', message }, 1, { compact: true }); return; } + printError(message); + this.exit(1); + return; + } + + try { + const engine = stack.kernel.getService('objectql') as SecretReferenceEngineLike | undefined; + if (!engine) { + const message = 'No ObjectQL engine on this runtime, so no producer family can be enumerated.'; + if (json) { await emitJson({ error: 'no_engine', message }, 1, { compact: true }); return; } + printError(message); + this.exit(1); + return; + } + + // `sys_secret` is read at DRIVER level and UNSCOPED, the same choice the + // union makes and for the same reason: every filter subtracts rows, and a + // row missing from THIS read is a row the report never mentions. + const secretDriver = engine.getDriverForObject('sys_secret'); + if (!secretDriver) { + const message = 'No driver resolves for `sys_secret`, so its rows could not be read.'; + if (json) { await emitJson({ error: 'no_sys_secret_driver', message }, 1, { compact: true }); return; } + printError(message); + this.exit(1); + return; + } + + const rawSecrets = rowsOf(await secretDriver.find('sys_secret', {})); + const rawById = new Map(rawSecrets.map((r) => [String(r.id), r])); + // ⛔ `ciphertext` is dropped here and not carried into the plan: the plan + // is printed and serialised, and cipher material must not be reachable + // from a surface that is expected to be safe to show. + const secrets: SecretRowSnapshot[] = rawSecrets.map((r) => ({ + id: String(r.id), + namespace: String(r.namespace ?? ''), + key: String(r.key ?? ''), + version: (r.version as number | null | undefined) ?? null, + kms_key_id: (r.kms_key_id as string | null | undefined) ?? null, + created_at: (r.created_at as string | null | undefined) ?? null, + rotated_at: (r.rotated_at as string | null | undefined) ?? null, + })); + + const settingDriver = engine.getDriverForObject('sys_setting'); + const settingRows: SettingRowSnapshot[] = settingDriver + ? rowsOf(await settingDriver.find('sys_setting', {})).map((r) => ({ + namespace: String(r.namespace ?? ''), + key: String(r.key ?? ''), + scope: (r.scope as string | null | undefined) ?? null, + user_id: (r.user_id as string | null | undefined) ?? null, + value_enc: (r.value_enc as string | null | undefined) ?? null, + })) + : []; + + const settings = stack.kernel.getService('settings') as + { listManifests?: () => unknown[] } | undefined; + const manifests = (settings?.listManifests?.() ?? []) as Parameters< + typeof collectEncryptedSpecifierRefs + >[0]; + + const union = await collectSecretReferenceUnion({ engine, declaredDatasources }); + const plan = planSysSecretOrphanSweep({ + secrets, + union, + attributableTo: collectEncryptedSpecifierRefs(manifests), + settingRows, + }); + + if (!flags.delete) { + if (json) { await emitJson({ mode: 'report', plan }, 0, { compact: true }); return; } + renderPlan(plan); + printInfo(`Report only — nothing was written or deleted. (${timer.display()})`); + printInfo('Re-run with --delete --export to remove the deletable rows.'); + return; + } + + // ── The falsifiable criterion: an incomplete union refuses, by family ── + if (plan.refusal) { + if (json) { + await emitJson({ mode: 'delete', refused: plan.refusal, plan }, 1, { compact: true }); + return; + } + renderPlan(plan); + printError(plan.refusal.message); + for (const gap of plan.refusal.gaps) printError(` family '${gap.family}': ${gap.reason}`); + this.exit(1); + return; + } + + if (plan.deletable.length === 0) { + if (json) { await emitJson({ mode: 'delete', deleted: [], plan }, 0, { compact: true }); return; } + renderPlan(plan); + printSuccess('Nothing to delete.'); + return; + } + + if (!json) { + renderPlan(plan); + printWarning( + `About to permanently delete ${plan.deletable.length} sys_secret row(s). This cannot be ` + + 'undone from inside the platform: the audit trail records digests, not handles.', + ); + } + if (!flags.yes) { + if (json) { + await emitJson({ error: 'confirmation_required', hint: 'pass --yes' }, 1, { compact: true }); + return; + } + const ok = await confirm( + chalk.yellow(` Delete ${plan.deletable.length} row(s) after writing the export? [y/N] `), + ); + if (!ok) { printInfo('Aborted — nothing was written or deleted.'); return; } + } + + // ── The export, written and READ BACK before anything is deleted ────── + const doc = buildPreDeleteExport({ + plan, + rawById, + producedBy: `${this.config.bin} secret orphans --delete`, + }); + let verified: PreDeleteExport; + try { + // Owner-only, and CREATED here or not written at all. `mode` is applied + // only when the file is created, so a default (truncating) write onto a + // path some other process won that moment would inherit THAT file's owner + // and permissions and put cipher material in it. The `existsSync` check + // above still runs — it gives the better early message — but it sits + // before the boot, the scan and the confirmation, so it is a pre-check + // across an arbitrarily long window, never the guarantee. `wx` makes the + // guarantee atomic: a file that already exists throws EEXIST straight into + // the `export_failed` refusal below, with nothing deleted. + writeFileSync(exportPath!, `${JSON.stringify(doc, null, 2)}\n`, { encoding: 'utf8', mode: 0o600, flag: 'wx' }); + // Read back rather than trusting the write. A write that reported + // success and produced a short or unreadable file would leave the + // delete with no record at all, which is the one outcome the export + // exists to prevent — and it would look exactly like a clean run. + verified = JSON.parse(readFileSync(exportPath!, 'utf8')) as PreDeleteExport; + } catch (error) { + const message = + `Refusing to delete: the pre-delete export could not be written and read back at ` + + `${exportPath} — ${error instanceof Error ? error.message : String(error)}`; + if (json) { await emitJson({ error: 'export_failed', message }, 1, { compact: true }); return; } + printError(message); + this.exit(1); + return; + } + const exported = new Set(verified.rows?.map((r) => r.id) ?? []); + const missing = plan.deletable.filter((id) => !exported.has(id)); + if (missing.length > 0) { + const message = + `Refusing to delete: the export at ${exportPath} does not record ${missing.length} of the ` + + `${plan.deletable.length} row(s) to be deleted (${missing.join(', ')}).`; + if (json) { await emitJson({ error: 'export_incomplete', message, missing }, 1, { compact: true }); return; } + printError(message); + this.exit(1); + return; + } + if (!json) printSuccess(`Pre-delete export written (owner-only): ${exportPath}`); + + // The union's driver port is READ-ONLY by construction, so the one write + // this command performs is asked for separately and CHECKED — ⛔ never + // cast onto the read port, which would erase exactly the property that + // makes the union safe to depend on. A driver with no `delete` refuses + // here, before the loop, rather than throwing partway through it. + const deleting = asDeletingDriver(secretDriver); + if (!deleting) { + const message = + 'Refusing to delete: the driver serving `sys_secret` exposes no delete(). The export has ' + + `already been written to ${exportPath} and no row was removed.`; + if (json) { await emitJson({ error: 'driver_cannot_delete', message }, 1, { compact: true }); return; } + printError(message); + this.exit(1); + return; + } + + const deleted: string[] = []; + const failed: Array<{ id: string; message: string }> = []; + for (const id of plan.deletable) { + try { + await deleting.delete('sys_secret', id); + deleted.push(id); + } catch (error) { + failed.push({ id, message: error instanceof Error ? error.message : String(error) }); + } + } + + if (json) { + await emitJson( + { mode: 'delete', export: exportPath, deleted, failed, plan }, + failed.length > 0 ? 1 : 0, + { compact: true }, + ); + return; + } + printSuccess(`Deleted ${deleted.length} sys_secret row(s) in ${timer.display()}.`); + for (const f of failed) printError(` ${f.id}: ${f.message}`); + printInfo(`Keep ${exportPath} until you are certain the sweep was correct — it is the only record.`); + if (failed.length > 0) this.exit(1); + } finally { + await stack.shutdown(); + } + } +} + +/** + * The single WRITE verb this command needs, declared apart from the union's + * read-only driver port so the two cannot be confused for one another. + */ +interface SecretDeleteDriverLike { + delete(object: string, id: string): Promise; +} + +/** The driver, if it can delete. `null` is a refusal, never an assumption. */ +export function asDeletingDriver(driver: unknown): SecretDeleteDriverLike | null { + const candidate = driver as Partial | null | undefined; + return candidate && typeof candidate.delete === 'function' + ? (candidate as SecretDeleteDriverLike) + : null; +} + +/** Normalise a driver result (`T[]` or `{ data: T[] }` or a single row). */ +function rowsOf(result: unknown): Array> { + if (!result) return []; + const list = Array.isArray(result) + ? result + : Array.isArray((result as { data?: unknown }).data) + ? (result as { data: unknown[] }).data + : [result]; + return list.filter((r): r is Record => !!r && typeof r === 'object'); +} + +/** + * Read the host's declared datasource artefacts. + * + * ⛔ Throws rather than returning `[]` on anything it cannot read. An empty + * array is the host STATING it has none; a file that does not parse is nobody + * having answered, and the two must not converge — the whole point of the flag + * is that only the host can answer for a datasource declared in code that + * nothing ever installed. + */ +export function readDeclaredDatasources(path: string): DatasourceArtefactLike[] { + const absolute = resolvePath(path); + if (!existsSync(absolute)) { + throw new Error(`--declared-datasources: no such file: ${absolute}`); + } + let parsed: unknown; + try { + parsed = JSON.parse(readFileSync(absolute, 'utf8')); + } catch (error) { + throw new Error( + `--declared-datasources: ${absolute} does not parse as JSON — ` + + `${error instanceof Error ? error.message : String(error)}`, + ); + } + const list = Array.isArray(parsed) + ? parsed + : Array.isArray((parsed as { datasources?: unknown })?.datasources) + ? (parsed as { datasources: unknown[] }).datasources + : undefined; + if (!list) { + throw new Error( + `--declared-datasources: ${absolute} must hold an array of datasource artefacts, or an ` + + 'object with a `datasources` array.', + ); + } + return list.filter((d): d is DatasourceArtefactLike => !!d && typeof d === 'object'); +} + +async function confirm(question: string): Promise { + if (!process.stdin.isTTY) return false; // non-interactive → require --yes + const rl = createInterface({ input: process.stdin, output: process.stdout }); + try { + const answer: string = await new Promise((resolve) => rl.question(question, resolve)); + return /^y(es)?$/i.test(answer.trim()); + } finally { + rl.close(); + } +} + +/** Render the plan for a human. Never prints cipher material. */ +function renderPlan(plan: SysSecretSweepPlan): void { + console.log(chalk.bold('\n Reference union — one line per producer family')); + for (const family of Object.values(plan.families)) { + if (family.status === 'enumerated') { + printSuccess(`${family.family}: enumerated, ${family.referenceCount} reference(s)`); + } else { + printError(`${family.family}: GAP — ${family.reason}`); + } + } + + console.log(chalk.bold('\n sys_secret rows')); + printInfo( + `total ${plan.counts.total} · referenced ${plan.counts.referenced} · ` + + `deletable ${plan.counts.deletable} · withheld ${plan.counts.withheld}`, + ); + for (const [cls, count] of Object.entries(plan.withheldByClass)) { + if (count > 0) printInfo(` withheld · ${cls}: ${count}`); + } + for (const row of plan.rows) { + if (row.decision === 'referenced') continue; // the operator is acting on the rest + const tag = row.decision === 'deletable' ? chalk.red('DELETABLE') : chalk.yellow('withheld '); + console.log(` ${tag} ${row.id} (${row.namespace}.${row.key})`); + console.log(chalk.dim(` ${row.reason}`)); + } + if (plan.legacyInlineRows.length > 0) { + printWarning(`${plan.legacyInlineRows.length} sys_setting row(s) still hold inline ciphertext.`); + } + + console.log(chalk.bold('\n Read before acting')); + for (const note of plan.notes) printWarning(note); +} diff --git a/packages/cli/src/utils/sys-secret-orphan-sweep.test.ts b/packages/cli/src/utils/sys-secret-orphan-sweep.test.ts new file mode 100644 index 0000000000..8e2265dd6e --- /dev/null +++ b/packages/cli/src/utils/sys-secret-orphan-sweep.test.ts @@ -0,0 +1,565 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * #8103 — pins for the `sys_secret` orphan SWEEP planner. + * + * Everything that can run against real code does: a real `ObjectQL`, the real + * `LocalCryptoProvider` (test mode, ephemeral key, no disk), the real + * datasource credential binder, the real shipped settings classifier and the + * real reference union, over a minimal in-memory driver double. The three + * `sys_secret` rows are not hand-written — each is minted by the producer that + * actually writes it, so the handle spellings under test come from the + * producers rather than from this file. + * + * ## The measurement the card turns on, reproduced here against the planner + * + * A LIVE, engine-owned credential is classified `orphaned` by the SHIPPED + * settings-scoped classifier — the OLD predicate — and is therefore in that + * predicate's deletable bucket. The same row under the COMPLETE union is + * `referenced` and never reaches the deletable list. Both halves are asserted + * in one test so neither can drift away from the other. + * + * ## Zero assertions carry positive controls + * + * Every "this is empty" assertion below is accompanied by a control that is + * NOT a substring of the term under test, because a query that matches nothing + * and a query that is broken read identically. Where the assertion is + * `deletable === []`, the control is a run of the SAME planner over the SAME + * fixture that yields a NON-empty deletable list — so an empty answer can only + * come from the guard under test. + */ + +import { describe, it, expect, beforeEach } from 'vitest'; +import { ObjectQL } from '@objectstack/objectql'; +import { createDatasourceSecretBinder } from '@objectstack/service-datasource'; +import { + classifySysSecretRows, + collectEncryptedSpecifierRefs, + isSecretHandle, + LocalCryptoProvider, +} from '@objectstack/service-settings'; +import type { SettingsManifest } from '@objectstack/spec/system'; +import { + buildSecretReferenceUnion, + collectSecretReferenceUnion, + type FamilyResult, + type SecretReferenceEngineLike, + type SecretReferenceUnion, +} from './secret-reference-union.js'; +import { + buildPreDeleteExport, + planSysSecretOrphanSweep, + useHandlePredicate, + SWEEP_EXPOSURE_NOTES, + WITHHELD_LEGACY_INLINE_SIBLING, + WITHHELD_UNATTRIBUTABLE, + WITHHELD_UNION_INCOMPLETE, + WITHHOLD_CLASSES, +} from './sys-secret-orphan-sweep.js'; + +// The command installs the producer's own predicate; the tests run through the +// same seam so nothing here is measured against the module's stand-in. +useHandlePredicate(isSecretHandle); + +type Row = Record; + +function makeDriver() { + const stores = new Map>(); + const storeFor = (object: string) => { + let s = stores.get(object); + if (!s) { s = new Map(); stores.set(object, s); } + return s; + }; + const matches = (row: Row, where: unknown): boolean => { + if (!where || typeof where !== 'object') return true; + for (const [k, v] of Object.entries(where as Row)) { + if (k.startsWith('$')) continue; + if ((row[k] ?? null) !== (v ?? null)) return false; + } + return true; + }; + // Rows leave as COPIES, like a real driver's: handing out the live object + // lets the engine's read-path mask stamp over the stored `secret:` ref. + const copy = (r: Row): Row => ({ ...r }); + let n = 0; + let throwOnFind: { object: string; error: Error } | undefined; + const deleted: Array<{ object: string; id: string }> = []; + + const driver = { + name: 'memory', + version: '0.0.0', + supports: {}, + async connect() {}, + async disconnect() {}, + async checkHealth() { return true; }, + async execute() { return null; }, + async find(object: string, ast?: Row) { + if (throwOnFind?.object === object) throw throwOnFind.error; + const matched = Array.from(storeFor(object).values()).filter((r) => matches(r, ast?.where)); + const page = typeof ast?.limit === 'number' ? matched.slice(0, ast.limit) : matched; + return page.map(copy); + }, + async create(object: string, data: Row) { + n += 1; + const id = (data.id as string) ?? `r_${n}`; + const row = { ...data, id }; + storeFor(object).set(id, row); + return copy(row); + }, + async delete(object: string, id: string) { + deleted.push({ object, id }); + return storeFor(object).delete(id); + }, + async count(object: string, ast?: Row) { + return (await this.find(object, ast)).length; + }, + async beginTransaction() { return { commit: async () => {}, rollback: async () => {} }; }, + async commit() {}, + async rollback() {}, + }; + + return { + driver, + seed(object: string, row: Row) { storeFor(object).set(String(row.id), { ...row }); }, + rowsOf(object: string) { return Array.from(storeFor(object).values()).map(copy); }, + failReadsOf(object: string, error: Error) { throwOnFind = { object, error }; }, + deleted, + }; +} + +const TEST_PACKAGE_ID = 'com.objectstack.test.8103'; +const textField = (name: string) => ({ name, label: name, type: 'text' as const }); + +const sysSecretObject = { + name: 'sys_secret', + label: 'Secret', + fields: { + ...Object.fromEntries( + ['id', 'namespace', 'key', 'kms_key_id', 'alg', 'ciphertext', 'created_at', 'rotated_at'] + .map((f) => [f, textField(f)]), + ), + version: { name: 'version', label: 'version', type: 'number' as const }, + }, +}; + +const sysSettingObject = { + name: 'sys_setting', + label: 'Setting', + fields: Object.fromEntries( + ['id', 'namespace', 'key', 'scope', 'user_id', 'value', 'value_enc'].map((f) => [f, textField(f)]), + ), +}; + +const sysMetadataObject = { + name: 'sys_metadata', + label: 'Metadata', + fields: Object.fromEntries( + ['id', 'name', 'type', 'scope', 'metadata', 'state'].map((f) => [f, textField(f)]), + ), +}; + +/** + * The business object family 2 holds its handle on. `smtp` / `password` makes + * `(namespace, key)` COLLIDE with a declared encrypted settings specifier — + * the collision is not contrived: the engine records (object name, field name) + * while settings records (namespace, specifier key), and nothing keeps the two + * vocabularies apart. + */ +const smtpObject = { + name: 'smtp', + label: 'SMTP account', + fields: { + id: textField('id'), + host: textField('host'), + password: { name: 'password', label: 'password', type: 'secret' as const }, + }, +}; + +const settingsManifests = [ + { + namespace: 'smtp', + specifiers: [ + { key: 'password', type: 'string', encrypted: true }, + { key: 'retired_token', type: 'string', encrypted: true }, + { key: 'inline_legacy', type: 'string', encrypted: true }, + { key: 'host', type: 'string' }, + ], + }, +] as unknown as SettingsManifest[]; + +const attributableTo = collectEncryptedSpecifierRefs(settingsManifests); + +async function buildRuntime() { + const store = makeDriver(); + const engine = new ObjectQL(); + engine.registerDriver(store.driver as never, true); + await engine.init(); + for (const object of [sysSecretObject, sysSettingObject, sysMetadataObject, smtpObject]) { + engine.registry.registerObject(object as never, TEST_PACKAGE_ID); + } + + const crypto = new LocalCryptoProvider({ mode: 'test' }); + engine.setCryptoProvider(crypto as never); + + const seedSecret = (handle: { id: string; kmsKeyId: string; alg: string; version: number; ciphertext: string }, + namespace: string, key: string, extra: Row = {}) => { + store.seed('sys_secret', { + id: handle.id, + namespace, + key, + kms_key_id: handle.kmsKeyId, + alg: handle.alg, + version: handle.version, + ciphertext: handle.ciphertext, + created_at: '2026-01-01T00:00:00.000Z', + ...extra, + }); + }; + + // --- family 1: a settings handle, minted by the real provider, IN FORCE --- + const settingsHandle = await crypto.encrypt('smtp-app-password', { namespace: 'smtp', key: 'password' }); + seedSecret(settingsHandle, 'smtp', 'password'); + store.seed('sys_setting', { + id: 'set_1', namespace: 'smtp', key: 'password', scope: 'tenant', user_id: null, + value_enc: settingsHandle.id, + }); + + // --- a GENUINE settings orphan: attributable, named by nothing ----------- + // Minted by the same real provider under a declared encrypted specifier, and + // deliberately not referenced by any sys_setting row. This is the one class + // the ruling permits deleting, and it is what keeps every "deletable is + // empty" assertion below falsifiable. + const orphanHandle = await crypto.encrypt('rotated-away-token', { namespace: 'smtp', key: 'retired_token' }); + seedSecret(orphanHandle, 'smtp', 'retired_token'); + + // --- family 2: the engine's own secret-field channel, LIVE --------------- + await engine.insert('smtp', { id: 'rec_1', host: 'mail.example.com', password: 'hunter2' }); + const objectFieldHandleId = String(store.rowsOf('smtp')[0].password).slice('secret:'.length); + + // --- family 3: the REAL datasource credential binder, LIVE --------------- + const binder = createDatasourceSecretBinder({ engine: engine as never, cryptoProvider: crypto as never }); + const credentialsRef = await binder.bind({ value: 'pg-password' }, { name: 'main' }); + store.seed('sys_metadata', { + id: 'meta_1', name: 'main', type: 'datasource', scope: 'platform', state: 'active', + metadata: JSON.stringify({ name: 'main', driver: 'postgres', external: { credentialsRef } }), + }); + + return { + engine: engine as unknown as SecretReferenceEngineLike, + realEngine: engine, + store, + crypto, + settingsHandleId: settingsHandle.id, + orphanHandleId: orphanHandle.id, + objectFieldHandleId, + datasourceHandleId: credentialsRef.slice('sys_secret:'.length), + }; +} + +type Runtime = Awaited>; + +const secretsOf = (rt: Runtime) => rt.store.rowsOf('sys_secret') as never[]; +const settingRowsOf = (rt: Runtime) => rt.store.rowsOf('sys_setting') as never[]; + +/** + * ⛔ `declared` is REQUIRED and has no default. A default here would swallow an + * explicitly passed `undefined` — which is precisely the value under test, and + * the first version of this file did exactly that and reported a complete union + * for the run that was meant to gap. + */ +const planOver = async (rt: Runtime, declared: readonly { name?: string }[] | undefined) => { + const union = await collectSecretReferenceUnion({ engine: rt.engine, declaredDatasources: declared }); + return { + union, + plan: planSysSecretOrphanSweep({ + secrets: secretsOf(rt), + union, + attributableTo, + settingRows: settingRowsOf(rt), + }), + }; +}; + +describe('#8103 reproduction — the old predicate vs the COMPLETE union, on real code', () => { + let rt: Runtime; + beforeEach(async () => { rt = await buildRuntime(); }); + + it('a LIVE engine-owned row is deletable under the OLD predicate and REFERENCED under the union', async () => { + // ── OLD predicate: the shipped settings-scoped classifier ────────────── + const shipped = classifySysSecretRows({ + secrets: secretsOf(rt), + settingRows: settingRowsOf(rt), + attributableTo, + }); + const oldVerdict = shipped.rows.find((r) => r.id === rt.objectFieldHandleId); + expect(oldVerdict?.verdict).toBe('orphaned'); + + // …and the row is LIVE: a business row's own column still names it. + expect(rt.store.rowsOf('smtp')[0].password).toBe(`secret:${rt.objectFieldHandleId}`); + + // ── The ruled predicate: unreferenced by the COMPLETE union ──────────── + const { union, plan } = await planOver(rt, []); + expect(union.complete).toBe(true); + expect(plan.refusal).toBeNull(); + + const swept = plan.rows.find((r) => r.id === rt.objectFieldHandleId); + expect(swept?.decision).toBe('referenced'); + expect(swept?.holders).toEqual(['object-field: smtp.password#rec_1']); + expect(plan.deletable).not.toContain(rt.objectFieldHandleId); + + // Positive control for that exclusion — the SAME planner over the SAME + // fixture DOES produce a deletable row, so "not in the list" is a decision + // about this handle and not an empty list. + expect(plan.deletable).toEqual([rt.orphanHandleId]); + }); + + it('the datasource handle escapes the old predicate only by NOT colliding, and the union names it', async () => { + const shipped = classifySysSecretRows({ + secrets: secretsOf(rt), settingRows: settingRowsOf(rt), attributableTo, + }); + // `unattributable` under the old predicate — a name match, not ownership. + expect(shipped.rows.find((r) => r.id === rt.datasourceHandleId)?.verdict).toBe('unattributable'); + + const { plan } = await planOver(rt, []); + const swept = plan.rows.find((r) => r.id === rt.datasourceHandleId); + expect(swept?.decision).toBe('referenced'); + expect(swept?.holders).toEqual(['datasource: datasource(main).external.credentialsRef']); + }); +}); + +describe('the falsifiable criterion — an incomplete union refuses, and NAMES the family', () => { + let rt: Runtime; + beforeEach(async () => { rt = await buildRuntime(); }); + + it('the host not declaring its datasources gaps family 3 and empties the deletable set', async () => { + // Control first: with the host answering `[]`, the sweep DOES delete. + const answered = await planOver(rt, []); + expect(answered.plan.deletable).toEqual([rt.orphanHandleId]); + + // `undefined` is "nobody answered", NOT "there are none". + const { plan } = await planOver(rt, undefined); + expect(plan.refusal?.code).toBe('PRECONDITION_REQUIRED'); + expect(plan.refusal?.status).toBe(428); + expect(plan.refusal?.gaps.map((g) => g.family)).toEqual(['datasource']); + expect(plan.refusal?.message).toContain('datasource'); + expect(plan.deletable).toEqual([]); + + // …and the genuine orphan is withheld under its OWN class, not silently dropped. + const orphan = plan.rows.find((r) => r.id === rt.orphanHandleId); + expect(orphan?.decision).toBe('withheld'); + expect(orphan?.withheld).toBe(WITHHELD_UNION_INCOMPLETE); + }); + + it('an unreadable sys_setting gaps family 1, and the refusal says so', async () => { + rt.store.failReadsOf('sys_setting', new Error('connection reset')); + const { plan } = await planOver(rt, []); + expect(plan.refusal?.gaps.map((g) => g.family)).toEqual(['settings']); + expect(plan.refusal?.gaps[0]?.reason).toContain('connection reset'); + expect(plan.deletable).toEqual([]); + expect(plan.families.settings.status).toBe('gap'); + // ⛔ Not flattened: the families that DID enumerate still say so, with counts. + expect(plan.families['object-field'].status).toBe('enumerated'); + expect(plan.families['object-field'].referenceCount).toBe(1); + expect(plan.families.datasource.status).toBe('enumerated'); + }); + + it('an engine with no listDatasourceDefs() gaps family 3 even when the host answered', async () => { + const narrowed: SecretReferenceEngineLike = { + getConfigs: () => rt.engine.getConfigs(), + getDriverForObject: (o) => rt.engine.getDriverForObject(o), + }; + const union = await collectSecretReferenceUnion({ engine: narrowed, declaredDatasources: [] }); + const plan = planSysSecretOrphanSweep({ + secrets: secretsOf(rt), union, attributableTo, settingRows: settingRowsOf(rt), + }); + expect(plan.refusal?.gaps.map((g) => g.family)).toEqual(['datasource']); + expect(plan.families.datasource.reason).toContain('listDatasourceDefs'); + expect(plan.deletable).toEqual([]); + }); + + it('a partially-gapped family still proves its gathered handles LIVE', () => { + // A gap does not discard the references collected before it opened: those + // handles are named, so they are referenced, not merely un-enumerated. + const partial: FamilyResult = { + family: 'object-field', + status: 'gap', + reason: 'reading secret field(s) of `other` threw — Error: table is locked', + references: [{ handleId: 'sec_partial', family: 'object-field', holder: 'smtp.password#rec_1' }], + }; + const union: SecretReferenceUnion = buildSecretReferenceUnion({ + settings: { family: 'settings', status: 'enumerated', references: [] }, + 'object-field': partial, + datasource: { family: 'datasource', status: 'enumerated', references: [] }, + }); + const plan = planSysSecretOrphanSweep({ + secrets: [{ id: 'sec_partial', namespace: 'smtp', key: 'password' }], + union, + attributableTo, + settingRows: [], + }); + expect(plan.rows[0]?.decision).toBe('referenced'); + expect(plan.counts.referenced).toBe(1); + }); +}); + +describe('the classes the ruling puts out of reach', () => { + let rt: Runtime; + beforeEach(async () => { rt = await buildRuntime(); }); + + it('`unattributable` is NEVER deletable, even with a complete union', async () => { + // Unreference the datasource handle so nothing names it at all: it is now + // unreferenced AND unattributable — the exact shape a settings-only sweep + // would have deleted. + rt.store.seed('sys_metadata', { + id: 'meta_1', name: 'main', type: 'datasource', scope: 'platform', state: 'active', + metadata: JSON.stringify({ name: 'main', driver: 'postgres', external: {} }), + }); + const { union, plan } = await planOver(rt, []); + expect(union.complete).toBe(true); + expect(union.handleIds.has(rt.datasourceHandleId)).toBe(false); + + const row = plan.rows.find((r) => r.id === rt.datasourceHandleId); + expect(row?.decision).toBe('withheld'); + expect(row?.withheld).toBe(WITHHELD_UNATTRIBUTABLE); + expect(row?.attributable).toBe(false); + expect(plan.deletable).not.toContain(rt.datasourceHandleId); + // Positive control: the run is live and does delete something. + expect(plan.deletable).toEqual([rt.orphanHandleId]); + }); + + it('a LEGACY INLINE sibling is withheld — the #8063 guard in the opposite direction', async () => { + const inlineHandle = await rt.crypto.encrypt('older-inline', { namespace: 'smtp', key: 'inline_legacy' }); + rt.store.seed('sys_secret', { + id: inlineHandle.id, namespace: 'smtp', key: 'inline_legacy', + kms_key_id: inlineHandle.kmsKeyId, alg: inlineHandle.alg, + version: inlineHandle.version, ciphertext: inlineHandle.ciphertext, + }); + // The pair now resolves through INLINE ciphertext, which names no handle. + rt.store.seed('sys_setting', { + id: 'set_legacy', namespace: 'smtp', key: 'inline_legacy', scope: 'tenant', user_id: null, + value_enc: 'AQIDBAUGBwgJCg==', + }); + + const { plan } = await planOver(rt, []); + const row = plan.rows.find((r) => r.id === inlineHandle.id); + expect(row?.decision).toBe('withheld'); + expect(row?.withheld).toBe(WITHHELD_LEGACY_INLINE_SIBLING); + expect(row?.legacyInlineSibling).toBe(true); + expect(row?.attributable).toBe(true); // attributable, and STILL not deleted + expect(plan.deletable).not.toContain(inlineHandle.id); + expect(plan.legacyInlineRows).toEqual([ + { namespace: 'smtp', key: 'inline_legacy', scope: 'tenant', user_id: null }, + ]); + // Positive control: the same run still deletes the genuine orphan. + expect(plan.deletable).toEqual([rt.orphanHandleId]); + }); + + it('a re-wrapped row is never read as a retirement — version/rotated_at are not verdict inputs', async () => { + // Re-wrap the IN-FORCE settings row the way `rotateKey` does: same id, a + // bumped version and a rotation timestamp. + const before = rt.store.rowsOf('sys_secret').find((r) => r.id === rt.settingsHandleId)!; + rt.store.seed('sys_secret', { ...before, version: 4, rotated_at: '2026-08-01T00:00:00.000Z' }); + + const { plan } = await planOver(rt, []); + const row = plan.rows.find((r) => r.id === rt.settingsHandleId); + expect(row?.decision).toBe('referenced'); + expect(row?.rewrapped).toBe(true); + expect(row?.reason).toContain('re-wrap is not a retirement'); + expect(plan.deletable).not.toContain(rt.settingsHandleId); + + // The inverse, which is the dangerous half: stamp the SAME re-wrap + // evidence onto the genuine orphan. Re-wrap evidence must move nothing — + // the orphan is deletable because nothing names it, and it stays deletable + // for that reason alone. + const orphanRow = rt.store.rowsOf('sys_secret').find((r) => r.id === rt.orphanHandleId)!; + rt.store.seed('sys_secret', { ...orphanRow, version: 4, rotated_at: '2026-08-01T00:00:00.000Z' }); + const after = await planOver(rt, []); + expect(after.plan.deletable).toEqual([rt.orphanHandleId]); + expect(after.plan.rows.find((r) => r.id === rt.orphanHandleId)?.rewrapped).toBe(true); + }); +}); + +describe('the operator-facing text says what was measured', () => { + let rt: Runtime; + beforeEach(async () => { rt = await buildRuntime(); }); + + it('never claims the sweep cleans up leaked or exposed old credentials', async () => { + const { plan } = await planOver(rt, []); + const prose = plan.notes.join(' ').toLowerCase(); + + // Positive control on the search itself: a phrase that IS present. + expect(prose).toContain('still in force'); + + // The inverted framing, stated rather than implied. + expect(prose).toContain('not "old credentials that were replaced"'); + expect(prose).toContain('retires nothing that is exposed'); + expect(prose).toContain('currently valid'); + // …and the false claim is absent. + expect(prose).not.toContain('cleans up'); + expect(prose).not.toContain('leaked'); + }); + + it('the exposure notes are always present, whatever the plan found', async () => { + const { plan } = await planOver(rt, []); + for (const note of SWEEP_EXPOSURE_NOTES) expect(plan.notes).toContain(note); + }); + + it('the withhold classes are a closed, countable set', async () => { + const { plan } = await planOver(rt, undefined); + expect(Object.keys(plan.withheldByClass).sort()).toEqual([...WITHHOLD_CLASSES].sort()); + const summed = Object.values(plan.withheldByClass).reduce((a, b) => a + b, 0); + expect(summed).toBe(plan.counts.withheld); + expect(plan.counts.total).toBe(plan.counts.referenced + plan.counts.deletable + plan.counts.withheld); + }); +}); + +describe('the mandatory pre-delete export', () => { + let rt: Runtime; + beforeEach(async () => { rt = await buildRuntime(); }); + + it('carries the cipher material and the holder evidence for every deletable row', async () => { + const { plan } = await planOver(rt, []); + const rawById = new Map(rt.store.rowsOf('sys_secret').map((r) => [String(r.id), r])); + const doc = buildPreDeleteExport({ plan, rawById, producedBy: 'os secret orphans --delete' }); + + expect(doc.format).toBe('objectstack.sys_secret.pre-delete-export.v1'); + expect(doc.rows.map((r) => r.id)).toEqual([rt.orphanHandleId]); + // Restorable: the ciphertext is the row's own, not a placeholder. + expect(doc.rows[0]?.ciphertext).toBe(rawById.get(rt.orphanHandleId)?.ciphertext); + expect(typeof doc.rows[0]?.ciphertext).toBe('string'); + expect(doc.rows[0]?.kms_key_id).toBe('local:v1'); + expect(doc.decisions[0]?.id).toBe(rt.orphanHandleId); + expect(doc.decisions[0]?.reason).toContain('COMPLETE reference union'); + expect(doc.families.datasource.status).toBe('enumerated'); + expect(doc.warning).toContain('CIPHER MATERIAL'); + }); + + it('refuses to build when a deletable row was not read — never a shorter export', async () => { + const { plan } = await planOver(rt, []); + expect(plan.deletable).toEqual([rt.orphanHandleId]); // control: there IS a row to miss + expect(() => buildPreDeleteExport({ + plan, rawById: new Map(), producedBy: 'test', + })).toThrow(/no raw sys_secret row was read/); + }); + + it('an incomplete union produces an empty export because nothing is deletable', async () => { + const { plan } = await planOver(rt, undefined); + const rawById = new Map(rt.store.rowsOf('sys_secret').map((r) => [String(r.id), r])); + const doc = buildPreDeleteExport({ plan, rawById, producedBy: 'test' }); + expect(doc.rows).toEqual([]); + // Control: the same rawById over a COMPLETE plan is non-empty. + const complete = await planOver(rt, []); + expect(buildPreDeleteExport({ plan: complete.plan, rawById, producedBy: 'test' }).rows) + .toHaveLength(1); + }); +}); + +describe('the handle predicate comes from the producer', () => { + it('agrees with a handle minted by the real LocalCryptoProvider', async () => { + const crypto = new LocalCryptoProvider({ mode: 'test' }); + const handle = await crypto.encrypt('x', { namespace: 'smtp', key: 'password' }); + expect(isSecretHandle(handle.id)).toBe(true); + // A legacy inline value is not a handle — the discriminator the guard rests on. + expect(isSecretHandle('AQIDBAUGBwgJCg==')).toBe(false); + }); +}); diff --git a/packages/cli/src/utils/sys-secret-orphan-sweep.ts b/packages/cli/src/utils/sys-secret-orphan-sweep.ts new file mode 100644 index 0000000000..75fa8fbfd7 --- /dev/null +++ b/packages/cli/src/utils/sys-secret-orphan-sweep.ts @@ -0,0 +1,608 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * #8103 — the deletion half: planning a `sys_secret` orphan sweep over the + * COMPLETE cross-producer reference union. + * + * Pure. Nothing here reads a database, writes one, or decrypts anything; the + * command (`os secret orphans`) does the I/O and hands the readings in. That + * split is what makes the safety property testable: every refusal below is a + * value returned by a function, not a branch that only exists inside a + * command. + * + * ## The predicate, and why it is not the shipped one + * + * The maintainer ruling (2026-08-27, option B') states the deletion predicate + * exactly: **attributable AND unreferenced by the COMPLETE union**. Both + * conjuncts are load-bearing and neither is the shipped classifier's: + * + * - `classifySysSecretRows` (`@objectstack/service-settings`) answers + * "unreferenced by `sys_setting`", because that is all its package can see. + * Under that predicate a LIVE, engine-owned credential lands in the + * `orphaned` bucket — measured against the real producers, not argued, in + * `secret-reference-union.test.ts` and again in this module's own test. + * ⛔ That bucket is therefore NOT a deletion list, and this module never + * treats it as one: the reference side comes from + * {@link SecretReferenceUnion} over all three producer families. + * - Attribution stays exactly as narrow as it is: `(namespace, key)` + * membership in the registered manifests' encrypted specifiers. It is a + * NAME MATCH, not proof of ownership, so it can only ever be used to + * SHRINK the deletable set — never to grow it. A row nothing attributes is + * `unattributable` and the ruling puts it permanently out of reach. + * + * ## What an incomplete union does here + * + * It empties the deletable set, and it says which family is missing. The union + * models three independent gap sources (the host did not declare; the engine + * exposes no accessor; neither reaches), and ⛔ this module does not flatten + * them into a boolean: {@link SysSecretSweepPlan.families} carries every + * family's own status and reason through to the operator and to `--json`, and + * a withheld row names `union_incomplete` as its own withhold class rather + * than being lumped in with the rest. + * + * "The read did not happen" and "there are no references" are different facts. + * Only the second is safe to delete on, and the audit trail cannot tell them + * apart afterwards: it records content digests, never handles (re-measured + * 2026-08-28 — `LocalCryptoProvider.digest()` is `sha256:` + sha256(plain), + * and the only values reaching the audit payload are that digest or `null`), + * so a row deleted in error can never be NAMED again, let alone recovered. + * That single fact is why the export is mandatory rather than advisory. + * + * ## The two look-alike classes, both withheld rather than warned about + * + * - **Legacy inline crypto, in the opposite direction from #8063's reaper.** + * The pre-Phase-3 path stores the ciphertext ITSELF in + * `sys_setting.value_enc`. Such a value names no handle, so it contributes + * nothing to the union — and a `sys_secret` row sharing its + * `(namespace, key)` therefore looks unreferenced by construction. The + * shipped report flags that pair `legacyInlineSibling` and tells an + * operator to look. A DELETING command cannot delegate to an operator's + * attention: here the class is {@link WITHHELD_LEGACY_INLINE_SIBLING}, + * excluded from the deletable set and reported. + * - **A re-wrap is not a retirement.** `rotateKey()` re-wraps in place and + * keeps `id` stable (re-measured 2026-08-28: `local-crypto-provider.ts` + * returns `id: handle.id` with `version + 1`), so a re-wrapped row is the + * row still in force. `version` and `rotated_at` are therefore REPORTED and + * ⛔ never verdict inputs — a classifier reading "version > 1" as "was + * rotated, therefore retired" deletes the value in force. + * + * ## What this sweep does NOT do, stated because the opposite reads better + * + * ⛔ It does not "clean up leaked old credentials". Measured on the pre-fix + * rotation path, the framing is inverted: the handle was never repointed, so + * the value STILL IN FORCE is the OLDEST one — the credential the + * administrator believed they had replaced — while each orphan holds a value + * the administrator INTENDED to set and which never took effect. Deleting the + * orphans therefore removes nothing that is exposed; the exposed credential is + * the referenced one and it stays. And if the administrator also rotated at + * the provider, the newest orphan may be a credential that is CURRENTLY VALID + * there. {@link SWEEP_EXPOSURE_NOTES} is that paragraph in the form the + * command prints, so the operator-facing wording is pinned by test rather than + * left to a rendering site. + */ + +import type { + EncryptedSpecifierRef, + SecretRowSnapshot, + SettingRowSnapshot, +} from '@objectstack/service-settings'; +import type { + SecretReferenceFamily, + SecretReferenceUnion, +} from './secret-reference-union.js'; + +export type { EncryptedSpecifierRef, SecretRowSnapshot, SettingRowSnapshot }; + +/** + * Why a row that is not referenced is still not deletable. + * + * Separate constants rather than free strings so each class is countable and + * each is pinned by its own test. Every one of them is a REFUSAL: the sweep + * only ever removes rows from the deletable set, it never adds one. + */ +export const WITHHELD_UNION_INCOMPLETE = 'union_incomplete'; +/** The class the ruling puts permanently out of reach. */ +export const WITHHELD_UNATTRIBUTABLE = 'unattributable'; +/** The legacy inline-crypto guard, in the opposite direction from #8063's. */ +export const WITHHELD_LEGACY_INLINE_SIBLING = 'legacy_inline_sibling'; + +/** The closed set of withhold classes. */ +export const WITHHOLD_CLASSES = [ + WITHHELD_UNION_INCOMPLETE, + WITHHELD_UNATTRIBUTABLE, + WITHHELD_LEGACY_INLINE_SIBLING, +] as const; + +export type WithholdClass = (typeof WITHHOLD_CLASSES)[number]; + +/** What the sweep decided about one `sys_secret` row. */ +export type SweepDecision = 'referenced' | 'deletable' | 'withheld'; + +/** + * One `sys_secret` row, as the sweep reports it. + * + * ⛔ No `ciphertext` member, the same discipline as the shipped report's + * `SecretRowSnapshot`: this shape is printed and serialised to `--json`, and + * cipher material must not be reachable from a surface that is expected to be + * safe to show. The pre-delete export is the one place cipher material travels + * (see {@link buildPreDeleteExport}), and it is a separate type for that + * reason. + */ +export interface SweptSecretRow { + id: string; + namespace: string; + key: string; + decision: SweepDecision; + /** Set only when `decision === 'withheld'`. */ + withheld?: WithholdClass; + /** Justification, safe to print. */ + reason: string; + /** + * Holder coordinates naming this handle, from the union — e.g. + * `object-field: smtp.password#rec_7`. Empty when nothing names it. This is + * the evidence the digest-only audit trail can never reconstruct after a + * delete, so it travels into the export too. + */ + holders: string[]; + /** Re-wrap evidence. Reported only — ⛔ never a verdict input. */ + rewrapped?: boolean; + /** Its `(namespace, key)` currently resolves through a legacy inline value. */ + legacyInlineSibling?: boolean; + /** `(namespace, key)` matches a declared encrypted settings specifier. */ + attributable: boolean; + version?: number | null; + kms_key_id?: string | null; + created_at?: string | null; + rotated_at?: string | null; +} + +/** Per-family passthrough of the union's own outcome. ⛔ Never a boolean. */ +export interface SweepFamilyStatus { + family: SecretReferenceFamily; + status: 'enumerated' | 'gap'; + /** Present only on a gap — the union's own words for why. */ + reason?: string; + /** References this family contributed (real even on a gap). */ + referenceCount: number; +} + +/** The refusal a plan carries when the union could not be completed. */ +export interface SweepRefusal { + /** ADR-0112 pair, mirroring `IncompleteSecretReferenceUnionError`. */ + code: 'PRECONDITION_REQUIRED'; + status: 428; + /** Which families could not be enumerated, and why. */ + gaps: ReadonlyArray<{ family: SecretReferenceFamily; reason: string }>; + message: string; +} + +/** The plan. Counts, row ids and holder coordinates — never cipher material. */ +export interface SysSecretSweepPlan { + /** + * `null` when every family enumerated. Otherwise the reason deletion is + * refused, naming the families — and {@link SysSecretSweepPlan.deletable} is + * then guaranteed empty. + */ + refusal: SweepRefusal | null; + counts: { + total: number; + referenced: number; + deletable: number; + withheld: number; + }; + /** Withheld rows by class. Keys are the closed {@link WITHHOLD_CLASSES}. */ + withheldByClass: Record; + rows: SweptSecretRow[]; + /** The ids `--delete` would remove. Empty whenever `refusal` is set. */ + deletable: string[]; + families: Record; + /** + * `sys_setting` rows still on the legacy inline path. They reference no + * `sys_secret` row at all; rows sharing their `(namespace, key)` are + * withheld. + */ + legacyInlineRows: Array<{ + namespace: string; + key: string; + scope?: string | null; + user_id?: string | null; + }>; + /** What an operator must read before acting. Never claims a cleanup. */ + notes: string[]; +} + +/** + * The composite `(namespace, key)` map key. + * + * NUL separator, written as the ESCAPE and never as a raw byte + * (`scripts/check-nul-bytes.mjs`) — byte-identical at run time. Same choice + * and same reason as the shipped classifier: neither a namespace nor a key can + * contain one, so no two distinct pairs alias into a single composite, and an + * aliased pair here would silently change a row's DECISION. + */ +const refKey = (namespace: string, key: string) => `${namespace}\u0000${key}`; + +/** + * The operator-facing exposure paragraph, as measured. + * + * Kept as data, and asserted by test, because the tempting sentence — "this + * cleans up leaked old credentials" — is false in this population and would be + * the single most damaging thing this command could tell an operator. + */ +export const SWEEP_EXPOSURE_NOTES: readonly string[] = [ + 'These rows are NOT "old credentials that were replaced". On the pre-fix rotation path the ' + + 'handle was never repointed, so the value STILL IN FORCE is the OLDEST one — the credential ' + + 'the administrator believed they had replaced — while each orphan holds a value the ' + + 'administrator INTENDED to set and which never took effect.', + 'Deleting these rows therefore retires NOTHING that is exposed. The exposed credential is the ' + + 'referenced one, and this command never touches a referenced row.', + 'If the administrator also rotated at the provider, the newest orphan may be a credential that ' + + 'is CURRENTLY VALID there. Read the export before deleting.', + 'The audit trail records content digests, never handles, so a row deleted in error cannot be ' + + 'named afterwards. The pre-delete export is the only record that survives the delete.', +]; + +/** + * Is this `sys_setting.value_enc` a handle rather than inline ciphertext? + * + * The producer's own published predicate is injected by + * {@link useHandlePredicate} rather than restated here — a restated `sec_` + * prefix is a second de-facto contract that drifts, and the failure it + * produces is a legacy inline row silently counted as a reference. The + * fallback keeps this module usable in isolation; every shipping path installs + * the real one, and the test pins that the two agree on a handle minted by the + * real `LocalCryptoProvider`. + */ +let isHandleShaped: (value: unknown) => boolean = (value) => + typeof value === 'string' && value.startsWith('sec_'); + +/** Install the producer's own handle predicate (`isSecretHandle`). */ +export function useHandlePredicate(predicate: (value: unknown) => boolean): void { + isHandleShaped = predicate; +} + +/** + * Plan a sweep. + * + * @param input.secrets every `sys_secret` row, read unscoped. + * @param input.union the cross-producer reference union. + * @param input.attributableTo `(namespace, key)` pairs the REGISTERED settings + * manifests declare encrypted (`collectEncryptedSpecifierRefs`). An empty set + * is legal and yields zero deletable rows — the safe direction. + * @param input.settingRows `sys_setting` rows, for the legacy inline guard. + */ +export function planSysSecretOrphanSweep(input: { + secrets: readonly SecretRowSnapshot[]; + union: SecretReferenceUnion; + attributableTo: readonly EncryptedSpecifierRef[]; + settingRows: readonly SettingRowSnapshot[]; +}): SysSecretSweepPlan { + const secrets = input.secrets ?? []; + const union = input.union; + const attributable = new Set( + (input.attributableTo ?? []).map((r) => refKey(r.namespace, r.key)), + ); + + // The legacy inline guard, computed the same way the shipped report computes + // it and for the same reason: a `value_enc` that is not a handle references + // no `sys_secret` row, so a row sharing its pair looks unreferenced BECAUSE + // of the legacy value rather than in spite of it. + const legacyInlineRows: SysSecretSweepPlan['legacyInlineRows'] = []; + const legacyInlinePairs = new Set(); + for (const row of input.settingRows ?? []) { + const enc = row?.value_enc; + if (typeof enc !== 'string' || enc === '') continue; + if (isHandleShaped(enc)) continue; + legacyInlineRows.push({ + namespace: row.namespace, + key: row.key, + scope: row.scope ?? null, + user_id: row.user_id ?? null, + }); + legacyInlinePairs.add(refKey(row.namespace, row.key)); + } + + const holdersById = new Map(); + for (const ref of union.references) { + const entry = `${ref.family}: ${ref.holder}`; + const list = holdersById.get(ref.handleId); + if (list) list.push(entry); + else holdersById.set(ref.handleId, [entry]); + } + + const refusal: SweepRefusal | null = union.complete + ? null + : { + code: 'PRECONDITION_REQUIRED', + status: 428, + gaps: union.gaps, + message: + `Refusing to delete: ${union.gaps.length} of 3 sys_secret producer families could not be ` + + `enumerated (${union.gaps.map((g) => g.family).join(', ')}). A handle absent from an ` + + 'INCOMPLETE union is not thereby unreferenced — the missing family may hold it — and ' + + 'the audit trail records digests rather than handles, so an erroneous delete cannot be ' + + 'named afterwards. Close the gap and re-run.', + }; + + const rows: SweptSecretRow[] = []; + const withheldByClass: Record = { + [WITHHELD_UNION_INCOMPLETE]: 0, + [WITHHELD_UNATTRIBUTABLE]: 0, + [WITHHELD_LEGACY_INLINE_SIBLING]: 0, + }; + let referenced = 0; + const deletable: string[] = []; + + for (const secret of secrets) { + // Reported, never a verdict input — a re-wrap keeps the handle stable. + const rewrapped = + (typeof secret.version === 'number' && secret.version > 1) + || (typeof secret.rotated_at === 'string' && secret.rotated_at !== ''); + const legacyInlineSibling = legacyInlinePairs.has(refKey(secret.namespace, secret.key)); + const isAttributable = attributable.has(refKey(secret.namespace, secret.key)); + const holders = holdersById.get(secret.id) ?? []; + + const common = { + id: secret.id, + namespace: secret.namespace, + key: secret.key, + holders, + attributable: isAttributable, + version: secret.version ?? null, + kms_key_id: secret.kms_key_id ?? null, + created_at: secret.created_at ?? null, + rotated_at: secret.rotated_at ?? null, + ...(rewrapped ? { rewrapped: true } : {}), + ...(legacyInlineSibling ? { legacyInlineSibling: true } : {}), + }; + + // 1. Referenced by ANY family — including a family that gapped partway, + // because the references it did gather are real. A handle a partial + // read already proved LIVE is live whatever happens to the rest. + if (union.handleIds.has(secret.id)) { + rows.push({ + ...common, + decision: 'referenced', + reason: rewrapped + ? `named by ${holders.length} live holder(s); re-wrapped in place (handle stable) — a ` + + 're-wrap is not a retirement' + : `named by ${holders.length} live holder(s)`, + }); + referenced += 1; + continue; + } + + // 2. The union is not complete ⇒ "absent" means nothing. This is the + // falsifiable criterion: the refusal is a per-row class, and the plan + // names which family is missing. + if (refusal) { + rows.push({ + ...common, + decision: 'withheld', + withheld: WITHHELD_UNION_INCOMPLETE, + reason: + 'no ENUMERATED family names this handle, but the union is incomplete ' + + `(${refusal.gaps.map((g) => g.family).join(', ')} could not be enumerated), so ` + + '"absent" does not mean "unreferenced"', + }); + withheldByClass[WITHHELD_UNION_INCOMPLETE] += 1; + continue; + } + + // 3. Never deletable by the ruling: nothing attributes it to settings, so + // the sweep cannot claim ownership of it even with a complete union. + if (!isAttributable) { + rows.push({ + ...common, + decision: 'withheld', + withheld: WITHHELD_UNATTRIBUTABLE, + reason: + 'no family names this handle, and its (namespace, key) matches no declared encrypted ' + + 'settings specifier — nothing attributes it to a producer this sweep speaks for. ' + + 'Never deletable.', + }); + withheldByClass[WITHHELD_UNATTRIBUTABLE] += 1; + continue; + } + + // 4. The legacy inline guard, in the opposite direction from #8063's + // reaper: this pair currently resolves through an inline ciphertext, + // which names no handle — so the row's absence from the union is + // explained by the legacy value rather than by retirement. + if (legacyInlineSibling) { + rows.push({ + ...common, + decision: 'withheld', + withheld: WITHHELD_LEGACY_INLINE_SIBLING, + reason: + 'attributable and unreferenced, but its (namespace, key) currently resolves through a ' + + 'LEGACY INLINE sys_setting value, which names no handle — the absence is explained by ' + + 'the legacy value, not by retirement. Migrate that setting off the inline path first.', + }); + withheldByClass[WITHHELD_LEGACY_INLINE_SIBLING] += 1; + continue; + } + + rows.push({ + ...common, + decision: 'deletable', + reason: + 'attributable to a declared encrypted settings specifier AND named by no family of the ' + + 'COMPLETE reference union', + }); + deletable.push(secret.id); + } + + const families = {} as Record; + for (const [family, result] of Object.entries(union.families) as Array< + [SecretReferenceFamily, SecretReferenceUnion['families'][SecretReferenceFamily]] + >) { + families[family] = { + family, + status: result.status, + ...(result.status === 'gap' ? { reason: result.reason } : {}), + referenceCount: result.references.length, + }; + } + + const notes = [...SWEEP_EXPOSURE_NOTES]; + if (attributable.size === 0) { + notes.push( + 'No encrypted specifiers were supplied, so nothing is attributable to the settings producer ' + + 'and nothing is deletable. Boot with the settings service registered so its manifests ' + + 'are readable.', + ); + } + if (legacyInlineRows.length > 0) { + notes.push( + `${legacyInlineRows.length} sys_setting row(s) still hold inline ciphertext rather than a ` + + 'handle. They reference no sys_secret row; rows sharing their (namespace, key) are ' + + 'withheld rather than deleted.', + ); + } + notes.push( + 'Attribution is by (namespace, key), which is a NAME MATCH and not proof of ownership — ' + + 'sys_secret carries no producer column. It is used here only to SHRINK the deletable set; ' + + 'a row nothing attributes is never deleted.', + ); + + return { + refusal, + counts: { + total: rows.length, + referenced, + deletable: deletable.length, + withheld: + withheldByClass[WITHHELD_UNION_INCOMPLETE] + + withheldByClass[WITHHELD_UNATTRIBUTABLE] + + withheldByClass[WITHHELD_LEGACY_INLINE_SIBLING], + }, + withheldByClass, + rows, + deletable, + families, + legacyInlineRows, + notes, + }; +} + +/** + * A `sys_secret` row as the EXPORT carries it — cipher material included. + * + * The one shape in this feature that holds `ciphertext`, and it is separate + * from {@link SweptSecretRow} for exactly that reason: the report type cannot + * accidentally acquire cipher material, and this type cannot accidentally be + * printed in place of the report. + * + * It carries the cipher material because the export's job is RESTORATION, not + * merely naming. The audit trail records digests rather than handles, so after + * an erroneous delete nothing in the database can say which row existed; an + * export that named the row without its ciphertext would make the mistake + * describable and still permanent. Every column `sys_secret` declares is + * carried, so a row can be re-inserted as it stood. + */ +export interface ExportedSecretRow { + id: string; + namespace: string; + key: string; + kms_key_id?: string | null; + alg?: string | null; + version?: number | null; + ciphertext?: string | null; + created_at?: string | null; + rotated_at?: string | null; +} + +/** The pre-delete export document. */ +export interface PreDeleteExport { + /** Format marker, so a restorer can refuse a document it does not know. */ + format: 'objectstack.sys_secret.pre-delete-export.v1'; + generatedAt: string; + /** The command that produced it, for the operator reading it months later. */ + producedBy: string; + /** + * ⛔ Reads as a warning, not a footnote: this file holds the cipher material + * of the rows that were deleted. It is a backup of part of the secrets table + * and must be handled as one. + */ + warning: string; + /** Per-family status of the union the decision was made on. */ + families: Record; + /** The rows about to be deleted, with the cipher material. */ + rows: ExportedSecretRow[]; + /** + * Why each id was judged deletable, and the holder evidence at the time — + * the reasoning the digest-only audit trail cannot reconstruct. + */ + decisions: Array<{ id: string; reason: string; holders: string[] }>; +} + +/** How to put a row back. Written into the export so the file is self-describing. */ +export const EXPORT_RESTORE_HINT = + 'To restore a row, re-insert it into `sys_secret` with every column below verbatim — the id ' + + 'included, since the id IS the handle every holder column names. Restoring the row does not ' + + 'restore any reference to it.'; + +/** + * Build the export document for the rows a `--delete` run is about to remove. + * + * Pure, and deliberately driven by the PLAN's deletable list rather than by a + * caller-supplied id set: the export and the delete then cannot disagree about + * which rows are in scope. A deletable id with no raw row available is a hard + * error rather than a silently shorter export — a delete whose export is + * missing a row is precisely the un-nameable outcome the export exists to + * prevent. + */ +export function buildPreDeleteExport(input: { + plan: SysSecretSweepPlan; + /** The raw `sys_secret` rows, keyed by id, as read from the driver. */ + rawById: ReadonlyMap>; + producedBy: string; + now?: () => Date; +}): PreDeleteExport { + const { plan, rawById } = input; + const rows: ExportedSecretRow[] = []; + const decisions: PreDeleteExport['decisions'] = []; + + for (const id of plan.deletable) { + const raw = rawById.get(id); + if (!raw) { + throw new Error( + `Cannot build the pre-delete export: no raw sys_secret row was read for '${id}', which the ` + + 'plan lists as deletable. Refusing to delete a row the export cannot record — the audit ' + + 'trail holds digests rather than handles, so it could never be named afterwards.', + ); + } + const swept = plan.rows.find((r) => r.id === id); + rows.push({ + id: String(raw.id), + namespace: String(raw.namespace), + key: String(raw.key), + kms_key_id: (raw.kms_key_id as string | null | undefined) ?? null, + alg: (raw.alg as string | null | undefined) ?? null, + version: (raw.version as number | null | undefined) ?? null, + ciphertext: (raw.ciphertext as string | null | undefined) ?? null, + created_at: (raw.created_at as string | null | undefined) ?? null, + rotated_at: (raw.rotated_at as string | null | undefined) ?? null, + }); + decisions.push({ + id, + reason: swept?.reason ?? 'deletable', + holders: swept?.holders ?? [], + }); + } + + return { + format: 'objectstack.sys_secret.pre-delete-export.v1', + generatedAt: (input.now?.() ?? new Date()).toISOString(), + producedBy: input.producedBy, + warning: + 'This file contains the CIPHER MATERIAL of the sys_secret rows that were deleted. It is a ' + + 'partial backup of the secrets table: protect it exactly as you protect a database backup, ' + + 'and delete it only once you are certain the sweep was correct. ' + + EXPORT_RESTORE_HINT, + families: plan.families, + rows, + decisions, + }; +} diff --git a/packages/cli/test/json-stdout-purity.e2e.test.ts b/packages/cli/test/json-stdout-purity.e2e.test.ts index b70b9423a0..82aed8ccfd 100644 --- a/packages/cli/test/json-stdout-purity.e2e.test.ts +++ b/packages/cli/test/json-stdout-purity.e2e.test.ts @@ -89,6 +89,10 @@ const FAMILY: Record = { 'migrate resume': [], 'migrate summary-nulls': [], 'migrate value-shapes': [], + // Report-only is its DEFAULT and the only form driven here: without + // `--delete` it boots, reports and writes nothing, so the family gains a + // member without this fixture gaining a destructive run. + 'secret orphans': [], 'storage orphans': [], };