Simplify operational tooling and runbooks by Copilot · Pull Request #62 · BigSimmo/Database

Copilot · 2026-06-23T04:39:40Z

Recovery scripts emitted raw JSON, required manual --apply flags with no interactive guidance, and the reindex runbook demanded 8+ hand-executed steps in strict order — all friction that increases operational error risk.

Changes

`scripts/cli-utils.ts` (new)

Shared confirm() utility using Node.js readline — single source for interactive yes/no prompts.

`scripts/recover-ingestion-queue.ts`

Replaces JSON dump with aligned human-readable summary (scanned jobs, resets, supersedes, retries)
Adds interactive confirmation prompt before any mutations when --apply is absent
--yes skips the prompt for CI/piped use; --apply remains backward-compatible

=== Ingestion Queue Recovery ===
Checking Supabase health...
Supabase is healthy.
Stale-after threshold : 45 min
Action limit : 20
Scanned jobs : 5
Documents to reset : 2
Jobs to supersede : 1
Jobs to retry : 1
Apply these changes? (y/N)

`scripts/supabase-recovery-status.ts`

Replaces JSON dump with a readable dashboard: status, checked-at, count table, any read errors, and a plain-English recommendation line.

`scripts/reindex.ts` + `npm run reindex` (new)

Consolidated pipeline that replaces the 8-step manual runbook:

Confirms target Supabase project ref
Probes Supabase health
Snapshots reindex health (document/job counts)
Applies queue recovery if stale/failed jobs present (interactive prompt)
Runs worker:once
Repeats until queue is clear or --max-rounds is reached

Flags: --yes (non-interactive), --max-rounds N (default 10), --limit N (recovery actions/round, default 20).

`docs/reindex-runbook.md`

Documents npm run reindex as the primary path with a flags table; retains the manual sequence for reference.

Verification

npm run verify:cheap
npm run verify:ui when UI, routing, styling, browser behavior, reduced-motion, or forced-colors behavior changed
npm run verify:release before release or handoff confidence claims
npm run format:check
npm run check:production-readiness when clinical workflow, privacy, environment, Supabase, source governance, or deployment behavior changed

Clinical Governance Preflight

Complete this section when the change touches ingestion, answer generation, search/ranking, source rendering, document access, privacy, production env, or clinical output.

Source-backed claims still require linked source verification before clinical use
No patient-identifiable document workflow was introduced or expanded without explicit governance approval
Supabase target remains Clinical KB Database (sjrfecxgysukkwxsowpy)
Service-role keys and private document access remain server-only
Demo/synthetic content remains clearly separated from real clinical sources
Source metadata, review status, and outdated/unknown-source behavior remain conservative
Deployment classification/TGA SaMD impact was checked when clinical decision-support behavior changed

Notes

RawJobRow type is kept inline per-script (it maps a Supabase query shape, not a domain type) rather than promoted to @/lib/ingestion-recovery
No mutations to core ingestion logic; all changes are to operator-facing CLI scripts

- recover-ingestion-queue.ts: replace JSON output with human-readable summary; add interactive confirmation prompt (--yes / --apply to skip); extract confirm() to shared cli-utils.ts - supabase-recovery-status.ts: replace JSON output with readable dashboard - scripts/reindex.ts: new consolidated pipeline command that encapsulates the full reindex runbook (health check, recovery, worker, repeat) - package.json: add 'reindex' npm script - docs/reindex-runbook.md: document the new npm run reindex command

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:c559343bf0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:a7393deefb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:6735eea7fc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

…ational-tooling

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 6 comments.

+ const child = spawn("npx", ["tsx", "worker/index.ts", "--once"], {
+ stdio: "inherit",
+ shell: false,
+ });


+ const staleAfterCutoff = new Date(staleCutoff);
+ const jobs = (data ?? [])
+ .map((job: RawJobRow) => ({
+ ...job,
+ documents: Array.isArray(job.documents)
+ ? (job.documents[0] as RecoveryDocument | undefined)
+ : (job.documents as RecoveryDocument | undefined),
+ }))
+ .filter((job) => {
+ if (job.status === "failed") {
+ return true;
+ }
+ if (job.status !== "processing") {
+ return false;
+ }
+ if (job.locked_at === null) {
+ return true;
+ }
+ return new Date(job.locked_at) <= staleAfterCutoff;
+ });


+ rl.question(`${question} (y/N) `, (answer: string) => {
+ rl.close();
+ resolve(answer.trim().toLowerCase() === "y");
+ });


+ ".claude/**",
+ "**/.claude/**",
+ ".tmp-visual/**",
+ "**/.tmp-visual/**",


 3. Apply any pending local migration only after the probe succeeds.
 4. Run `npm run reindex:health`.
-5. If stale or failed jobs exist, run `npm run recover:ingestion -- --apply --limit 20`.
+5. If stale or failed jobs exist, run `npm run recover:ingestion` and confirm the prompt.
 6. Run `npm run worker:once` with the conservative defaults from `.env.example`.


+ const args = parseArgs(process.argv.slice(2));
+ const staleAfterMinutes = env.WORKER_STALE_AFTER_MINUTES;
+ const limit = args.limit;
+


chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:d7d3b464dc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-06-26T08:01:37Z

+ if (hasActiveProcessingJobs) {
+ console.log(
+ "\n Active processing jobs detected; skipping worker run to avoid concurrency spikes.",
+ );
+ continue;


Do not spend worker rounds while waiting
When a fresh processing job exists, this new guard immediately continues into the next loop iteration and counts it against --max-rounds without sleeping or running a worker. Because the runbook documents --max-rounds as the maximum worker iterations, npm run reindex -- --yes --max-rounds 5 can burn all five rounds in seconds and exit 1 while another worker is still making normal progress. Return with an operator message, or wait/back off without consuming a worker round.
Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-06-26T08:01:37Z

-## Safe sequence
+## Consolidated pipeline command
+
+The quickest way to run a full reindex cycle is the consolidated pipeline command:


Queue documents before calling this a full reindex
In the common release case where all documents are already indexed and there are no open ingestion_jobs, scripts/reindex.ts hits its queue-clear check and exits successfully without enqueueing any document for the new indexing code. Since this runbook is specifically for applying RAG indexing changes, describing npm run reindex as a full reindex cycle can leave the corpus on the old chunks/enrichment while operators proceed to evals; either add an enqueue step/command or label this as draining an already queued reindex.
Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-06-26T08:01:37Z

+ const child = spawn("npx", ["tsx", "worker/index.ts", "--once"], {
+ stdio: "inherit",
+ shell: false,
+ });


Run the worker with recovery-safe concurrency
When WORKER_CONCURRENCY is not set in the environment, the worker spawned here falls back to the code default of 8 and processes the claimed jobs concurrently, while the recovery runbook/.env.example expect concurrency 1 for this operation. In that environment, npm run reindex -- --yes can fan out eight document reindexes during a supposedly conservative recovery run; set a safe env override for this child process or fail until conservative worker settings are present.
Useful? React with 👍 / 👎.

Initial plan
6a2dd93

CopilotAI assigned Copilot and BigSimmo Jun 23, 2026

Copilotstarted work on behalf of BigSimmo June 23, 2026 04:47 View session

CopilotAI changed the title ~~[WIP] Simplify operational tooling and runbooks~~Simplify operational tooling and runbooksJun 23, 2026

CopilotAI requested a review from BigSimmo June 23, 2026 04:57

Copilotfinished work on behalf of BigSimmo June 23, 2026 04:57

Merge branch 'main' into copilot/simplify-operational-tooling
c559343

BigSimmo marked this pull request as ready for review June 23, 2026 05:55

chatgpt-codex-connectorBot reviewed Jun 23, 2026
View reviewed changes

Comment threadscripts/reindex.ts
Comment threadscripts/reindex.ts Outdated
Comment threadscripts/reindex.ts Outdated
Comment threadscripts/cli-utils.ts

Fix reindex pipeline recovery edge cases
a7393de

CopilotAI review requested due to automatic review settings June 26, 2026 05:52

CopilotAI reviewed Jun 26, 2026

chatgpt-codex-connectorBot reviewed Jun 26, 2026
View reviewed changes

Comment threadscripts/reindex.ts
Comment threadscripts/reindex.ts Outdated
Comment threadscripts/reindex.ts Outdated
Comment threadscripts/reindex.ts Outdated

chore(reindex): harden worker recovery and completion paths
6735eea

chatgpt-codex-connectorBot reviewed Jun 26, 2026
View reviewed changes

Comment threadscripts/reindex.ts Outdated
Comment threadscripts/reindex.ts Outdated

Ignore Claude and visual temp files in ESLint
34de6f1

CopilotAI review requested due to automatic review settings June 26, 2026 06:51

CopilotAI reviewed Jun 26, 2026

BigSimmo added 3 commits June 26, 2026 14:55

Merge remote-tracking branch 'origin/main' into copilot/simplify-oper…
3c29f7b
…ational-tooling

Update work
64411cf

fix(reindex): tighten recovery completion checks
d7d3b46

CopilotAI review requested due to automatic review settings June 26, 2026 07:50

Copilotstarted reviewing on behalf of BigSimmo June 26, 2026 07:51 View session

BigSimmo merged commit 17fb704 into mainJun 26, 2026
5 checks passed

CopilotAI reviewed Jun 26, 2026
View reviewed changes

chatgpt-codex-connectorBot reviewed Jun 26, 2026
View reviewed changes

cursorBot mentioned this pull request Jul 24, 2026
docs: restore confirmed universal ledger tasks #1115
Closed
8 tasks

BigSimmo mentioned this pull request Jul 24, 2026
Fix Database↔Supabase interface audit findings (#052, #062, schema hygiene, table-facts latency) #1133
Merged
10 tasks

BigSimmo mentioned this pull request Jul 24, 2026
docs: close missing relevance issue #1136
Merged

Conversation

CopilotAI commented Jun 23, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

scripts/cli-utils.ts (new)

scripts/recover-ingestion-queue.ts

scripts/supabase-recovery-status.ts

scripts/reindex.ts + npm run reindex (new)

docs/reindex-runbook.md

Verification

Clinical Governance Preflight

Notes

Uh oh!

chatgpt-codex-connectorBot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connectorBot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connectorBot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

chatgpt-codex-connectorBot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connectorBotJun 26, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connectorBotJun 26, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connectorBotJun 26, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

CopilotAI commented Jun 23, 2026•
edited
Loading

`scripts/cli-utils.ts` (new)

`scripts/recover-ingestion-queue.ts`

`scripts/supabase-recovery-status.ts`

`scripts/reindex.ts` + `npm run reindex` (new)

`docs/reindex-runbook.md`