Skip to content

test(eval): force-embedding flag + 10 vector-exercising golden cases#249

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
BigSimmo merged 3 commits into from
Jul 3, 2026
Merged

test(eval): force-embedding flag + 10 vector-exercising golden cases #249

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
4f03aef
test(eval): add force-embedding flag + 10 vector-exercising golden cases
BigSimmo Jul 3, 2026
b62f516
test(eval): address Codex review — isolate vector, key cache, propaga…
BigSimmo Jul 3, 2026
3bdb703
Merge branch 'main' into claude/vector-golden-cases
BigSimmo Jul 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions scripts/eval-quality.ts
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -753,6 +753,7 @@ async function runRetrievalQualityCases(args: {
topK: testCase.topK,
minSimilarity: 0.12,
skipCache: true,
forceEmbedding: testCase.forceEmbedding,

@chatgpt-codex-connectorchatgpt-codex-connectorBotJul 3, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Gate release eval on forced misses

Forwarding forceEmbedding here makes eval:quality:release run the new vector probes, but that command still exits based only on the aggregate 0.8 retrieval thresholds in buildEvalQualityReport; with 34 cases, six forced vector misses can still leave hit/recall at 28/34 = 0.823 and pass release. If these cases are meant to protect re-index quality, forced-case failures (or any golden failed_cases) need to be blocking rather than only reported.

Useful? React with 👍 / 👎.

}),
);
const latencyMs =
Expand Down
11 changes: 11 additions & 0 deletions scripts/eval-retrieval.ts
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -19,6 +19,10 @@ const goldenCaseSchema = z.object({
expectedContentTerms: z.array(contentExpectationSchema).default([]),
topK: z.number().int().positive().default(8),
expectTableEvidence: z.boolean().default(false),
// Bypass the lexical text-fast-path so this case always exercises the embedding/vector
// index. Use for "vector-*" probes that would otherwise be answered by a lexical shortcut,
// so a re-index's effect on vector retrieval is actually measured.
forceEmbedding: z.boolean().optional(),
});

const goldenCasesSchema = z.array(goldenCaseSchema);
Expand All@@ -37,6 +41,7 @@ type EvalArgs = {
caseTimeoutMs: number;
p90BudgetMs: number;
p50BudgetMs: number;
forceEmbedding: boolean;
};

export type GoldenRetrievalResult = {
Expand DownExpand Up@@ -116,6 +121,7 @@ function parseArgs(argv: string[]): EvalArgs {
caseTimeoutMs: inferredMode === "latency" ? 25_000 : 0,
p90BudgetMs: 20_000,
p50BudgetMs: 8_000,
forceEmbedding: false,
};

for (let index = 0; index < argv.length; index += 1) {
Expand All@@ -138,6 +144,10 @@ function parseArgs(argv: string[]): EvalArgs {
if (args.caseTimeoutMs <= 0) args.caseTimeoutMs = 25_000;
continue;
}
if (token === "--force-embedding") {
args.forceEmbedding = true;
continue;
}

const value = argv[index + 1];
if (!value || value.startsWith("--")) throw new Error(`Missing value for ${token}`);
Expand DownExpand Up@@ -743,6 +753,7 @@ async function main() {
topK: testCase.topK,
minSimilarity: 0.12,
skipCache: args.mode !== "latency",
forceEmbedding: testCase.forceEmbedding || args.forceEmbedding,
}),
);
const searchOutcome = await withCaseTimeout(searchPromise, args.caseTimeoutMs);
Expand Down
103 changes: 103 additions & 0 deletions scripts/fixtures/rag-retrieval-golden.json
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -224,5 +224,108 @@
],
"topK": 12,
"expectTableEvidence": true
},
{
"id": "vector-ptsd",
"query": "How is post-traumatic stress disorder managed in a person with intrusive flashbacks, nightmares, hyperarousal and avoidance after a traumatic event?",
"expectedQueryClass": "broad_summary",
"expectedDocumentSubstrings": ["Traumatic Stress"],
"expectedContentTerms": [["trauma", "traumatic", "ptsd", "flashback"]],
"topK": 8,
"expectTableEvidence": false,
"forceEmbedding": true
Comment thread
BigSimmo marked this conversation as resolved.
},
{
"id": "vector-ocd",
"query": "How is obsessive-compulsive disorder managed in a person with distressing intrusive obsessions and repetitive compulsive rituals?",
"expectedQueryClass": "broad_summary",
"expectedDocumentSubstrings": ["Obsessive"],
"expectedContentTerms": [["obsess", "compuls", "intrusive", "ocd"]],
"topK": 8,
"expectTableEvidence": false,
"forceEmbedding": true
},
{
"id": "vector-panic",
"query": "How is panic disorder managed in a person with recurrent unexpected panic attacks, palpitations and anticipatory anxiety?",
"expectedQueryClass": "broad_summary",
"expectedDocumentSubstrings": ["Panic"],
"expectedContentTerms": [["panic", "attack", "anxiety"]],
"topK": 8,
"expectTableEvidence": false,
"forceEmbedding": true
},
{
"id": "vector-anorexia",
"query": "How is anorexia nervosa managed in a person with severe dietary restriction, intense fear of weight gain and body-image disturbance?",
"expectedQueryClass": "broad_summary",
"expectedDocumentSubstrings": ["Anorexia"],
"expectedContentTerms": [["anorexia", "eating", "weight"]],
"topK": 8,
"expectTableEvidence": false,
"forceEmbedding": true
},
{
"id": "vector-gad-worry",
"query": "How is a patient managed who has persistent, excessive worry about many everyday things that they find very hard to control, along with restlessness and muscle tension?",
"expectedQueryClass": "broad_summary",
"expectedDocumentSubstrings": ["Generalised Anxiety"],
"expectedContentTerms": [
["worry", "anxiety", "generalised"],
["cbt", "ssri", "snri", "antidepressant", "psychotherapy"]
],
"topK": 8,
"expectTableEvidence": false,
"forceEmbedding": true
},
{
"id": "vector-tourette",
"query": "How is Tourette syndrome managed in a child with chronic motor tics and vocal tics?",
"expectedQueryClass": "broad_summary",
"expectedDocumentSubstrings": ["Tourette"],
"expectedContentTerms": [["tic", "tics", "tourette"]],
"topK": 8,
"expectTableEvidence": false,
"forceEmbedding": true
},
{
"id": "vector-postnatal",
"query": "How is postnatal (postpartum) depression managed in a new mother who develops low mood and poor bonding with her infant in the first weeks postpartum?",
"expectedQueryClass": "broad_summary",
"expectedDocumentSubstrings": ["Postnatal"],
"expectedContentTerms": [["postnatal", "postpartum", "perinatal", "depression"]],
"topK": 8,
"expectTableEvidence": false,
"forceEmbedding": true
},
{
"id": "vector-bipolar",
"query": "How is bipolar disorder managed in an adult with recurrent episodes of mania and depression?",
"expectedQueryClass": "broad_summary",
"expectedDocumentSubstrings": ["Bipolar"],
"expectedContentTerms": [["bipolar", "mania", "manic", "mood"]],
"topK": 8,
"expectTableEvidence": false,
"forceEmbedding": true
},
{
"id": "vector-adhd",
"query": "How is attention deficit hyperactivity disorder managed in an adult with inattention, distractibility and impulsivity?",
"expectedQueryClass": "broad_summary",
"expectedDocumentSubstrings": ["Attention Deficit Hyperactivity"],
"expectedContentTerms": [["attention", "adhd", "hyperactiv", "impuls"]],
"topK": 8,
"expectTableEvidence": false,
"forceEmbedding": true
},
{
"id": "vector-opioid",
"query": "How is opioid use disorder managed in a person dependent on heroin?",
"expectedQueryClass": "broad_summary",
"expectedDocumentSubstrings": ["Opioid Use Disorder"],
"expectedContentTerms": [["opioid", "heroin", "opiate", "methadone", "buprenorphine"]],
"topK": 8,
"expectTableEvidence": false,
"forceEmbedding": true
}
]
24 changes: 18 additions & 6 deletions src/lib/rag.ts
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -307,6 +307,10 @@ export type SearchChunksArgs = {
// Internal: set when this call is a re-run on a trigram-corrected query, to prevent the
// unsupported-short-circuit typo-correction path from recursing more than once.
typoCorrected?: boolean;
// Diagnostic/eval-only: bypass every lexical text-fast-path so retrieval always exercises
// the embedding/vector stage. Lets the golden eval measure the vector index directly for a
// re-index, instead of being masked by lexical shortcuts. Never set on production paths.
forceEmbedding?: boolean;
};

export type AnswerProgressEvent = {
Expand DownExpand Up@@ -1429,7 +1433,7 @@ function stableHash(value: string) {
export function retrievalPlanCacheQuery(
args: Pick<
SearchChunksArgs,
"query" | "documentId" | "documentIds" | "ownerId" | "queryMode" | "topK" | "minSimilarity"
"query" | "documentId" | "documentIds" | "ownerId" | "queryMode" | "topK" | "minSimilarity" | "forceEmbedding"
>,
queryClass?: RagQueryClass,
queryVariants: string[] = [],
Expand All@@ -1445,6 +1449,7 @@ export function retrievalPlanCacheQuery(
`topK:${args.topK ?? 8}`,
`min:${args.minSimilarity ?? 0.15}`,
`rag:${ragDeepMemoryVersion}`,
`force:${args.forceEmbedding ? 1 : 0}`,
].join("|");
return queryCacheKeyForStorage(cacheKey);
}
Expand DownExpand Up@@ -5516,7 +5521,7 @@ export async function searchChunksWithTelemetry(args: SearchChunksArgs) {
});

const baseTextFastPath = decideTextFastPath(args.query, baseTextResults, queryClassification.queryClass);
if (shouldReturnBeforeMemory(queryClassification.queryClass, baseTextFastPath)) {
if (!args.forceEmbedding && shouldReturnBeforeMemory(queryClassification.queryClass, baseTextFastPath)) {
Comment thread
BigSimmo marked this conversation as resolved.
textFastResults = await attachPageVisualEvidence(supabase, baseTextResults);
textFastResults = applySecondStageRerankIfNeeded({
queryClass: queryClassification.queryClass,
Expand DownExpand Up@@ -5567,7 +5572,7 @@ export async function searchChunksWithTelemetry(args: SearchChunksArgs) {
telemetry.rerank_latency_ms += Date.now() - rerankStartedAt;

const boostedTextFastPath = decideTextFastPath(args.query, textFastResults, queryClassification.queryClass);
if (boostedTextFastPath.returnFastPath) {
if (!args.forceEmbedding && boostedTextFastPath.returnFastPath) {
Comment thread
BigSimmo marked this conversation as resolved.
markEmbeddingSkippedByTextFastPath(telemetry, boostedTextFastPath.reason);
telemetry.retrieval_strategy = "text_fast_path";
recordSearchScoreTelemetry(telemetry, textFastResults);
Expand DownExpand Up@@ -5673,7 +5678,7 @@ export async function searchChunksWithTelemetry(args: SearchChunksArgs) {
documentLookupResults,
queryClassification.queryClass,
);
if (documentLookupFastPath.returnFastPath) {
if (!args.forceEmbedding && documentLookupFastPath.returnFastPath) {
markEmbeddingSkippedByTextFastPath(
telemetry,
documentLookupFastPath.reason ? `document_lookup_fast_path:${documentLookupFastPath.reason}` : null,
Expand All@@ -5700,7 +5705,7 @@ export async function searchChunksWithTelemetry(args: SearchChunksArgs) {
});
const coverageGate = evaluateEvidenceCoverageGate(args.query, coverageGateResults, queryClassification.queryClass);
applyCoverageGateTelemetry(telemetry, coverageGate, coverageGate.accepted);
if (coverageGate.accepted) {
if (!args.forceEmbedding && coverageGate.accepted) {
telemetry.retrieval_strategy = coverageGate.strategy;
recordSearchScoreTelemetry(telemetry, coverageGateResults);
setCachedSearch(args, coverageGateResults, telemetry, queryVariants);
Expand DownExpand Up@@ -5744,6 +5749,13 @@ export async function searchChunksWithTelemetry(args: SearchChunksArgs) {
latencyMs: telemetry.embedding_latency_ms,
});

if (args.forceEmbedding) {

@chatgpt-codex-connectorchatgpt-codex-connectorBotJul 3, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Fail forced evals when embeddings fail

Because the lexical candidates are cleared only after embedTextWithTelemetry succeeds, the default RAG_PROVIDER_MODE=auto failure path above this block still returns the pre-embedding lexical textFastResults on quota/rate-limit/transient embedding errors. In that scenario a forced vector-* eval can pass without exercising any vector retrieval, which defeats the new guard; when forceEmbedding is set, the embedding failure should fail the search or at least not fall back to lexical results.

Useful? React with 👍 / 👎.

// Force-embedding eval isolation: drop the lexical / memory-card / table candidates gathered
// before embedding so the returned results reflect the embedding-driven retrieval layers only
// (otherwise a broken vector index could still be masked by the lexical text candidate path).
textFastResults = [];
}

// A1: the embedding-field, index-unit, and chunk-hybrid RPCs each depend only on the
// already-computed query embedding and have no data dependency on one another, so run
// them concurrently instead of as three sequential Supabase round-trips. The two helper
Expand DownExpand Up@@ -5780,7 +5792,7 @@ export async function searchChunksWithTelemetry(args: SearchChunksArgs) {
const startedAt = Date.now();
const { data, error } = await supabase.rpc("match_document_chunks_hybrid", {
query_embedding: embedding as unknown as string,
query_text: textSearchQuery,
query_text: args.forceEmbedding ? "" : textSearchQuery,

@chatgpt-codex-connectorchatgpt-codex-connectorBotJul 3, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Make all forced RPCs vector-only

Fresh evidence after the prior isolation fix: this only blanks query_text for match_document_chunks_hybrid, while the same forced path still calls searchEmbeddingFieldCandidates and searchIndexUnitCandidates with buildClinicalTextSearchQuery(args.query). Those hybrid RPCs select/boost lexical text hits, so a vector-* golden case can still pass through embedding-field or index-unit text branches even if the chunk vector index/regeneration is broken; propagate the forced mode to those helpers or assert vector-only provenance before treating these cases as vector probes.

Useful? React with 👍 / 👎.

match_count: candidateCount,
min_similarity: minSimilarity,
document_filters: documentFilterList ?? undefined,
Expand Down
Loading