Uh oh!
There was an error while loading. Please reload this page.
translation bench - #2831
Draft
Dominic Nguyen (datduyng) wants to merge 80 commits into
Draft
Conversation
Empty expectedActions is scored as zero actions. Stop minting contrastive adjacent commands and refuse-then-alternate forms as empty-gold negatives; format_checker now hard-rejects them as BAD_NEGATIVE. Allowed negatives: pure refusal, non-action status/howto questions, and missing-info clarifications. Align synthesizer and quality-verifier prompts with the zero-action contract; add unit coverage including adversarial holes.
- Drop ACTION_VP / refusal regex classifier (unmaintainable verb lists). - Require semantic_checker negativeAssessments (kind + fairEmptyGold). - Code hard-fails unfair assessments as BAD_NEGATIVE; format stays structural. - Tests cover assessment parse/enforce and mock-LLM semantic gate.
- Single zod schema; derive OpenAI JSON schema via z.toJSONSchema - Match assessments by order (equal count); drop path-index maps - Simplify force-reject merge; strip explanatory comments
- Drop dead exports; parse via shared parseWithZod - Derive OpenAI schema from zod without void/\$schema clutter - Add test that assessments pair to negatives by order
- Join assessments to negatives by exact genCase path (bijective set); reject unknown/duplicate/missing paths as BAD_NEGATIVE instead of silent index pairing. - Parse reviewer decision before assessments so structured reject issues survive missing/invalid negativeAssessments. - Share TRANSLATION_BENCH_NEGATIVE_FAIRNESS_RULE in synthesizer context; tighten howto/soft-solicit unfair rules and untrusted payload framing. - Extend reviewerDecision mocks with path-keyed fair assessments; replace order-only path tests with multi-negative path-join coverage.
- Fixed short PATH_MSG/FIX only; use LLM reason as issue message - Drop rewrite-hint and per-case string assembly - Keep path 1:1 cover + fairEmptyGold/kind gate
- Honor concurrency with a worker pool and serialized checkpoint commits
- Keep parameters:{} after stripEmpty for required empty-object schemas
- Inject constant string-union fields (e.g. settings id) during gold validate
- Isolate per-slot generation failures so other workers keep committing- Fair empty gold is pure_refusal/leave-alone only - Reject definition/status/Q&A and missing_info as empty gold - Synthesizer + quality-verifier prompts match any-tool abstain test - Extend unit coverage for definition Q and soft-solicit rejects
Join assessments to genCases via a path→case map built from the same keys the checker emits, then read dimensions.negativeKind directly.
…params - Allow incomplete case sets when requireCompleteCoverage is false - Prompt: nested objects must carry required schema fields (e.g. timeRange)
…ial gen - Partial-gen branch previously emitted planned caseCount/coverage, so validateTranslationBenchBenchmark always threw and the branch was unreachable dead code - Recompute scheduledActionCount, complete, and caseCount from the cases actually emitted; happy-path (complete) output is unchanged - Makes requireCompleteCoverage=false produce a valid draft directly
Derive FAIR_KINDS from TRANSLATION_BENCH_FAIR_EMPTY_GOLD_KINDS so the allowlist cannot drift from the exported constant.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a fairness contract ensuring empty-gold translation-bench negatives represent only pure refusals.
Changes:
- Adds path-keyed LLM fairness assessments and enforcement.
- Updates generation/verifier prompts and gold-action validation.
- Supports concurrent generation and accurate partial-result metadata.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
README.AUTOGEN.md | Refreshes generated documentation metadata. |
actionValidation.ts | Normalizes gold actions for schema validation. |
benchmark.ts | Uses normalized gold-action validation. |
dataQualityVerifier.ts | Enforces semantic negative-fairness assessments. |
datasetGenerator.ts | Adds concurrency and partial-generation handling. |
generationCandidate.ts | Preserves required parameter objects during cleanup. |
index.ts | Exports the fairness module. |
negativeFairness.ts | Implements fairness parsing and enforcement. |
quality-verifier.prompt.yaml | Defines full-catalog fairness review requirements. |
synthesizer.prompt.yaml | Restricts empty-gold negatives to pure refusals. |
utteranceDisambiguation.ts | Documents separate negative handling. |
translationBench.datasetGenerator.spec.ts | Updates generation fixtures for fairness assessments. |
translationBench.negativeFairness.spec.ts | Tests fairness validation and semantic enforcement. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
…s, ambiguous-route guard Row-by-row 3-model review of the 1k eval showed most all-models "failures" were dataset/scoring fairness issues, not real misses. Address the generator-side ones: - Wire parameterScore specs into every generated case at finalize (`parameterScoreSpecsForExpectedActions`). Free-text echo params such as `originalRequest` and setting free-text now score `nonempty` instead of exact, matching the grader's own classification. Canonical payload hash excludes parameterScore, so dataset identity is preserved. - Add `HARDCODED_NON_EVAL_ACTION_IDS` (`chat.generateResponse`, `utility.claudeTask`) as a single source of truth, unioned into the packaged exclusion set so they are never targeted. - Drop cross-schema duplicate action names from targeting: when the same bare action name is owned by more than one schema (e.g. `deleteWebFlow`), the single gold route is ambiguous, so exclude every sibling. Tests: parameterScore wiring + canonical-hash stability, non-eval exclusion set, and ambiguous cross-schema guard. Full benchmarks suite green (138).
…eckpoint, param-spec cleanup Code-review + Copilot follow-ups on TB generation fairness: - negativeFairness: require dimensions.negativeKind to exactly equal the accepted empty-gold assessment kind. Previously a fair pure_refusal assessment was accepted even when negativeKind was missing/numeric/boolean, letting unlabeled empty-gold rows through. Now the label must match. - datasetGenerator: persist the checkpoint row BEFORE mutating casesBySlot / usedUtterances in commitAccepted, so an I/O failure can no longer leave an uncheckpointed case that the partial-coverage path would return. - Unify the duplicated param-field-mode union: grader now imports TranslationBenchParamFieldMode / TranslationBenchParameterScoreSpec from benchmark instead of re-declaring RunnerParamFieldMode; collapse the spec derivation to Object.fromEntries. - benchmark: enforce parameterScore aligns 1:1 with expectedActions in the probe payload schema (validateProbePayload). - Extract ambiguousCrossSchemaActionIds helper; drop the size===0 ternary. Tests: new integration coverage for generateTranslationBenchBenchmark (concurrent full run + partial run past a failed slot, asserting checkpoint contents and coverage); negativeKind-missing rejection; fixtures now label negatives pure_refusal. Full suite green (141).
George Ng (GeorgeNgMsft)
approved these changes
Aug 9, 2026
- Seed 24 empirically-mined cross-schema collision pairs into KNOWN_CONFUSABLE_PAIRS (browser tab routes, code/utility file ops, visualStudio/code-debug, desktop/display settings, player queues, etc.) - Add generic cross-schema near-duplicate detector in findTranslationBenchConfusableSiblings: flags equivalent actions in different schemas when BOTH action-name token overlap >=0.5 AND description overlap >=0.34, so shared generic verbs alone do not over-flag unrelated actions - Add significantTokensFromText helper for description-token overlap - Extend unit tests with cross-schema newTextFile<->writeFile case plus a readFile negative control Root cause: synthesizer scheduled every catalog action as a gold target but only compared same-schema siblings for disambiguation, so cross-schema near-synonyms produced no disambiguation constraint and yielded ambiguous-gold cases the models unanimously routed elsewhere.
- Move catalog/grader generation into translationBench/policy - Add action-eligibility hard bans + LLM quality picker (model required) - Ship eligible-gold-actions.generated.json; fail-closed load/integrity - Schedule lattice + eval pin allowlist hash; copyAssets requires assets - Tests for picker, policy, nested llmAsAJudge, schedule allowlist-on
…ypeAgentQuestion from eligible actions
Dominic Nguyen (datduyng)force-pushed
the
domnguyen/tb-negative-fairness
branch
from
August 20, 2026 23:58
dff6546 to
805f6a3Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a production translation benchmark pipeline:
tb-generateandtb-evalcommands with checkpoints, reports, scoring, and an ambiguity probe.Verification
pnpm --filter @typeagent/benchmarks buildpnpm run prettierints/packages/benchmarks