Skip to content

test(headless): split prompt A/B test boundaries - #346

Merged
Astro-Han merged 2 commits into
mainfrom
opencode/prompt-ab-test-boundary
Jun 28, 2026
Merged

test(headless): split prompt A/B test boundaries#346
Astro-Han merged 2 commits into
mainfrom
opencode/prompt-ab-test-boundary

Conversation

@Astro-Han

@Astro-HanAstro-Han commented Jun 28, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Split prompt A/B tests along the production module boundaries instead of keeping generic A/B coverage under the prompt wrapper.
  • Move generic summary assertions to ab-summary.test.ts and markdown rendering assertions to ab-render.test.ts.
  • Keep prompt-ab-summary.test.ts as a thin prompt id/report-title adapter test.
  • Split shared test helpers into focused files: ab-run-fixtures.ts, ab-summary-fixtures.ts, hash-fixture.ts, and temp-dir.ts.

Why

PR #340 intentionally left this as follow-up test organization work. prompt-ab-run.test.ts had grown into a mixed boundary covering generic A/B execution, prompt-specific manifest/resume behavior, source fingerprints, summary/rendering, and prompt run wrapper tests. Splitting it makes future A/B harness changes easier to review without changing production behavior.

Scope

Changed:

  • Added focused tests for generic A/B manifest, execution, summary, and markdown rendering.
  • Added focused tests for prompt A/B selection, manifest, source fingerprints, summary wrapper, and run wrapper behavior.
  • Replaced the old combined A/B fixture helper with smaller run, summary, hash, and temp-dir helpers.
  • Updated the hash fixture to generate real sha256:<64 hex> fingerprints.
  • Relaxed two resume tests that asserted incidental same-pair A/B start order after pair concurrency; they still assert the same arms are rerun or skipped.
  • Added a review-fix commit, 33f1fb02, to address the summary/render and helper-boundary review feedback.

Not included:

  • No production code changes.
  • No behavior changes to A/B execution, summaries, manifests, or reports.
  • No generated benchmark artifacts.

Verification

  • node ../../worktree-bootstrap.mjs
  • npm --workspace @maka/headless test

Result: 531 headless tests passed.

User-facing impact

None. This is test organization only. No changelog, docs, breaking changes, or migrations are included.

Reviewer notes

The PR now has two commits: the initial test-boundary split, plus a review-fix commit that moves generic summary/render coverage out of the prompt wrapper and splits the helper fixtures by test boundary.

@Astro-Han
Astro-Han merged commit 8b5606f into mainJun 28, 2026
@Astro-Han
Astro-Han deleted the opencode/prompt-ab-test-boundary branch June 28, 2026 11:39
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@Astro-Han