fix(experiments): hypothesis skill semantic validity + JSON output mode (EXPT-4644)

[hsadhvani]

Summary

Fixes over-lenient grading where a structurally-valid but nonsense hypothesis ("If apple pie, then elephant, because purple") scored 3/3 "Looks strong" on the Fast path (EXPT-4644; the Agent/Sonnet path already rejected it). Adds a semantic-validity rule: a slot counts only if its content is genuinely that component, and the change / measurement / rationale must be causally connected; If/then/because filled with non-sequiturs routes to junk.

Also adds a structured output mode: when the caller requests JSON only (the experiment builder's headless entry point), the skill emits the machine contract (schema_version / route / components / hypothesis-with-holes / measurements), matching the o11y experiment-hypothesis skill. The conversational flow and handoff payload are unchanged. This is the shared, portable structured section so the o11y skill can pull from this one later.

Targets feat/experiment-hypothesis-builder (stacks on the in-progress skill).

Note: the skill validator currently fails on a pre-existing name vs directory mismatch (...-assistant vs ...-builder), unrelated to this change.

Testing approaches

• No unit tests (skill prompt content); graded behavior validated against the EXPT-4644 nonsense repro. The o11y-side change carries the deterministic contract test.

Feature flags

• N/A

---

Note

Low Risk
Prompt-only skill documentation with no runtime or auth changes; main risk is inconsistent LLM grading until validated by existing o11y contract tests.

Overview
Fixes over-lenient grading where grammatically correct If/then/because text could score 3/3 despite nonsense. A new semantic validity rule says Change / Measurement / Rationale slots count only when the content is genuinely that component and causally linked; non-sequiturs are treated as absent and routed to junk instead of “Looks strong.”

Adds structured output mode for headless callers (experiment builder): when JSON-only is requested, the skill must return a single object (schema_version, route, components, hypothesis with {{component:hint}} holes, measurements) matching the o11y experiment-hypothesis contract. Conversational flow and the existing handoff payload are unchanged.

^{Reviewed by Cursor Bugbot for commit 747caee. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

Skill eval results

Skill	Before	After	Δ
agentcontrol/configs-create	100/100 (4/4)	100/100 (4/4)	no change
agentcontrol/configs-update	80/100 (4/5)	80/100 (4/5)	no change
agentcontrol/configs-variations	80/100 (4/5)	80/100 (4/5)	no change
agentcontrol/tools	75/100 (3/4)	75/100 (3/4)	no change
experiments/launchdarkly-experiment-hypothesis-builder	-	0/100 (0/5)	new
feature-flags/launchdarkly-flag-command	-	100/100 (3/3)	new
feature-flags/launchdarkly-flag-create	100/100 (3/3)	100/100 (3/3)	no change

Only suites whose source actually changed since their last recorded score were re-run. Soft-failing while we stabilise the baseline.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 8c6daf4. Configure here.}