Conversation
Two independent breakages on this branch turned 4 checks red. Neither came from #139 — it only touched SKILL.md, which is what diff-changed-skills.js uses to decide the suite must re-run, so it surfaced both. validate: SKILL.md declared name: launchdarkly-experiment-hypothesis-assistant while the directory is ...-builder, and validate_skills.py requires the two to match. Landed in 7e486ab (v0.3.0). Everything else in the repo already said -builder (_manifest.js, the evals/<suite> dir, the provider's skill_slug, the eval:hypothesis-builder script), so the frontmatter was the lone outlier. Renamed it, plus the handoffFrom value where the skill names itself — the only other occurrence in the repo, with no consumer matching the literal. This also realigns the eval's forced invocation: the provider invokes /<skill_slug> = /launchdarkly-experiment-hypothesis-builder, which did not match the name the skill declared. Evaluate suite: all 5 tests errored in 0s with "Template render error [Line 190, Column 90] expected variable end" — none reached an API call. The suite registered SKILL.md as promptfoo's prompt, promptfoo renders prompts through nunjucks, and SKILL.md documents its own hole syntax ({{measurement:...}}, {{component:hint}}) in the output-contract examples. Nunjucks reads `measurement` as a variable, hits the `:`, and throws. Landed in 849ee1e (#134), whose CI never ran these checks. Replaced the prompt with a suite-local function prompt that returns SKILL.md wrapped in {% raw %}. A function prompt does not skip nunjucks — renderPrompt assigns the return value to basePrompt and still calls renderString on it — and promptfoo's own autoWrapRawIfPartialNunjucks only fires on unclosed tags, so closed-but-invalid {{measurement:...}} sails through. Wrapping supplies the {% raw %} that helper would have added; renderString then returns the file byte-for-byte, so results.json still shows the exact skill text and SKILL.md on disk keeps its hole syntax byte-identical to the o11y contract. Kept suite-local rather than in evals/shared/, which is a GLOBAL_TRIGGERS entry and would force all 7 suites to re-run. Verified: validate_skills.py passes 46/46; nunjucks reproduces the exact Line 190 Column 90 failure on the bare file and renders the wrapped prompt byte-identically; npm test 70/70; one live suite test passes in 1m2s (was 5 errors in 0s). Aggregate scores and Evaluate gate need no separate fix — both were cascades of the errored suite.
ericoh-ld
changed the base branch from
main
to
feat/experiment-hypothesis-builder
August 3, 2026 19:25
ericoh-ld
marked this pull request as ready for review
August 3, 2026 19:44
Skill eval results
Only suites whose source actually changed since their last recorded score were re-run. Soft-failing while we stabilise the baseline. |
hsadhvani
approved these changes
Aug 3, 2026
ericoh-ld
merged commit Aug 3, 2026
0e81d12
into
feat/experiment-hypothesis-builder
14 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the 4 failing checks on
feat/experiment-hypothesis-builder. There were 2 independent root causes plus 2 cascades, none of which were caused by the changes toSKILL.mdin PR#139Checks and failures:
validate: frontmatter name ≠ directory name.SKILL.mddeclared name:launchdarkly-experiment-hypothesis-assistantwhile the directory is...-builder, andvalidate_skills.pyrequires the two to matchEvaluate launchdarkly-experiment-hypothesis-builder: nunjucks template render error. The suite registeredSKILL.mdas promptfoo's prompt, promptfoo renders prompts through nunjucks, andSKILL.mddocuments its own hole syntax ({{measurement:...}},{{component:hint}}) in the output-contract examples. Nunjucks reads measurement as a variable, hits the:, and throws the error.Aggregate scores: suite scored 0% → below the 75% threshold. These were outdated and not appropriate for the new version of the skill.Evaluate gate: suite scored 0% → the evaluate job failed. Cascades from the errored evaluations.Testing
Not applicable
Manual (describe below)
Rendered prompt vs SKILL.md: 27763 vs 27763 chars, byte-identical
Checked that
{% raw %}tags do not leak into the promptChecked that the hole syntax
{{measurement:is preservedpython3 scripts/validate_skills.py → 46/46 pass
npm test (evals) → 70/70 pass
promptfoo eval on the realigned suite → 10/10 pass
Note
Low Risk
Changes are limited to skill frontmatter/handoff strings and eval configuration; no runtime product logic or LaunchDarkly write paths are modified.
Overview
Unblocks CI for launchdarkly-experiment-hypothesis-builder by aligning skill metadata with the directory name and fixing promptfoo’s nunjucks render failure on hole syntax in
SKILL.md.Skill metadata: Frontmatter
nameand the handoff example’shandoffFromare renamed from…-assistantto…-buildersovalidate_skills.pypasses (name must match the parent folder).Eval suite redesign: The promptfoo config shifts from conversational E2E checks (MCP read tools,
ask-question, LLM rubrics, no-write assertions) to headless structured-output contract tests. A newprompt.jsloadsSKILL.mdwrapped in{% raw %}so{{measurement:…}}/{{component:hint}}examples are not parsed as nunjucks. The provider runs with MCP and ask-question disabled; a shared binarycontractassertion (weight 3) validates JSON-only replies against per-caseexpect(route, components, holes, measurements, echo rules). Cases cover A/A, junk, scaffold edge cases, XSS non-echo, unfalsifiable outcomes, and multi-measurement primary rules.Reviewed by Cursor Bugbot for commit 4889195. Bugbot is set up for automated code reviews on this repo. Configure here.