bug: LLM grader reports 'no response provided' when system prompt is present in input

[christso]

Bug Description

When running evals with a system prompt in the input messages, the LLM grader frequently reports "No response content was provided to evaluate" even when the model actually generated a correct response. This was discovered during benchmarking with with-superpowers vs without-superpowers experiments.

Reproduction

1. Create an eval with a system message followed by a user message:

tests:
 - id: test-1
 input:
 - role: system
 content: "You are a helpful assistant that thinks step by step..."
 - role: user
 content: "Solve this logic puzzle..."
 assertions:
 - type: llm-grader
 prompt: "Check if the response correctly solves the puzzle"

2. Run against multiple targets:

agentv eval test.EVAL.yaml --target azure --experiment with-system-prompt
agentv eval test.EVAL.yaml --target gemini --experiment with-system-prompt

3. Observe that the LLM grader returns low scores with reasoning like "No response content was provided to evaluate" even when the model's actual response is correct.

Observed behavior

Experiment	Target	Actual Response	Grader Score	Grader Reasoning
without-superpowers	gemini	Correct (A=Knave, B=Knave)	1.0	Correct analysis
with-superpowers	gemini	Correct (A=Knave, B=Knave)	0.5	"No response text was provided"
without-superpowers	azure	Correct	0.97	Good solution
with-superpowers	azure	Correct	0.0	"No solution steps were included"

The contains evaluator confirms the correct content IS present in the response. The issue is specific to the LLM grader's prompt construction when system messages are in the input.

Expected behavior

The LLM grader should correctly receive and evaluate the model's response regardless of whether the input includes system messages.

Environment

• agentv: latest main (post PR feat: experiment-based results layout and read-only Studio mode #977)
• Targets: azure (gpt-5.4-mini), gemini (gemini-3-flash-preview)
• Both used as grader and generator

Likely cause

The LLM grader prompt construction in packages/core/src/evaluation/evaluators/llm-grader.ts may not correctly handle multi-turn inputs with system messages when building the evaluation context for the judge model.

[christso]

Root Cause Analysis

Investigated the codebase — found two related bugs in the system message handling pipeline:

Bug 1: System message not extracted in buildPromptInputs

File: packages/core/src/evaluation/formatting/prompt-builder.ts (lines 80-90)

The buildPromptInputs function correctly includes system messages in the chatPrompt object, but never extracts them separately into the systemMessage field of the returned PromptInputs. The return statement is:

return { question, chatPrompt }; // Missing: systemMessage

Bug 2: System prompt passed in wrong location

File: packages/core/src/evaluation/orchestrator.ts (lines ~1220 and ~2673)

Even if systemMessage were populated, it's being placed in metadata instead of as a direct systemPrompt field:

metadata: {
 systemPrompt: promptInputs.systemMessage ?? '', // always '' due to Bug 1
},

This should be:

systemPrompt: promptInputs.systemMessage,

Effect Chain

1. Eval YAML includes role: system message in input
2. buildPromptInputs returns systemMessage: undefined
3. Orchestrator passes metadata.systemPrompt: ''
4. Provider invocation doesn't receive the system prompt
5. Something in the response handling gets confused with multi-turn inputs
6. LLM grader can't find the response content → "No response provided"

Evidence

From the benchmark results:

• without-superpowers (no system message): Grader scores correctly (gemini 100%, azure 73.5%)
• with-superpowers (has system message): Grader fails with "no response provided" even though the contains evaluator confirms correct content IS present

Proposed Fix

See the detailed code changes in the investigation notes. Two files need changes:

1. prompt-builder.ts — extract system messages and return them in PromptInputs
2. orchestrator.ts — pass systemPrompt as a direct field, not in metadata

[christso]

Updated Root Cause (from E2E investigation)

The initial analysis was incorrect. The bug is NOT caused by system message extraction. The real root cause is in resolveCustomPrompt() in packages/core/src/evaluation/evaluators/prompt-resolution.ts (lines 61-63):

const promptValue = promptConfig.prompt;
if (typeof promptValue === 'string') {
 return promptValue;
}

When an assertion has prompt: "Check if the response...", this returns the bare prompt text as evaluatorTemplateOverride, which completely replaces the DEFAULT_EVALUATOR_TEMPLATE. Since the bare prompt has no {{output}}, {{input}}, or {{criteria}} template variables, the grader never sees the question or response.

Evidence

• criteria: at test level → grader gets full template (662 input tokens) → score 1.0
• prompt: in assertion → grader gets bare text only (124 input tokens) → score 0.0 ("no response provided")
• This occurs with AND without system messages — system messages are irrelevant

The with-superpowers correlation was a red herring

The with-superpowers eval file happened to use prompt: in assertions while without-superpowers used criteria: at the test level. The failure was caused by the template replacement, not by system messages.

What PR #983 actually does

PR #983's changes (extracting systemMessage and passing systemPrompt on ProviderRequest) are a valid improvement but do NOT fix this bug. The PR needs to be revised to fix resolveCustomPrompt() instead.