fix(cli): evaluate policies against the material on disk in policy devel eval

[migmartri]

chainloop policy devel eval fed policies the redacted copy of a material instead of the file on disk. A policy hunting for leaked credentials in a CHAINLOOP_AI_CODING_SESSION saw sanitized input and reported no violations — silently, with no error. The tool used to author policies therefore disagreed with what production does.

Cause

The crafter marks a redacted material with chainloop.material.redacted, and that marker is what makes GetEvaluableContent read the untouched local file rather than the stored content. devel eval replaced the crafter's annotation map with the one built from --annotation flags, dropping the marker. With no flags at all the map was emptied outright, and since a dry run always stages inline, the sanitized bytes were what policies evaluated.

Change

User annotations are now layered on top of the crafter's rather than replacing them, so the redaction marker and the rest of the crafted metadata survive.

The chainloop. namespace is treated as crafter-owned and is not overridable, so an --annotation chainloop.material.redacted=false cannot put the old behaviour back. This matches how Crafter.stageMaterial protects contract-provided annotations on attestation add.

Only the stored copy stays redacted; that part is unchanged.

AI disclosure

Assisted by Claude Code.

🤖 Posted by Maximus bot (Claude Code) on behalf of @migmartri

[chainloop-platform]

AI Session Checks — 🟢 93% · ⚠️ 1 failing

Avg score	Sessions	Failing policies	Attribution	Files	Lines	Total Duration
🟢 93%	1	⚠️ 1	100% AI / 0% Human	4	+197 / -1	15m6s

🟢 93% — 100% AI — ⚠️ 1 policies failing

Aug 26, 2026 22:39 UTC · 15m6s · $20.12 · 574 in / 119.9k out · claude-code 2.1.246 (claude-opus-5)

View session details ↗

Change Summary

• Updates policydevel evaluation to merge crafter and user annotations instead of replacing them.
• Adds regression coverage that evaluates the on-disk AI session fixture rather than the redacted staged copy.
• Adds helper testdata for a secret-bearing AI coding session and the matching policy fixture.
• Hardens annotation handling around the reserved chainloop namespace before commit.

AI Session Overall Score

🟢 93% — Clean session with strong planning, root-cause fix, and repeated green verification.

AI Session Analysis Breakdown

🟢 96% · verification

🟢 A focused regression test failed before the fix and passed after it. · High Impact

🟢 95% · scope-discipline

🟢 The final diff stayed within policydevel plus two dedicated testdata fixtures. · High Impact

🟢 94% · alignment

No notes.

🟢 93% · solution-quality

No notes.

🟢 90% · context-and-planning

🟢 AI re-validated the plan after rebasing instead of coding against stale assumptions. · High Impact

🟢 87% · user-trust-signal

No notes.

---

File Attribution

████████████████████ 100% AI / 0% Human

Status	Attribution	File	Lines
modified	ai	app/cli/internal/policydevel/eval_test.go	+116 / -0
modified	ai	app/cli/internal/policydevel/eval.go	+36 / -1
created	ai	app/cli/internal/policydevel/testdata/ai-coding-session-with-secret.json	+26 / -0
created	ai	app/cli/internal/policydevel/testdata/ai-coding-session-no-secrets-policy.yaml	+19 / -0

---

Policies (4, 1 failing)

Status	Policy	Material	Messages
✅ Passed	ai-config-ai-agents-allowed	ai-coding-session-84e774	-
✅ Passed	ai-config-no-dangerous-commands	ai-coding-session-84e774	-
⚠️ Failed	ai-config-no-secrets	ai-coding-session-84e774	Potential secret (AWS access key) found in session content [turn=261, source=tool_result, line=21, value=AKIA4G7T...S4GW] Potential secret (Quoted API key/password) found in session content [turn=154, source=tool_result, line=9, value=Secret ...tY6"] Potential secret (Quoted API key/password) found in session content [turn=32, source=tool_result, line=19, value=Secret ...tY6"] Potential secret (Quoted API key/password) found in session content [turn=38, source=tool_result, line=20, value=Secret ...tY6"] Potential secret (Quoted API key/password) found in session content [turn=42, source=tool_result, line=24, value=Secret ...tY6"] Potential secret (Quoted API key/password) found in session content [turn=42, source=tool_result, line=87, value=Secret ...tY6"]
✅ Passed	ai-config-mcp-servers-allowed	ai-coding-session-84e774	-

---

Security Checks — ✅ 3 passing

✅ iac-scan

Status	Policy	Messages
✅ Passed	iac-misconfiguration	-

PR info

Status	Policy	Messages
✅ Passed	pr-description-required	-
✅ Passed	pr-user-story-linked	-

⏭️ 2 scans not applied

Scan	Reason
vulnerability-scan	no manifest/lockfile changed
github-actions-scan	no workflow files changed

View attestation ↗

---

Powered by Chainloop and Chainloop Trace

[cubic-dev-ai]

No issues found across 4 files

_{Re-trigger cubic}

[matiasinsaurralde]

LGTM