Uh oh!
There was an error while loading. Please reload this page.
fix(policies): evaluate redacted materials against the content that was stored - #3383
Draft
jiparis wants to merge 5 commits into
Draft
Conversation
…as stored Secret redaction only rewrote the copy of an AI coding session that leaves the machine; policies were still evaluated against the untouched original on disk. Since a policy is user-authored Rego that can send what it reads to an allowed hostname, that made the policy engine a path for the very credentials redaction exists to remove. It also does not survive server-side evaluation, where the redacted copy is the only content available. Crafters that do not store the artifact verbatim now report the bytes they stored through materials.Craft, and the crafter passes them to both policy verifiers. Content resolution fails closed: a material annotated as redacted refuses to resolve at all without its sanitized copy, uniformly across CAS backends, rather than falling back to the file on disk. This also covers skip-upload materials, whose sanitized bytes were previously retained nowhere. Nothing is held in memory for materials that were not transformed: those keep resolving their content from the inline copy or the file, as before. Assisted-by: Claude Code Signed-off-by: Jose I. Paris <jiparis@chainloop.dev>
Contributor
AI Session Checks — 🟢 92% · |
| Avg score | Sessions | Failing policies | Attribution | Files | Lines | Total Duration |
|---|---|---|---|---|---|---|
| 🟢 92% | 1 | 100% AI / 0% Human | 99 | +1046 / -491 | 6h54m9s |
🟢 92% — 100% AI — ⚠️ 1 policies failing
Aug 28, 2026 11:10 UTC · 6h54m9s · $112.27 · 1.3k in / 360.2k out · claude-code 2.1.250 (claude-opus-5)
Change Summary
- Plumbs sanitized AI-session content through crafting and policy evaluation so policies read stored bytes, not the original file.
- Updates policy evaluation tests and bundled no-secrets policy inputs around redacted placeholders and annotations.
- Refactors the materials API around
CraftResult, foldsGetEvaluableContentFromintoGetEvaluableContent, and renamesCraftResult.TransformedtoContent.
AI Session Overall Score
🟢 92% — Well-planned, aligned session with strong verification and calm user follow-up.
AI Session Analysis Breakdown
🟢 94% · user-trust-signal
No notes.
🟢 93% · alignment
🟢 Follow-up refactors tracked specific user requests rather than drifting into unrelated work. · High Impact
🟢 92% · scope-discipline
No notes.
🟢 92% · verification
🟢 AI ran focused tests, full suites, vet, and lint before reporting completion. · High Impact
🟢 91% · solution-quality
No notes.
🟢 90% · context-and-planning
🟢 AI wrote and revised a substantive plan before major edits began. · High Impact
File Attribution
████████████████████100% AI / 0% Human…and 74 more file(s).
Policies (4, 1 failing)
Status Policy Material Messages ✅ Passed ai-config-ai-agents-allowedai-coding-session-06ec99- ✅ Passed ai-config-no-dangerous-commandsai-coding-session-06ec99- ⚠️ Failedai-config-no-secretsai-coding-session-06ec99- Potential secret (Quoted API key/password) found in session content [turn=436, source=tool_result, line=10, value=Secret ...tY6"]
- Potential secret (Quoted API key/password) found in session content [turn=58, source=tool_result, line=28, value=Secret ...tY6"]
✅ Passed ai-config-mcp-servers-allowedai-coding-session-06ec99-
Powered by Chainloop and Chainloop Trace
There was a problem hiding this comment.
1 issue found across 17 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="pkg/attestation/crafter/api/attestation/v1/crafting_state.go">
<violation number="1" location="pkg/attestation/crafter/api/attestation/v1/crafting_state.go:160">
P2: When callers supply material content for a material without an `Artifact`, this branch is never reached, so policies receive `{}` rather than the supplied bytes. Select the explicit content before artifact-dependent resolution so `WithMaterialContent` honors its override contract for every material kind.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| switch { | ||
| case useInlineContent: | ||
| case len(content) > 0: |
There was a problem hiding this comment.
P2: When callers supply material content for a material without an Artifact, this branch is never reached, so policies receive {} rather than the supplied bytes. Select the explicit content before artifact-dependent resolution so WithMaterialContent honors its override contract for every material kind.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At pkg/attestation/crafter/api/attestation/v1/crafting_state.go, line 160:
<comment>When callers supply material content for a material without an `Artifact`, this branch is never reached, so policies receive `{}` rather than the supplied bytes. Select the explicit content before artifact-dependent resolution so `WithMaterialContent` honors its override contract for every material kind.</comment>
<file context>
@@ -110,30 +111,62 @@ func (m *Attestation_Material) NormalizedOutput() (*NormalizedMaterialOutput, er
-
switch {
- case useInlineContent:
+ case len(content) > 0:
+ // NOTE: ingestMaterialToJSON re-reads the artifact from `value` for
+ // the kinds it projects from a path (JUNIT_XML, HELM_CHART), so
</file context>
jiparis
marked this pull request as draft
August 28, 2026 17:12
Reporting the bytes a crafter stored in place of the artifact was bolted on as a second, optional interface method alongside Craft. Fold it into Craft itself: Craftable returns a CraftResult carrying the material and, when the artifact was not stored verbatim, the bytes that replaced it. One method instead of two, so a crafter cannot report content through a path the generic craft flow does not take, and the capability is visible in the interface rather than discovered by type assertion. Every crafter but the AI coding session stores the artifact as it found it and leaves Transformed nil. Assisted-by: Claude Code Signed-off-by: Jose I. Paris <jiparis@chainloop.dev> Chainloop-Trace-Sessions: 06ec9990-4ad5-4b3c-ad86-2421b24dba57
…Content The single-argument GetEvaluableContent had no callers left outside tests once both policy verifiers moved to the variant taking an explicit content source, so keeping it as a delegating wrapper only offered a second way in — one that resolves a redacted material's content from the file on disk and fails. Give GetEvaluableContent the content parameter and drop the wrapper. Callers that have nothing to supply pass nil, which is the behaviour the old signature had. Assisted-by: Claude Code Signed-off-by: Jose I. Paris <jiparis@chainloop.dev> Chainloop-Trace-Sessions: 06ec9990-4ad5-4b3c-ad86-2421b24dba57
The field holds the content the crafter stored, which is what every consumer wants from it. Naming it after the transformation described how it came to exist rather than what it is. Locals that carried the field's value are renamed to match; the prose that describes crafters transforming an artifact is unchanged. Assisted-by: Claude Code Signed-off-by: Jose I. Paris <jiparis@chainloop.dev> Chainloop-Trace-Sessions: 06ec9990-4ad5-4b3c-ad86-2421b24dba57
…option The functional-option machinery existed to carry a single []byte, which is more ceremony than the value deserves: a type, a constructor, an accumulator and an option struct so that one call site could pass one slice. VerifyMaterial takes the content as a plain parameter instead. Callers that have nothing to supply pass nil, which reads no worse than omitting an option and makes the alternative visible in the signature. Assisted-by: Claude Code Signed-off-by: Jose I. Paris <jiparis@chainloop.dev> Chainloop-Trace-Sessions: 06ec9990-4ad5-4b3c-ad86-2421b24dba57
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Secret redaction (#3368) only rewrites the copy of a
CHAINLOOP_AI_CODING_SESSIONthat leaves the machine. Policies were still evaluated against the untouched original on disk, having access to any potential secret that could have leaked into the AI session.The invariant now implemented is that the policy engine receives exactly the bytes that were stored.
Changes
transformCrafterseam and report the bytes they stored.materials.Craftsurfaces them asCraftResult.EvaluableContent, and the crafter passes them to both policy verifiers via the newpolicies.WithMaterialContentper-call option.chainloop.material.redacted=truereturnsErrRedactedContentRequiredrather than falling back to the file on disk. This is uniform across CAS backends, inline included.chainloop policy devel evalthreads the same content through, so a policy is developed against the input the real run produces.[REDACTED:<rule>]placeholders and the redaction annotations, since the raw pattern it looked for is no longer present in its input.Materials that were not transformed are unaffected and hold nothing extra in memory: their content still resolves from the inline copy or the file on disk. This also closes a gap for
skipUploadmaterials, whose sanitized bytes were previously retained nowhere at all.Notable consequences
A redacted material's policy input can no longer be reconstructed from persisted crafting state alone, so a future push-time or server-side material evaluation must plumb the bytes through as well.
materials.Craftnow returns*CraftResultandpolicies.Verifier.VerifyMaterialtakes variadic options — both breaking for external importers of these packages.Not covered here
Two gaps remain that are properties of redaction rather than this plumbing:
--skip-secret-redactionstores the session as captured, and secrets counted asreport.Unlocated(landing in a protected path, or spanning two JSON leaves) are not redacted in the stored copy either. Detecting placeholders in the upstreamai-coding-session-no-secretspolicy is the second part of the issue and lives in another repository; that policy should keep its pattern matching alongside the placeholder check, since pattern matching remains the only thing that catches the secrets redaction could not reach.AI disclosure
Produced with assistance from Claude Code; every commit carries an
Assisted-by: Claude Codetrailer.