feat(materials): add PITEST_XML material type for PIT mutation testing reports

[migmartri]

Summary

Add a material type for PIT mutation testing reports, so a mutations.xml can be attested like any other test or coverage artifact.

Mutation testing measures test-suite quality rather than code quality: the tool injects small deliberate faults ("mutants") into compiled code, reruns the covering tests, and reports which mutants went undetected. A surviving mutant is code your tests execute but do not actually verify — the gap that line coverage cannot see. Chainloop can already attest JaCoCo and Cobertura coverage; mutation results are the natural companion and there is currently no way to bring them in except the generic EVIDENCE kind, which cannot take XML at all since the policy engine only handles JSON.

PIT is the de-facto standard on the JVM. It mutates bytecode rather than source, so it needs no recompilation per mutant and reruns only the tests that cover each mutated line, which is what makes mutation testing practical on Java in the first place.

The report

PIT's native output formats are HTML, XML and CSV, selected with outputFormats. XML is the one worth taking — CSV lacks structure and HTML is not machine-readable. It is written to target/pit-reports/<yyyyMMddHHmm>/mutations.xml.

Produced by adding pitest-maven (plus pitest-junit5-plugin for JUnit 5 projects) and running:

mvn org.pitest:pitest-maven:mutationCoverage

Report shape

From PIT's XMLReportListener, the document is a <mutations> root wrapping a flat list of <mutation> elements, with no aggregate summary anywhere. Any score is derived by counting:

<mutations>
 <mutation detected='true' status='KILLED' numberOfTestsRun='2'>
 <sourceFile>Owner.java</sourceFile>
 <mutatedClass>org.springframework.samples.petclinic.owner.Owner</mutatedClass>
 <mutatedMethod>getPet</mutatedMethod>
 <methodDescription>(Ljava/lang/String;)L…</methodDescription>
 <lineNumber>119</lineNumber>
 <mutator>org.pitest.mutationtest.engine.gregor.mutators.ConditionalsBoundaryMutator</mutator>
 <indexes><index>7</index></indexes>
 <blocks><block>2</block></blocks>
 <killingTest>…</killingTest>
 <description>changed conditional boundary</description>
 </mutation>
</mutations>

Field	Notes
detected	Boolean — whether the test suite caught the mutant. The primary signal
status	KILLED, SURVIVED, TIMED_OUT, NO_COVERAGE, NON_VIABLE, MEMORY_ERROR, RUN_ERROR
numberOfTestsRun	Tests executed against this mutant
sourceFile / mutatedClass / mutatedMethod / lineNumber	Location
mutator	Which mutation operator was applied
killingTest	The test that caught it, when one did
description	Human-readable summary of the change

With fullMutationMatrix enabled, killingTest is replaced by killingTests, succeedingTests and coveringTests.

Note that NO_COVERAGE means no test touched the mutated line at all — a plain coverage hole rather than a weak assertion. Consumers will generally want to distinguish it from SURVIVED, so the status attribute must be preserved as-is and not collapsed into the detected boolean.

Proposed work

• app/controlplane/api/workflowcontract/v1/crafting_schema.proto — add PITEST_XML at the next free enum value. Note that feat(materials): add OVERSECURED_JSON material type for Oversecured mobile scan reports #3369 proposes taking 43, so whichever lands second should take the following number.
• app/controlplane/api/workflowcontract/v1/crafting_schema_validations.go
• pkg/attestation/crafter/materials/pitest.go + test, following cobertura.go: read the file, unmarshal, validate, then uploadAndCraft
• register the crafter in pkg/attestation/crafter/materials/materials.go
• regenerate the jsonschema / TS artifacts and add the row to the material types docs table

Format detection

cobertura.go is the right model here — pin XMLName to the mutations root so xml.Unmarshal rejects a mismatched document (JaCoCo's <report>, JUnit's <testsuite>, Cobertura's <coverage>) rather than silently accepting it.

One decision worth making explicitly: whether to accept a report containing zero <mutation> elements. Unlike a coverage report, an empty mutation report is not "0%" — it means nothing was analysed, and it would make any consumer divide by zero. Rejecting it at craft time gives a clearer error than letting an empty document through, but it does risk failing a pipeline whose module legitimately produced no mutants. cobertura.go sets a precedent for rejecting genuinely dataless input.

Related, for a follow-up rather than this issue

A second material type for the Stryker mutation-testing-elements schema would cover the same domain for other ecosystems — it is emitted natively by Stryker (JS), Stryker.NET, Stryker4s and Infection (PHP).

It is deliberately not proposed as the Java path here: PIT does not emit that schema natively, only through io.github.wmaarts:pitest-mutation-testing-elements-plugin, a third-party plugin with a single maintainer. PIT's own XML is the safer canonical route for JVM projects.

🤖 Posted by Maximus bot (Claude Code) on behalf of @migmartri