Skip to content

feat: Build a structured diff engine to detect vulnerability regressions between commits #13

Description

@Nanle-code

Overview

ChainProof currently produces a standalone scan result per invocation. There is no mechanism to compare results between two commits, branches, or points in time — meaning teams cannot easily answer: "Did this PR introduce new vulnerabilities?" or "Did we successfully fix the issues from last sprint?". A structured diff engine addresses this gap.

Problem

A PR review process without a diff-aware scanner is noisy:

  • A PR touching 2 files out of 100 will still surface findings from all 100 files
  • No way to distinguish "pre-existing" from "newly introduced" without manual comparison
  • Re-auditing the entire project on every PR is expensive and slow

Proposed Design

Core Diff Types

interface ScanDiff {
  introduced: Finding[];    // In new scan, not in old scan
  resolved: Finding[];      // In old scan, not in new scan
  persisted: Finding[];     // In both scans (matched by fingerprint)
  summary: {
    newCritical: number;
    newHigh: number;
    resolvedTotal: number;
  }
}

Fingerprinting for Diff Matching

Findings are matched across scans using the same fingerprint strategy as the baseline feature: SHA-256(ruleId + normalized_path + line + snippet_hash). Line-number tolerance: ±3 lines to handle minor code movements.

Usage Modes

CLI — compare two JSON reports:

chainproof diff old-results.json new-results.json
# Outputs: 2 new critical, 1 resolved high

CLI — compare against git ref:

chainproof scan contracts/ --diff HEAD~1
# Internally: stash, scan at HEAD~1, restore, scan at HEAD, diff

GitHub Action:

  • On PRs: automatically diff against base branch scan
  • Only fail CI on introduced findings above min-severity
  • PR comment shows diff-aware summary: "2 new critical introduced, 1 resolved"

Report Format

Markdown diff report:

## ChainProof Diff Report

### 🔴 Newly Introduced (2)
| Rule   | File                  | Line | Severity |
|--------|-----------------------|------|----------|
| CP-107 | contracts/Vault.sol   | 42   | Critical |

### ✅ Resolved Since Last Scan (1)
| Rule   | File                  | Line | Severity |
|--------|-----------------------|------|----------|
| CP-115 | contracts/Auth.sol    | 18   | High     |

Acceptance Criteria

  • diffScans(oldResult, newResult): ScanDiff exported from core
  • chainproof diff <old.json> <new.json> CLI command
  • chainproof scan --diff <git-ref> flag using git stash/checkout
  • GitHub Action diff mode: compare against base branch
  • PR comment updated to show diff-aware summary
  • Diff report in Markdown and JSON formats

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions