Skip to content

repair-proof contract between neighbor_sync and audit/pruning is implicit and unenforced #1

Description

@RexBytes

Summary

The replication repair-proof gate (anti-outsourcing: a node must prove it
sent replica-repair hints before it may audit / prune-gate its peers) is
implemented as a producer → shared-state → consumer pipeline whose
correctness depends on invariants the type system and call graph do not
enforce. Producer and consumers never reference each other; they communicate
only through the RepairProofs struct, so any semantic drift compiles cleanly
and silently weakens or disables a security mechanism.

Surfaced as hidden coupling (files that co-change in git but share no
structural edge) and confirmed by reading the code.

The pipeline

neighbor_sync.rs (PRODUCER)         types.rs (RepairProofs)        audit.rs + pruning.rs (CONSUMERS)
build_replica_hints_for_peer   →    record_replica_hint_sent(  →   has_mature_replica_hint(
  SentReplicaHint{ key,               peer, key,                      challenged_peer, key,
    close_peers }  ← snapshot         current_close_peers,            close_peers,   ← recomputed independently
                                      hinted_at_epoch }               current_sync_epoch, now )
       │ NeighborSyncOutcome                 ▲                        gate: proof.hinted_at_epoch < current_epoch
       └ mod.rs::record_sent_replica_hints ──┘                              + close-group match + time maturity
         (hinted_at_epoch = sync_cycle_epoch)
  • Producer: neighbor_sync.rs:145 builds SentReplicaHint { key, close_peers };
    recorded via mod.rs::record_sent_replica_hints
    RepairProofs::record_replica_hint_sent with hinted_at_epoch = *sync_cycle_epoch.
  • Two consumers, each independently recomputing the close group:
    • audit.rs:194mature_audit_keys_for_peerhas_mature_replica_hint (audit.rs:423)
    • pruning.rs:490has_mature_replica_hint

What's wrong (three unenforced invariants)

  1. The close-group snapshot must be computed identically by the producer and
    both consumers.
    The producer snapshots a "self-inclusive close group" into
    SentReplicaHint.close_peers; each consumer recomputes close_peers from its
    own lookup. If the computations diverge (different K, distance metric,
    routing-table vs DHT source, self-inclusion), proofs silently fail to match
    and audits / prune-gates silently never mature. There are three
    independent close-group computations (1 producer + 2 consumers) that must
    agree by developer discipline.

  2. The epoch maturity convention must agree. Producer stamps
    hinted_at_epoch; consumers require proof.hinted_at_epoch < current_epoch
    (types.rs:412), i.e. "≥ one full sync cycle old," carried only by a raw
    u64. Drift toward lenient → hints mature too early → peers audited /
    struck before they could repair (honest peers penalized). Drift toward
    strict → hints never mature → outsourcing / freeloading undetected.

  3. Self-inclusion is a convention, not a type. Both
    SentReplicaHint.close_peers and RepairProofEntry.close_peers are
    documented "self-inclusive"; the membership gates (types.rs:369,
    audit.rs:195) are off-by-one-member if any side computes it self-exclusive.

The compiler and call graph miss all three because every field is a
structurally valid HashSet / HashMap / u64 on every end.

Evidence this is real, not theoretical

Three separate changes have edited the two ends of this same contract in
lockstep — the classic "co-change with no code edge" signal:

  • gate audits by mature repair proofs (the feature itself)
  • batching neighbor-sync hint construction (producer side)
  • "tolerate one lagging peer in the prune proof gate" (consumer side)

What's worth implementing (ranked)

A. One shared close-group function (highest value, low risk). Extract a
single fn close_group_for(key) -> HashSet<PeerId> (one definition of K +
distance + self-inclusion) and call it from the hint producer and both
consumers (audit, pruning), instead of three independent computations.
Makes invariants #1 and WithAutonomi#3 structural — they can no longer drift.

C. A cross-module round-trip contract test. Isolated unit tests on
RepairProofs exist (audit.rs:1504+, types.rs:967) but none span producer →
proofs → consumer. Add a test that builds a hint against close group G, then
asserts (i) it matures & gates when the consumer still sees G, (ii) it does
not mature after the close group rotates, (iii) it does not mature
before one epoch elapses — for both the audit and pruning consumers.

B. (optional) SyncEpoch newtype wrapping the u64 with a single
is_mature_for_audit(hinted, current) -> bool, so the maturity rule lives in
one place and can't diverge across the producer and two consumers.

D. (floor) Document the rendezvous on RepairProofs: name its producer
(neighbor_sync::build_replica_hints_for_peer) and consumers
(audit / pruning::has_mature_replica_hint) and the three invariants.

Recommendation: A + C. A removes two of the three drift risks at the source
(now across three call sites, not two); C pins the epoch / maturity behaviour A
doesn't cover and gives a regression guard.

Severity

Hardening, not a known live bug — the current computations do appear to agree.
But the contract gates a security mechanism, breaks silently in either
direction, and now has two independent consumers, so making it structural is
worth more than a typical refactor.

Provenance

Surfaced via stitchgraph hidden-coupling analysis (git co-change, no structural
edge) and confirmed by reading neighbor_sync.rs / types.rs / audit.rs /
pruning.rs / mod.rs. Captured in-repo at docs/findings/repair-proof-coupling.md
(branch claude/loving-cori-l6rv6j).

The feature is intended design upstream (WithAutonomi/ant-node): it landed in
PR WithAutonomi#106 ("gate audits by mature repair proofs"), with later re-tunings in PR
WithAutonomi#150 (producer / neighbor-sync) and PR WithAutonomi#134 (prune-proof gate). None of those
introduced a shared close-group computation, an epoch newtype, or a
producer→consumer round-trip test.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions