Skip to content

feat(node): store-melt P2P propagation [BLOCKED — a dust coin deletes a live store network-wide] - #148

Merged
MichaelTaylor3d merged 4 commits into
mainfrom
feat/store-melt-propagation
Aug 4, 2026
Merged

feat(node): store-melt P2P propagation [BLOCKED — a dust coin deletes a live store network-wide]#148
MichaelTaylor3d merged 4 commits into
mainfrom
feat/store-melt-propagation

Conversation

@MichaelTaylor3d

@MichaelTaylor3dMichaelTaylor3d commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Closes dig_ecosystem#1316 (pieces #3 + #4).

Propagates a store MELT across the peer network so every holder stops hosting the store's .dig
content and reclaims disk. Wire is dig-gossip opcode 221 (StoreMeltedAnnounce), a public
all-peers broadcast — §5.4-EXEMPT from recipient-sealing, mTLS-authenticated and signed, with the
signature serving as attribution/anti-spam only and never as delete authority.

The delete authority

This is an irreversible, peer-triggered, network-correlated deletion, so the melt verdict is the
whole PR. Two cheaper signals were tried for it and both were found unsound before merge; the
history is kept in SPEC.md §14.5 and in the module docs because both shortcuts will look
attractive again.

  • anchored_root() == Ok(None) — rejected. That value is the node's fail-closed sentinel for
    no confirmed generation everywhere else, and CoinsetResolver produces it for a store that is
    not minted yet. A genuine melt does not even produce it.
  • The store_id hint index — rejected. A hint is an unauthenticated CREATE_COIN memo over an
    arbitrary 32-byte value, so anyone can write under any store's hint for the price of a dust coin.
    Enumerating all 53 DataLayer launcher coins on mainnet: 30 of the 53 live stores have a
    completely EMPTY store_id hint index
    , so for each of them one planted spent coin would have
    made the index non-empty and entirely spent — indistinguishable from a terminated lineage.

What ships instead: a forward walk of the singleton lineage along real COIN PARENTAGE.

  1. Identity + minted — the launcher coin whose coin_id == store_id exists and is SPENT.
    coin_id == store_id is a 256-bit hash preimage that cannot be ground. An unspent launcher is
    Live. On its own this discriminates nothing (it holds for every minted store); it anchors where
    the walk starts.
  2. Walk forward — follow the single ODD-amount child at each hop. An UNSPENT successor is
    Live. A spent coin whose children page is completely empty is Melted.

A coin's parent_coin_info is fixed by which coin was actually spent to create it, so placing a coin
anywhere in this walk requires spending a generation of the store — which requires the owner's
authority. The walk is unwritable by anyone but the owner and never consults a hint; the test mock
panics if either hint query is touched.

Fail-closed everywhere else: any transport error including mid-walk (an outage must not read as
"the lineage ended here"), more than one odd child, an absent launcher, exceeding the hop ceiling,
zero children at hop 0 (a minted launcher always created the eve singleton — and this also closes the
trap that coin_records_by_parent_ids has an empty DEFAULT impl on the trait), and any non-empty
page with no singleton in it
.

That last one is the completeness rule. The children query honours a server-side limit and truncation
surfaces spent records first — measured on coinset against the sibling hint query, no limit returns
349 records with 243 unspent while limit=5 returns 5 with zero unspent. A truncated page that
kept an even change coin but dropped the odd successor would read exactly like a terminated lineage.
Requiring the page to be entirely empty asserts completeness rather than trusting a page:
truncation cannot turn a non-empty result set into an empty one short of a zero limit, which is never
sent.

Measured against mainnet, not derived

All 53 DataLayer stores (global hint sha256("datastore")), run through this exact gate: 51
Live, 1 Melted
— the one genuinely terminated store, ending at hop 1 — and no ambiguous fork
anywhere. Deepest live lineage is 599 generations; 29 stores have their tip one hop from the
launcher (mean ~7), which is what MAX_LINEAGE_HOPS is sized from. The four stores previously
identified as live-with-empty-hint-index all classify Live here.

Because the walk costs one read per generation and the receive path runs per inbound announcement,
verdicts are memoised for a short TTL so a flood of announcements for one held store cannot multiply
into repeated walks. A stale verdict can only DELAY a real melt, never cause a delete.

Propagation

  • Holder path — for a held store the chain confirms melted and that is not already tombstoned:
    delete every held generation via the audited path-contained cache-remove, broadcast a signed
    announcement, tombstone. Live/Unknown are no-ops, retried next tick.
  • Receiver path — per inbound frame, in strictly increasing cost order: held-check first (an
    un-held flood costs no chain work at all), then the tombstone, then the on-chain verify, then
    delete + rebroadcast once, gated on a compare-and-set so only the holding→deleted transition
    re-emits. The epidemic quiesces after every holder has deleted once.

Tests

8 adversarial policy cases against spy seams, plus 13 cases driving the real ChainReads trait
with a crafted lineage — including the composition that broke the previous design (an empty hint
index plus one planted spent coin) both beside a resolvable live lineage and beside an unresolvable
one, asserting Live and Unknown respectively.

All 12 inverting mutations of the gate were confirmed to fail their test. The hop-cap test
asserts the EXACT read count rather than a <= bound, because a bound that is merely "not exceeded"
is also satisfied by a walk that stops far too early.

Both new background loops wrap only their per-iteration body in shared::catch_iteration with
recv()/tick() outside the guard (the #173/#174/#175 pattern); TombstoneSet recovers from lock
poisoning so one contained panic cannot disable melt propagation for the process's lifetime.

SPEC.md §14.5 documents the melt authority, the two rejected signals, the completeness rule, and
the mainnet conformance numbers.

Hardening added after the security gate's PASS

The gate passed the deletion path and then found four things worth fixing anyway; all four are in
this PR.

  • A latent delete bug.held_store_ids decodes hex (case-insensitive) while
    delete_all_generations compared the hex TEXT against hex::encode (always lowercase).
    CapsuleKey::parse admits and preserves mixed case, so for a mixed-case store directory the
    held-check passed, the delete matched nothing, and the node would tombstone the store, broadcast a
    melt of generations: 0, and keep serving the content it had just announced as deleted. Now
    matched on the parsed 32 bytes.
  • Tests over the REAL MeltCache. Every prior test drove CacheSpy, so the only code that
    actually unlinks files was untested — inverting its match to delete every other store stayed
    green. Three tests now drive the real impl against a real cache directory (correct store deleted,
    bystander survives, mixed-case store deleted, unheld store is a no-op).
  • Operator kill switch.DIG_NODE_STORE_MELT (default ON, explicit off/0/false/no
    disables), mirroring DIG_NODE_BACKFILL_ON_MISS. This is the node's only path that irreversibly
    deletes content in response to chain state and it propagates, so a fault is correlated across
    holders. Disabling is lossless — melted stores just keep costing disk.
  • Parentage re-check. The walk now discards any returned coin whose parent_coin_info is not the
    current coin, instead of trusting the server's notion of "children of X" — that trust would have
    handed back the very argument the design rests on.

Also closed two test-vacuity gaps the gate identified: the hop-cap test asserted
parent_queries == MAX_LINEAGE_HOPS, the same symbol on both sides, so cutting the ceiling to 100
kept it green — it now asserts the literal 10_000, with a compile-time const assertion pinning
the ceiling above the deepest measured mainnet lineage (599), and the live-tip floor raised from 60
to 599. And the gate-4 tombstone CAS had no test — receipts were driven sequentially so gate 2 always
caught the echo — so dropping the CAS stayed green; a deterministic two-task race (both held at a
barrier inside confirm_melt, guaranteeing both pass gate 2 before either inserts) now pins it.

18/18 mutants killed, single-threaded against a committed baseline, tree verified clean after.

Filed as follow-ups rather than fixed here

  • DIG-Network/dig_ecosystem#2090 — StoreMeltedAnnounce::verify is never called on ingest, and the
    rebroadcast re-emits the frame verbatim including sender_peer_id. Not a deletion vector (the
    receiver re-derives from chain and the signature is never the gate); forensic/reputational only.
  • DIG-Network/dig_ecosystem#2093 — every node melts through the same api.coinset.org that
    anchored_root uses, so per-hop re-derivation buys no independence against an oracle-wide fault.
    Cheap mitigation is a confirmation-depth requirement on the terminal coin's spent_block_index,
    already present on CoinRecord and unused.

@MichaelTaylor3dClaude

Copy link
Copy Markdown
ContributorAuthor

🔴 Review gate: CHANGES-REQUIRED — the melt signal is unsound (do not merge)

The production MeltChain impl derives melt from AnchoredRootResolver::anchored_root() == Ok(None), which is doubly wrong (code-traced):

  1. A genuine melt never yields Ok(None) — it yields Err.coinset_resolver.rs:60-66 maps only "not minted"/"unspent" error strings to Ok(None); a real melt (tip singleton spent with no datastore child) resolves to Err("singleton spend did not yield a store") (singleton.rs:887-889, confirmed by owner_melts_store_on_simulator) → store_melted.rs:331Err ⇒ Unknown ⇒ Ignore. So the feature never deletes on a real melt — it is inert against the exact event it exists to propagate.
  2. Ok(None) is the codebase's fail-closed sentinel (chain_view.rs:20-27: "not minted / no confirmed generation yet, treated as fail-closed"). For a held store it can arise transiently (external-indexer inconsistency, launcher-spend reorg), so Ok(None)⇒delete is also a wrongful-delete-of-live-data vector — exactly the transient false-positive flagged pre-merge.
  3. SPEC.md §14.5 (added in this PR) contradicts the code — it states "the melt authority is the lineage walk, NEVER a bare anchored_root() == Ok(None)," while the impl is that bare check.
  4. The 8 adversarial tests all drive a scripted ChainSpy, so none exercise the real resolver→MeltStatus mapping; findings feat: serve dig.getManifest locally from the embedded public manifest #1/ci: add PR quality gates (fmt/clippy/test/build) [#230] #2 pass the suite.

Sound as-is (keep on re-spin):decide_melt fail-closes on Live/Unknown; the receiver held-check precedes any chain call (DoS guard); the tombstone CAS gives race-free single-broadcast termination; deletion uses the audited cache_remove_cached; blast radius additive; 0.76.0 bump correct.

Required fix (direction): the MeltChain production impl must derive Melted from a positive lineage observation of a closed/melted singleton (the melt-spend terminal state), NOT from Ok(None); Ok(None) (never-minted/launcher-unspent) must map to Unknown/no-delete like the rest of the codebase. This needs a first-class resolver signal that distinguishes melted from never-minted from unreachable (the Result<Option<_>> tri-state can't — melt is currently indistinguishable from a generic Err). Add a test running a real melted-simulator lineage through the production resolver asserting Melted (and a never-minted lineage asserting no delete).

Awaiting the loop-security + adversarial gates, then this re-spins with a proper melt-detection primitive and re-runs all three gates. The green CI + green unit tests are not sufficient assurance for an irreversible network-triggered delete.


Generated by Claude Code

@MichaelTaylor3dClaude

Copy link
Copy Markdown
ContributorAuthor

Triple gate: all three independently blocked — re-implementing the melt signal (converting to draft)

Three fresh, independent adversarial contexts reached the same verdict:

  • loop-reviewer: CHANGES-REQUIRED
  • loop-security: CHANGES-REQUIRED (+ a concrete correlated mass-deletion exploit)
  • adversarial loop-decider: REFUTED

Unanimous root cause: the production MeltChain impl derives melt from anchored_root() == Ok(None), which is inverted-broken:

  • Ok(None) = launcher-unspent / never-minted (coinset_resolver.rs:55-66digstore-chain/singleton.rs:851-853), reachable transiently for a live/nascent held store (launcher-spend reorg, or a stale/hostile shared coinset.org snapshot reporting spent=false) → wrongfully deletes live data + broadcasts a false melt network-wide, correlated across nodes.
  • A genuine meltErr("singleton spend did not yield a store") (singleton.rs:889) → Unknown → Ignore, so the feature never fires on a real melt.
  • The sibling chainwatch.rs:89 correctly treats the same Ok(None) as Skip(NoConfirmedGeneration) (benign retry).
  • All 8 tests drive a scripted ChainSpy, so none exercise the real resolver→MeltStatus mapping — the exact broken link.

Sound and unchanged on the re-spin (all three cleared these):decide_melt fail-closes on Live/Unknown; the receiver held-check precedes any chain call (DoS guard); the tombstone CAS gives race-free single-broadcast termination; deletion uses the audited path-contained cache_remove_cached; blast radius additive; §5.4/§5.1 correct.

Required fix: the melt gate must key on a melt-SPECIFIC positive confirmation — the singleton lineage terminated in an owner-authorized melt spend (launcher spent + terminal child yielding no datastore output) — NOT anchored_root() == Ok(None) (which must map to no-delete like everywhere else). Plus a test running a real melted lineage through the production resolver (what the spy tests couldn't). A fix-design decider is determining whether this is achievable dig-node-only or needs a first-class Melted verdict in digstore-chain (a dig-store change, release-first). This PR is draft until re-implemented and re-run through all three gates.

This is the triple gate doing exactly its job: green CI and green unit tests were not assurance for a network-triggered irreversible delete.


Generated by Claude Code

@MichaelTaylor3d
MichaelTaylor3dforce-pushed the feat/store-melt-propagation branch 2 times, most recently from 1799221 to 001fc10CompareAugust 3, 2026 20:46
@MichaelTaylor3d

Copy link
Copy Markdown
ContributorAuthor

Re-implemented the melt signal + rebased onto main (0.93.7 → 0.94.0)

Independent re-trace of the code reached the same root cause the triple gate did, so the fix
targets exactly that link. Branch: feat/store-melt-propagation.

Rebase

3 commits replayed onto origin/main at c341d15. Only conflicts were the workspace version in
Cargo.toml + Cargo.lock (branch had 0.76.0, main is at 0.93.7). Resolved to 0.94.0
minor, a compatible new capability (§2.4).

The melt signal, replaced

impl MeltChain for Arc<dyn AnchoredRootResolver> is deleted. anchored_root() == Ok(None) now
means what it means everywhere else in the node: no confirmed generation, fail-closed, never an
authorization to delete.

The gate is now confirm_melt_via_chain(&dyn ChainReads, store_id), composed from two positive
chain facts
— it does not re-walk the singleton lineage (which parses every intermediate spend and
aborts on one unparseable generation, #747):

  1. The launcher coin coin_id == store_id EXISTS and is SPENT.coin_id == store_id is a
    256-bit hash preimage that cannot be ground, so identity is pinned to the one unforgeable anchor
    on chain — never to a look-alike singleton that merely currieslauncher_id == store_id
    (forgeable, #1473). spent is what proves the store was ever minted. An unspent launcher is
    Live
    — "not minted yet" is the opposite of a melt, and it was the exact state the previous
    cut deleted on.
  2. The launcher's hint index is NON-EMPTY and every generation under it is SPENT — the lineage
    terminated. One coin_records_by_hint(store_id, include_spent = true) read gives both halves.
    The non-empty half is load-bearing: an empty index means this index knows nothing about the
    store
    , which is indistinguishable from an un-indexed store, so it resolves to Unknown. Asking
    only for UNSPENT coins and deleting when none come back cannot make that distinction — that
    simplification is now pinned shut by a test.

Every error, absence, or ambiguity resolves away from deletion. Fact 2 is deliberately
conservative: candidates are not launcher-anchored before they count as "live", so an adversary who
can plant hints can only ever suppress a deletion, never cause one.

Tests

8 new cases drive the real ChainReads trait with crafted coin records — the exact link the
8 spy-driven policy tests could not reach. Every unused ChainReads method on the mock is
unimplemented!(), so the gate cannot silently grow a chain dependency; the mock also asserts the
gate asks with include_spent = true.

spent-launcher + non-empty + all-spent is the ONLY shape that yields Melted. Unspent launcher,
absent launcher, empty hint index, a single unspent generation, a planted unspent hinted coin, and a
transport failure on either read all resolve to Live/Unknown.

Also in this pass

Not addressed here (reported, not silently expanded)

Receiver gate 1 calls cache_list_cached(), a blocking two-level std::fs::read_dir walk of the
modules cache on the async runtime thread — once per inbound opcode-221 frame, i.e. driven by
unauthenticated remote input at attacker-chosen rate. It is a correct ordering (cheapest gate
first, no chain work for un-held stores) but "O(local)" is O(cache size) blocking I/O. A cheap
holds_store(store_id) probe belongs on the capsule seam, which this lane does not own.


Generated by Claude Code

@MichaelTaylor3d
MichaelTaylor3dforce-pushed the feat/store-melt-propagation branch from 001fc10 to 127a485CompareAugust 3, 2026 20:50
@MichaelTaylor3d

Copy link
Copy Markdown
ContributorAuthor

Gate-ready: CI fully green on 127a485, all three prior objections addressed

mergeable_state: clean, base is main's tip 1e9dd52.

Check
Rustfmt · Clippy · Lint commit messages · Check version incrementsuccess
Test + coveragesuccess
CodeQL · Analyze (rust / actions / javascript-typescript)success
build .deb amd64 · .deb arm64 · .msi windows-x64 · .pkg macos-universalsuccess

Zero review threads open (no inline review comments, no GHAS comments on the PR).

Local: cargo test -p dig-node-core --lib683 passed, 0 failed; cargo test --workspace,
cargo clippy --workspace --all-targets -D warnings, cargo fmt --check all clean.

The three gate objections, point by point

  1. "A genuine melt never yields Ok(None) — it yields Err, so the feature is inert." — the gate no
    longer consults anchored_root() at all. A melt is now a positive observation of the terminated
    lineage: launcher spent + a non-empty, all-spent hint index.
  2. "Ok(None) is the fail-closed sentinel; Ok(None) ⇒ delete is a wrongful-delete vector." — the
    state that produced it (launcher unspent / not minted) is now explicitly Live, pinned by
    unspent_launcher_is_not_minted_never_melted, which also asserts the second read is never reached.
  3. "All 8 tests drive a scripted ChainSpy, so none exercise the real resolver → MeltStatus
    mapping."
    — 8 new tests drive the real ChainReads trait with crafted coin records, against the
    production confirm_melt_via_chain. Unused trait methods on the mock are unimplemented!() so the
    gate cannot silently acquire a chain dependency, and the mock asserts include_spent = true.

Falsification

Each guard was inverted in the source and the corresponding test confirmed to FAIL. The harness
asserts the patch actually applied to the bytes on disk before running, so a no-op mutation reports
PATCH-DID-NOT-APPLY rather than a false "killed".

killed M1 unspent-launcher guard removed unspent_launcher_is_not_minted_never_melted
killed M2 empty hint index treated as melted an_unindexed_store_is_unknown_not_melted
killed M3 hint-read error treated as melted unreachable_chain_on_the_hint_read_is_unknown
killed M4 absent launcher treated as melted absent_launcher_coin_is_unknown
killed M5 all-spent weakened to any-spent a_single_unspent_generation_keeps_the_store_live
killed M6 launcher-read error treated as melted unreachable_chain_on_the_launcher_read_is_unknown
mutants killed: 6/6

Residual gap, stated plainly

The tests prove the mapping from chain facts to verdict. They do not prove the premise that a
melted store really presents on chain as spent launcher + all-spent hint index — that is reasoned
from sync_datastore's walk and from verify_pinned_root's reliance on the same launcher-id hint
(#1473), not observed on a simulator. Closing it properly means a first-class melt verdict in
digstore-chain, where the lineage machinery and its chia_sdk_test::Simulator coverage already
live — release-first, then this consumes it. Building a second simulator harness in dig-node would
duplicate that authority.

Until then the failure mode is bounded in the safe direction for every case except one: if the hint
index were populated but omitted a live tip, a live store would read as melted. Every other
divergence — unreachable chain, malformed answer, empty index, planted coins — declines to delete.


Generated by Claude Code

@MichaelTaylor3d

Copy link
Copy Markdown
ContributorAuthor

Gate verdict: CHANGES-REQUIRED — do not merge. The asymmetry is inverted.

The rewrite is a large improvement on the Ok(None) version, and the fail-closed reasoning holds for
every branch except one. But Fact 2 is attacker-satisfiable for the majority of real stores, and
the gate proved it against mainnet rather than by argument.

The claim that fails

store_melted.rs:365-368 states that an attacker who can plant hints "can only ever SUPPRESS a
deletion here, never cause one."
That rests on an unstated premise: that an honest live store's
generations always appear in coin_records_by_hint(store_id)
, so a planted spent coin can only sit
beside a real unspent tip.

That premise is false on mainnet. The gate enumerated all 53 real DataLayer launcher coins
(global hint sha256("datastore")), derived each store_id, and ran this exact two-fact gate against
every one:

0 Melted 23 Live 30 Unknown-via-EMPTY-index

30 of 53 live stores (57%) have an empty store_id hint index. Four were forward-walked via
get_coin_records_by_parent_ids423449b6…, 254b8e34…, c592b3ac…, 10d15285… — and each has an
unspent singleton tip one hop from its launcher. Their generations simply are not hinted to
store_id.

Those 30 resolve Unknown today only because the index is empty. Emptiness is one dust coin away
from being non-empty.

The attack

A hint is an unauthenticated CREATE_COIN memo over an arbitrary 32-byte value — this repo already
says so (digstore-chain/singleton.rs:1436-1441, #1473), and the sha256("datastore") hint queried
above is not a puzzle hash at all, which proves arbitrary values are indexed.

For store S = 423449b6… (live on chain right now, unspent tip a2f9ebc5…):

  1. Attacker spends any coin of their own to create a 1-mojo coin whose CREATE_COIN memos begin with
    S, then spends that coin. Cost: dust plus fee. No permission, no P2P access, no key material
    related to S.
  2. Fact 1coin_record(S) is Some, spent = true. Satisfied, as it has been since mint.
  3. Fact 2coin_records_by_hint(S, include_spent=true) returns exactly one record, the
    attacker's, spent. Non-empty, and .all(|g| g.spent) is true.
  4. confirm_melt_via_chainMelted; decide_melt(true, false, Melted)DeleteAndPropagate.
  5. Every holder runs delete_all_generations(S) and then broadcasts a signed opcode-221 announcement
    that drives the same verdict at the next hop.

No announcement is needed to start it.peer.rs:2847+ spawns run_melt_tick on the chainwatch
interval, calling confirm_melt on every held store unprompted — so the deletion fires network-wide on
a timer as soon as the planted coin confirms.

Fact 1 contributes no discrimination: launcher exists and is spent is true of every minted store. All
the load is on Fact 2, which is the attacker-writable one. The 256-bit-preimage argument correctly pins
identity, but proves nothing about termination.

Second issue, same direction — truncation reads as all-spent

get_coin_records_by_hint honours a server-side limit, and a truncated page reads all-spent.
Measured against api.coinset.org on hint ec7c3047…: no limit → 349 records, 243 unspent;
limit=5 → 5 records, 0 unspent.
The truncation ordering surfaces spent records first — precisely the
order that manufactures a false all-spent verdict.

digstore-chain passes (hint, None, None, Some(include_spent)) with no limit and no completeness
check
, so the gate's safety currently rests on an unverified coinset default cap. Fact 2 must assert
completeness, not accept a page as the whole index.

The test coverage has exactly the shape of the hole

a_planted_unspent_hinted_coin_suppresses_the_delete (:1138) tests the planted coin only in the SAFE
direction. an_unindexed_store_is_unknown_not_melted (:1124) tests the empty index with NO planted
coin. The composition — honest empty index PLUS one planted spent coin — is untested, and it is the
one that deletes live data.
Add it asserting Unknown.

Also worth correcting before merge

The PR body on the API is stale — it still describes the pre-rewrite anchored_root() == Ok(None)
design at version 0.76.0, while head 127a485 is 0.94.0 with confirm_melt_via_chain. Refresh it, or
the squash commit records the superseded and unsound model as the shipped one.

Verified sound, do not re-litigate

TombstoneSet::locked recovers from poisoning and never holds the std Mutex across an await; both new
loops wrap only their per-iteration body via shared::panic_guard::catch_iteration with recv()/
tick() outside the guard (the #173/#174/#175 pattern); delete_all_generations matches store_id
exactly and cannot spill into another store; every transport Err and the absent/unspent-launcher
cases resolve to Unknown/Live as claimed.

The lesson worth keeping

The previous design was found unsound by reasoning about what Ok(None) means. This one was found
unsound by asking the chain what real stores actually look like — and 57% of them do not match the
shape the gate assumes. For a peer-triggered, unrecoverable, network-correlated deletion, the premise
about honest state needs measuring, not deriving.

@MichaelTaylor3dMichaelTaylor3d changed the title feat(node): store-melt P2P propagation — verify-then-delete + convergent epidemic (#1316)feat(node): store-melt P2P propagation [BLOCKED — a dust coin deletes a live store network-wide]Aug 3, 2026
@MichaelTaylor3d
MichaelTaylor3dforce-pushed the feat/store-melt-propagation branch 2 times, most recently from c5c3b66 to 87234e2CompareAugust 4, 2026 00:10
@MichaelTaylor3d

Copy link
Copy Markdown
ContributorAuthor

Re-spun on coin parentage — the hint index is gone, and both gate findings are closed by test

Head 87234e2, rebased on main @ 95141b7.

1. "An honest empty index plus one planted spent coin must not resolve Melted"

Closed by removing the signal, not by patching it. The gate no longer consults a hint index at
all — the test mock unreachable!()s on bothcoin_records_by_hint and unspent_coins_by_hint, so
any future gate that reaches for one fails the suite outright.

The composition is now covered in both directions, because the safe direction alone is what made the
previous coverage vacuous:

testscenarioasserts
empty_hint_index_plus_a_planted_spent_coin_is_still_livehonest lineage resolvable + planted spent coinLive
a_planted_coin_beside_an_unresolvable_lineage_is_unknownhonest lineage unresolvable (chain fails mid-walk) + planted spent coinUnknown

The second is the one that matters: the planted coin must never supply the evidence the honest
lineage failed to. Structurally it cannot — a coin's parent_coin_info is fixed by which coin was
actually spent to create it, so placing a coin anywhere in this walk requires spending a generation
of the store, which requires the owner's key.
Fact 2 is no longer attacker-writable; it is
owner-writable only.

2. "Fact 2 must assert completeness, not accept a page"

Taking this seriously found the same class of hole in the new design, which I have fixed. The walk
originally concluded Melted on "no ODD child". coin_records_by_parent_ids honours a server-side
limit too — so a truncated page that kept an even change coin while dropping the odd successor would
have read exactly like a terminated lineage. No attacker required, same as your 349→5 measurement.

The rule is now: only a COMPLETELY EMPTY page may conclude a melt, and never at hop 0. A
non-empty page with no singleton in it is Unknown. That asserts completeness rather than trusting a
page — truncation cannot turn a non-empty result set into an empty one short of a zero limit, which
is never sent. Pinned by a_page_with_children_but_no_singleton_is_unknown_not_melted and mutant M11.

Hop 0 additionally closes a trap you would have hit next: coin_records_by_parent_ids has an empty
default impl
on the ChainReads trait, so any implementation that does not override it would
otherwise return Melted for every store instantly.

Measured against mainnet before designing, per your instruction

All 53 DataLayer launcher coins (global hint sha256("datastore")), run through this exact gate:

51 Live 1 Melted 1 previously mis-capped (real depth 599)
  • The 1 Melted is the genuinely terminated store cee3e2b0…, ending at hop 1 — spent parent,
    zero children, terminal spend retrievable.
  • The four stores you named as live-with-empty-hint-index — 423449b6…, 254b8e34…,
    c592b3ac…, 10d15285… — all classify Live at depth 1 here. I re-confirmed each has hint index
    size 0, so the 57% blind spot is exactly what parentage fixes.
  • Deepest live lineage 599 generations; 29 stores have their tip one hop from the launcher (mean
    ~7). MAX_LINEAGE_HOPS is sized from that. No ambiguous fork anywhere in ~380 hops.
  • Because the walk costs one read per generation, verdicts are memoised for a short TTL so an
    announcement flood for one held store cannot multiply into repeated walks. A stale verdict can only
    DELAY a real melt, never cause a delete.

Verification

Workspace green: cargo fmt --check, cargo clippy --workspace --all-targets -D warnings, and the
full cargo test --workspace (dig-node-core 690 passed; every crate 0 failed).

12/12 mutants killed — every inverting mutation of the gate confirmed to fail its test: the
unspent-launcher guard, hop-0 emptiness, mid-walk transport error, launcher-read failure, ambiguous
fork, unspent-successor stop, even-child acceptance, the hop ceiling, hint-index consultation, the
completeness rule, the planted-coin-plus-unresolvable-lineage case, and cache expiry.

Two things worth recording about the verification itself, since a clean mutation report is exactly
the kind of evidence that can lie:

  • The hop-cap test originally asserted parent_queries <= MAX_LINEAGE_HOPS, which stays true when the
    ceiling is cut to 2. It reported a kill while proving nothing. It now asserts the exact read
    count.
  • Two mutation harnesses briefly raced on the same file; one restored the source while the other
    measured, reporting a genuinely-killed mutant as UNFALSIFIED — and left an applied mutation
    behind in the working tree
    (Ok(None) | Err(_) => Melted, i.e. an unreachable chain authorising
    a delete). I caught it by grepping every Melted return rather than trusting the report, fixed it,
    re-ran the full suite, and re-ran all 12 mutants single-threaded against a committed baseline with
    an interference guard that aborts if the source changes underneath it. The final 12/12 above is
    from that clean run, with git status verified clean afterwards.

Version

main is 0.93.9; this takes 0.95.0, skipping the 0.94.0 that #149 is holding, so the
version-increment gate passes whichever of us merges first.

PR body refreshed — it no longer describes the superseded Ok(None) model, so the squash records
what actually ships.

Not merging. Ready for you to re-gate the deletion path.


Generated by Claude Code

@MichaelTaylor3d
MichaelTaylor3dforce-pushed the feat/store-melt-propagation branch from 87234e2 to a1d7326CompareAugust 4, 2026 00:31
@MichaelTaylor3d

Copy link
Copy Markdown
ContributorAuthor

Resume-ready state — gate-ready, not merged

Branchfeat/store-melt-propagation · HEADa1d7326 · base603187a (chore(release): v0.93.9) · mergeable_stateclean · CI 0 non-green (14 success, 1 skipped).

Done

The rebase that was interrupted by the cap had in fact completed and pushed — verified from remote,
not memory: no .git/rebase-merge state, clean worktree, local HEAD == origin head.

The blocking gate finding is closed by removing the signal, not patching it. Verified against the
pushed artifact:

  • coin_records_by_hint / unspent_coins_by_hint appear in the file exactly three times: once in a
    doc comment explaining why the hint index is rejected, and twice as unreachable!() stubs in the
    test mock. Zero production calls. Any future gate that reaches for a hint index fails the suite.
  • The "can only ever SUPPRESS a deletion here" claim is gone — grep returns nothing.
  • The verdict now walks the singleton lineage along real coin parentage. A coin's
    parent_coin_info is fixed by which coin was actually spent to create it, so placing a coin
    anywhere in the walk requires spending a generation of the store — the owner's key. The 57%
    empty-hint-index population is irrelevant to it; the four stores named in the gate all classify
    Live at depth 1.
  • Completeness, not a page: taking the truncation point seriously found the same class of hole in
    the new design — concluding on "no ODD child" would let a truncated page that kept an even change
    coin read as terminated. Only a completely empty page may conclude a melt, and never at hop 0.
    A non-empty page with no singleton is Unknown.
  • The untested composition is now tested in both directions:
    empty_hint_index_plus_a_planted_spent_coin_is_still_live (resolvable lineage → Live) and
    a_planted_coin_beside_an_unresolvable_lineage_is_unknown (unresolvable lineage → Unknown).

Measured on mainnet before designing: all 53 DataLayer stores → 51 Live, 1 Melted (the one genuinely
terminated store, hop 1), deepest live lineage 599 generations, no ambiguous fork in ~380 hops.

Workspace green (fmt, clippy -D warnings, cargo test --workspace, dig-node-core 690 passed).
12/12 mutants killed on a single-threaded run against a committed baseline.

PR body refreshed — 0 references to the superseded Ok(None) model, so the squash records what ships.

Version

main 0.93.9; #149 and #179 both hold 0.94.0; this PR takes 0.95.0, which clears whichever of
those two lands first. If BOTH merge before this one, main reaches 0.95.0 and this needs a bump
to 0.96.0 — one-line change plus a rebase.

Exact next action

None on my side. Awaiting the full re-gate of the deletion path. If the gate returns
CHANGES-REQUIRED, resume in C:\tmp\worktrees\dn148 on feat/store-melt-propagation; the mutation
harness is C:\tmp\worktrees\falsify2.py (single-threaded only — two concurrent runs corrupt the
source, see the previous comment) and the mainnet probes are probe.py / probe2.py in the same
directory.


Generated by Claude Code

MichaelTaylor3dand others added 3 commits August 3, 2026 20:19
First red stub for store-melt P2P propagation: the pure decide_melt
gate (only a held, on-chain-confirmed melt deletes) and the set-once
TombstoneSet that bounds each node to one broadcast per store. The 8
adversarial actuator tests + the receive/holder wiring follow.
Refs #1316
Co-Authored-By: Claude <noreply@anthropic.com>
Piece #3 (receiver) + #4 (holder) of store-melt propagation, over
spy-testable seams (MeltChain/MeltCache/MeltBroadcast) with the
fail-closed NC-9 gate: only a held, on-chain-confirmed melt deletes;
Err/Unknown never deletes; held-check precedes any chain read; the
TombstoneSet CAS bounds each node to one rebroadcast so the epidemic
terminates. All 8 adversarial tests green.
Refs #1316
Co-Authored-By: Claude <noreply@anthropic.com>
…PEC (#1316)
Spawn the store-melt receive ingest (opcode 221) beside holdings ingest
off a second inbound receiver, and a holder watch loop that deletes +
announces this node's own on-chain-melted stores — both sharing one
tombstone. Bump workspace 0.75.6 -> 0.76.0 (feat/minor) + Cargo.lock.
SPEC.md §14.5 documents the receive->verify->delete->rebroadcast
contract, the NC-9 fail-closed gate, and the §5.4 public-broadcast
exemption.
Refs #1316
Co-Authored-By: Claude <noreply@anthropic.com>
@MichaelTaylor3d
MichaelTaylor3dforce-pushed the feat/store-melt-propagation branch from a1d7326 to e650a04CompareAugust 4, 2026 03:21
…-writable hint
The melt gate is the authority for an irreversible, peer-triggered, network-correlated
delete. Two cheaper signals were tried for it and both were unsound; this replaces the
second with the singleton lineage itself, and the choice is settled by measurement
against mainnet rather than by argument.
What was wrong
--------------
The previous cut concluded "melted" from a NON-EMPTY, all-spent `store_id` hint index.
A hint is an unauthenticated CREATE_COIN memo over an arbitrary 32-byte value (#1473),
so ANY party can place a record under ANY store's hint for the price of a dust coin.
Enumerating all 53 DataLayer launcher coins on mainnet shows why that is fatal: 30 of
the 53 LIVE stores have a completely EMPTY store_id hint index — their generations are
not hinted to store_id at all. For every one of them a single planted spent coin makes
the index non-empty and entirely spent, which the gate could not distinguish from a
terminated lineage. Cost to erase a live store network-wide: dust plus fee, no
permission, no P2P access, no key material. `run_melt_tick` would have fired it on a
timer with no announcement at all. `get_coin_records_by_hint` is also truncatable, and
truncation surfaces spent records first — the exact order that manufactures a false
melt.
What replaces it
----------------
A forward walk of the singleton lineage along real COIN PARENTAGE:
1. Identity + minted — the launcher coin whose `coin_id == store_id` exists and is
SPENT. An unspent launcher is Live (not minted yet is the opposite of melted). This
fact discriminates nothing by itself; it anchors where the walk starts.
2. Walk forward — follow the single ODD-amount child at each hop. An UNSPENT successor
is Live. A spent coin with NO successor is Melted.
A coin's `parent_coin_info` is fixed by which coin was actually spent to create it, so
placing a coin anywhere in this walk requires spending a generation of the store, which
requires the owner's authority. The walk is unwritable by anyone but the owner, and it
never consults a hint — the mock panics if either hint query is touched.
Fail-closed everywhere else: any transport error INCLUDING mid-walk (an outage must not
read as "the lineage ended here"), more than one odd child, an absent launcher, and
exceeding the hop ceiling. Zero children at hop 0 is Unknown, not a melt: a minted
launcher always created the eve singleton, so an empty first hop means the answer is
untrustworthy — which also closes the trap that `coin_records_by_parent_ids` has an
empty DEFAULT impl on the trait.
Measured against mainnet
------------------------
All 53 DataLayer stores: 51 Live, 1 Melted (the one genuinely terminated store, ending
at hop 1), 1 previously mis-capped. Deepest live lineage 599 generations; 29 stores have
their tip one hop from the launcher; no ambiguous fork anywhere. MAX_LINEAGE_HOPS is
sized from that measurement. The four stores the gate named as live-with-empty-hint-index
all classify Live here.
Because the walk costs one read per generation and the receive path runs per inbound
announcement, verdicts are memoised for a short TTL so a flood of announcements for one
held store cannot multiply into repeated walks. A stale verdict can only DELAY a real
melt, never cause a delete.
Tests: 12 cases drive the real ChainReads trait with a crafted lineage, including the
composition the gate flagged as untested and lethal — an empty hint index plus one
planted spent coin — asserting Live. All ten inverting mutations of the gate were
confirmed to fail their test; the hop-cap test asserts the EXACT read count, because a
`<=` bound is also satisfied by a walk that stops far too early.
root [workspace.package].version 0.94.0 -> 0.96.0 (minor, new capability). Skips
0.95.0, which PR #179 holds.
Co-Authored-By: Claude <noreply@anthropic.com>
@MichaelTaylor3d
MichaelTaylor3dforce-pushed the feat/store-melt-propagation branch from e650a04 to 85a115fCompareAugust 4, 2026 03:53
@MichaelTaylor3d

Copy link
Copy Markdown
ContributorAuthor

Gate-ready — 85a115f, all checks green, four post-PASS fixes landed

Rebased on main @ 7833ee5 (v0.94.0). mergeable_state: clean, 0 unresolved threads, 15/15
checks green
(Clippy, Rustfmt, Test + coverage, CodeQL + all 3 analyzers, commitlint, version
increment, release-script tests, all 4 platform builds).

Version: 0.96.0. main is 0.94.0 and #179 holds 0.95.0, so this skips 0.95.0 and clears
both. Re-read from origin/main immediately before pushing, not assumed.

The four findings

1 — the latent delete bug (highest value, and it was real).held_store_ids decodes hex and is
case-insensitive; delete_all_generations compared the hex TEXT against hex::encode, which is
always lowercase. CapsuleKey::parse admits and preserves mixed case (is_ascii_hexdigit, with its
own test asserting Ab..cD parses). So for a mixed-case store directory the held-check passed, the
delete matched nothing, and the node would tombstone the store, broadcast generations: 0, and keep
serving content it had just announced as deleted. Now matched on the parsed 32 bytes, so both sides
look at the same identity.

Three tests now drive the realMeltCache for Arc<Node> against a real cache directory — the
only code here that unlinks files, previously untested because all 25 tests drove CacheSpy. The
inverted-match mutation you found (delete every other store) is killed by
the_real_cache_deletes_only_the_named_store, which asserts the bystander survives.

2 — the hop-cap assertion was still vacuous, and you were right about why.parent_queries == MAX_LINEAGE_HOPS puts the same symbol on both sides, so it moves with the constant; switching <=
to == changed nothing. It now asserts the literal 10_000. The floor in
a_lineage_with_an_unspent_tip_is_live went 60 → 599, the deepest lineage that actually exists
on mainnet. And the constant-to-constant relationship is now a compile-timeconst assertion
rather than a runtime one — clippy was correct that a folded assert! proves nothing, so cutting the
ceiling below the measured floor now fails the build. I did not #[allow] it.

3 — operator kill switch.DIG_NODE_STORE_MELT, default ON, disabled by an explicit
off/0/false/no — the exact shape of DIG_NODE_BACKFILL_ON_MISS. Both loops sit behind it and
bring-up logs when it is off. Disabling is lossless: melted stores keep costing disk, and nothing
else depends on melt propagation having run.

4 — parentage re-check. The walk now discards any returned coin whose parent_coin_info is not
the current coin. Two tests pin it in both directions: a foreign coin alone is Unknown, and a
foreign coin beside the genuine successor still resolves Live — a filter that dropped everything
would otherwise make the first pass for the wrong reason.

Also — the tombstone CAS gap.convergence_terminates_at_holder_count drives receipts
sequentially, so gate 2 always caught the echo and gate 4 was never under test. The race is made
deterministic rather than hoped for: a barrier inside confirm_melt holds both callers until both
have passed gate 2, so without the CAS both would delete and broadcast. Dropping the CAS now fails.

Verification

18/18 mutants killed, single-threaded against a committed baseline, with an interference guard
that aborts if the source changes underneath and a final git status check. Mutants cover all four
fixes: inverted delete match, case-sensitive text compare, ceiling cut to 100, parentage filter
removed, parentage filter over-filtering, CAS dropped, kill switch un-disableable — plus the original
twelve.

Before touching anything on resume I re-ran your three checks: tree clean, and exactly one
production MeltStatus::Melted return (:453), reachable only from hop >= 1 && children.is_empty()
after an Ok(_) page, with :424 and :435 both returning Unknown. No mutation rode in behind
the lint fix.

Filed as follow-ups, not fixed here

  • DIG-Network/dig_ecosystem#2090 — StoreMeltedAnnounce::verify never called on ingest; rebroadcast
    re-emits the frame verbatim including sender_peer_id. Forensic/reputational only — the receiver
    re-derives from chain and the signature is never the delete gate.
  • DIG-Network/dig_ecosystem#2093 — every node melts through the same api.coinset.org as
    anchored_root, so per-hop re-derivation buys no independence against an oracle-wide fault.
    Proposed cheap mitigation: confirmation-depth on the terminal coin's spent_block_index, already
    on CoinRecord and unused.

Not merging. Ready for the scoped re-gate of the delete path.


Generated by Claude Code

@MichaelTaylor3d

Copy link
Copy Markdown
ContributorAuthor

Merge ordering, recorded so it is not lost: #179 merges first, then this.

This PR sits at 0.96.0, #179 at 0.95.0, main at 0.94.0. If this lands first, #179's 0.95.0 fails the version-increment check and needs another bump; the reverse order works with no further edits. Both are green and both are under scoped security re-gates right now.

Two notes on the delta while it waits:

The clippy resolution is better than what I asked for. I said to assert a literal. Making the constant-to-constant relationship a compile-timeconst assertion is stronger — cutting MAX_LINEAGE_HOPS below the measured 599-hop mainnet floor now fails the build rather than a test, which is the one place it cannot be skipped or deleted. Raising the live-tip floor 60 → 599 matters too: the old floor sat below a real store's lineage depth, so it could not have caught a cap set too low for production.

The case-sensitivity bug is the one worth remembering from this PR.held_store_ids decoded hex (case-insensitive) while the delete compared hex text against a lowercase hex::encode. A mixed-case store directory passed the held-check, deleted nothing, and the node would then tombstone it, broadcast generations: 0, and keep serving content it had just announced as deleted — a node lying about its own state, with no error anywhere. It was invisible because every one of the 25 tests drove CacheSpy; the only code that actually unlinks files had no test at all, which is why inverting the delete match to delete every other store stayed green.

That is now three of this session's false-green findings that share one shape: the test drove a substitute for the thing under test. Filed as #2094.

On the local cranelift_codegen ... rlib format error you flagged — CI is green and your change touches nothing wasmtime-related, so the stale-artifact reading is almost certainly right. For the record it was not caused by my target/ cleanup: I deleted only dn-1974, dig-app-pr78, dig-relay-1938, hub-1998 and digapp-gui-preview, and deliberately left dn148 and dn-2071 alone as live lanes. Worth a cargo clean in that worktree if it recurs.

@MichaelTaylor3d
MichaelTaylor3d merged commit 52ed50a into mainAug 4, 2026
16 checks passed
@MichaelTaylor3d
MichaelTaylor3d deleted the feat/store-melt-propagation branch August 4, 2026 04:44
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@MichaelTaylor3d