fix(read): trace every locator + prove the real chain resolves a Resource (#836, #1590)

[MichaelTaylor3d]

TL;DR — the #836 read-leg hypothesis is REFUTED (misdiagnosis #3), not fixed-blind

The read-leg DATA root has been misdiagnosed 2-3x as a locator-composition bug: "the Downloader re-locates by the raw resource id UNBRIDGED → 0 providers → no fetchRange." I traced the ACTUAL production download-locator chain and proved that is false. This PR ships the instrumentation + the proof; it does NOT close #836 (the true cause is elsewhere — see below). Draft, do not merge as-is.

Traced ground truth (the end of the misdiagnosis)

The production download locator (NodeContent::new + for_dht) is:

download_locator = SelfExcludingLocator(UnionLocator([
PoolProviderLocator(connected_pool), # source 0 (pool-first)
CapsuleFallbackLocator(SelfExcludingLocator( # source 1
UnionLocator([DhtProviderLocator, Empty, Empty]))),
]))

For a ContentId::Resource query, BOTH sources resolve the holder:

• PoolProviderLocator (pool_locator.rs) offers every connected pool peer for ANY content id — resource key included — keyed to the resource, at the peer's reachable pool address. It is NOT capsule-only.
• CapsuleFallbackLocator (capsule_fallback.rs) bridges the resource key → the announced parent capsule record, and it sits inside download_locator (via locator.clone()), so the bridge is applied on the Downloader's own locate_and_confirm path — not only on the engine's find_providers probe.

Confirmed facts from reading the code + dig-download 0.7.1 source:

• dig_dht::ContentId::to_key() includes the retrieval_key, so download_key(resource) != download_key(capsule). The arbiter log line fetch_resource: ... content=bf77d1d6 located=1 (bf77d1d6 = the capsule key) is therefore a fetch for the capsule id (the #1586 backfill), not the resource read's bridged probe — one root of the prior misreads.
• The holder's getAvailability (peer.rs::availability_presence) treats a RESOURCE item as capsule-presence (available iff the capsule root is held), so a resource confirm succeeds at a holder that holds the capsule.
• dig-download's locate_and_confirm (orchestrator.rs:860) calls the injected download_locator with the full resource id; run_inner errors immediately (NotFound) if the first confirm is empty — it relocates only on a progress stall, never on an empty first locate.

Why the prior tests never caught it

Every prior read-leg test used dig-download's MockProviderLocator, which ignores the content id and answers every query (resource key included). That bypasses the resource→capsule bridge entirely and models a DHT that announced the resource key — a condition that never exists in production. So the tests were green against the wrong condition.

What this PR adds

1. Per-locator DEBUG ground-truth tracing (capsule_fallback.rs, self_excluding_locator.rs, union_locator.rs; pool_locator.rs already logged): input content id (variant/key) + per-branch/per-source output counts + merged count. The next arbiter e2e will show which source offered (or failed to offer) the holder for the failing read — the decisive per-source fact the prior misdiagnoses lacked.
2. Two characterization/regression tests (download.rs) composing the REAL chain (CapsuleFallback bridge + pool union + self-exclusion) around a capsule-only DHT source and a best_address()-faithful transport:
   • resource_read_resolves_via_the_real_chain_when_only_the_capsule_is_announced — pool holder @ reachable + DHT capsule @ unreachable → serves via the pool.
   • resource_read_bridges_to_capsule_on_the_download_locate_without_a_pool_entry — empty pool + reachable DHT capsule → serves via the bridge alone.
     Both are GREEN at a51980f, proving the download locate resolves a Resource whenever a holder is present.

No behaviour change (observability + tests only). Patch bump: workspace 0.58.3 → 0.58.4, dig-node-core 0.18.2 → 0.18.3.

Reframed root cause + next step (for the orchestrator)

Since the real chain resolves a Resource whenever a reachable holder exists in either source, the e2e DATA 404 means that at the failing read moment neither source had a reachable holder: the gossip pool had not yet converged to include the holder (so no reachable pool address), leaving only the DHT capsule record whose advertised address is unreachable on the relayed net — and dig-download errors immediately on that empty-confirm (no relocate). i.e. a discovery-convergence / reachability race on the first read, not a locator bug. The four prior locator PRs (#95/#96/#97/#1590) could not fix it because the locator was never the fault.

Resume step: re-run runlegs-read-arbiter.sh on cached a51980f + this branch. The new tracing will print, at the failing resource read, union_locator: per_source=[<pool>, <capsule-fallback>] and capsule_fallback: by_resource/by_capsule/merged — confirming whether the pool was empty (race) and whether the DHT capsule branch returned an unreachable-only record. That pins the fix to one of: (a) make the Tier-2 read resilient to the convergence race (bounded relocate/retry while peers are connecting, since dig-download won't relocate on an empty first confirm), or (b) seed the connected pool + DHT routing synchronously before the node begins serving reads. I deliberately did NOT ship a speculative retry — that would be fixing blind against an unconfirmed condition, the exact trap that produced the prior misdiagnoses.

Closes none. Refs #836, #1590, #1586, #1572, #1062.

[MichaelTaylor3d]

e2e race-diagnostic verdict: RACE REFUTED (retry-after-convergence probe)

Ran the decisive retry-after-convergence probe on this branch (c8992c0, base v0.58.4) with the per-locator DEBUG tracing, on the isolated DIG_TESTNET ephemeral EC2 fleet (relayed / shared-DHT, connect-before-pin). run-id e2e-1062-20260725-121024.

CONNECT / ANNOUNCE / DISCOVER all PASS. Both nodes connected_peers=1 (direct); holder #1586 announce fired (refreshed provider records ... announced=2); reader getAvailability returns the holder as a provider with reachable :9444 direct addrs.

The verdict-defining result

	read #1	read #2 (after +30s convergence)
HTTP	404	404
pool had reachable holder :9444	yes	yes (getAvailability = 1 provider)
fetch_resource located	1 (connected_pool=1)	1 (connected_pool=1)
inbound fetchRange at holder	no	no

read #2, with the holder fully converged in the pool, still 404s. The pool was already converged at read #1 too. The race reframe is WRONG.

Real root cause — a self-dial / reflexive-candidate bug in the resource-FETCH leg

The locator composition works: capsule_fallback bridges the resource to the parent capsule (by_resource=0 by_capsule=1 merged=1), pool_locator offers the holder with dial_targets=[[::ffff:172.31.73.226]:9444], and a Direct connection to the holder's real :9444 is actually established. But the actual fetch transport then does:

peer connection established kind=Direct remote=[::ffff:172.31.73.226]:9444 <- holder reached (control)
dial failed; falling through kind=Direct reason=all candidates failed:
[[::ffff:172.31.71.193]:44690: tcp connect ...:44690: Connection refused (os error 111)] <- READER's OWN ip:ephemeral
dial failed; falling through kind=Relayed reason=refusing relayed self-dial (target == local peer_id)
peer serve: fetch missed error=download failed: content not found: no providers located for ContentId::Resource

172.31.71.193 is the reader's own IP (holder is 172.31.73.226). The fetch-transport candidate set contains the LOCAL node's reflexive address and resolves the target as local peer_id, so the reader tries to fetch from itself (ephemeral port → refused) then refuses a relayed self-dial → collapses to no providers located. The holder receives inbound connections on :9444 but serves no fetchRange. This is a WU7-class reflexive-address/self-dial candidate bug in the resource-download path — not discovery timing.

Implication for the fix

Bounded relocate / retry-while-converging will not fix this (the pool is already converged and the holder already located). The fix must correct the peer-fetch dial-candidate handoff (dig_nat::strategy + locator→transport) so it dials the located holder peer_id + the :9444pool_locator already offered, instead of the local reflexive addr / a relayed self-dial. Secondary (separate): reader §21 whole-store backfill fails HTTP 400 no §21 identity, not authorized, or served root differs.

Evidence: s3://dig-loop-e2e-transfer-873139760123/e2e-verdicts/verdict-pr98-race-diag.json + full-logs-pr98-race-diag.txt.

Fleet torn down, zero survivors (verified). No issue closed.

[MichaelTaylor3d]

Closing stale trace-investigation draft per read-leg resume anchor (dig_ecosystem#1586, 2026-07-25T15:26Z). Superseded by the ground-truth runtime key-trace lane: LOCATE-empty root persists on main 77c1422 after PR#100; new lane prints runtime .to_key() at pool_locator offer + download-core lookup and asserts a fetchRange RPC is TRANSMITTED. Branch fix/836-downloader-resource-locate stays on remote as prior art.